The lonely only

Okay, smarties. Lend me your ears. Anyone have a problem with the placement of "only" in the following sentence, or does it sound just fine to you?

"It had only taken me five minutes to reach the forest."

I know that strict grammarians hold that "only" must be adjacent to the word or phrase it modifies (in this example, "five minutes"), but in contemporary prose the strict version sounds strange to me. I'm free use either version, but I'm concerned that the preference for the incorrect place...