Do it. Just write something. Don't worry about publishing it, just create the piece and then read it for yourself. Then tweak it. Maybe rewrite it. Maybe push it aside and write something different that came to you. I'm not talking about full novels, start with a short story. I'm late to the party... and I wish I'd begun this ten years ago. It... is... fun.
I knew I wanted to set the next book on the Dutch island of Saba but I didn't know much more than that. I took a scuba trip there and toured various sites, and while in one particular location it all came to me in a rush.