You may not think of writers as being inventors, but really, that's exactly what we are. Sometimes we're just inventing a simple story, other times an entire universe. Even though it's all in our heads (or on paper, once it's written), there's a lot of work that goes into it.
If you read a fantasy or a sci-fi novel, you can understand how much work went into inventing the world of that story. I used to write fantasy (and probably will again) and I feel like I only scratched the surface on developing the world in which my characters lived. You can have fantasy elements in the real world, or you can completely create your own universe. Either way, there is a lot of work involved. While fantasy gives you a bit more room to do what you want, it still has to make sense. Once you create the world, you have to stick to the rules of that world, or the reader is going to notice. If you're writing a vampire story, and suddenly you feel the need for a dragon to show up, you'd better sure that all of these creatures can exist within the same universe. Nothing can be random and there has to be a reason for every choice you make in the story.
If you're writing any genre of realistic fiction, it's less about the world the characters live in and more about the story itself. You have to be sure any interaction with the outside world makes sense, like researching your setting, making sure the way your characters talk is actually realistic. If your characters go to a real place, then you're not going to be able to fool anyone who's actually been there. As for the story itself, it has to make sense, too. You want to make sure the characters act in a way that's realistic. You're not inventing a a whole universe, but the tiny little world that your characters live in. A lot of stories are character driven, and in this case, your characters are going to be your most important inventions.
I think most writers understand that when they get the first spark of a story idea, there's going to be a lot of work involved with getting that story written. You have to invent characters, situations, and sometimes, an entire world. Of course, it's not all work. It can be a lot of fun, too. :)