When I was a boy, my piano teacher once told me that playing a solo is “writing a song onstage”. Being a Dungeon Master isn’t much different than any other type of performance art in this regard. To really differentiate yourself at the table, you need to be able to improvise, and have those amazing moments that will keep people talking about your game week to week. You can spend months preparing a setting and an adventure, and your players will always be able to find things that you haven’t thought about.
So when presented with an adventuring party that does what you didn’t expect, you have two options. The first one is to stop whatever they’re doing, and force them to get back on your very specific path. There’s a temptation to say “Don’t ever railroad your players”, but a little bit isn’t terrible. Forcing the players out of a line of action is really only a good idea when they’re about to start an encounter you didn’t expect, and you don’t want to scramble and quickly roll up a bunch of guards because the players decided to take over a town. I try to avoid having to do this, because it can break the flow of the game, which can lead to distractions, and further derailment.
The second option, and the one that should be preferred, is to roll (no pun intended) with whatever the players want to do. Not only does this not stop the flow of the game, but it also leads to situations that are much more fun than whatever you planned before. If the players want to stop their quest, charter a ship, enslave the crew and become pirates, let them try. If the half-orc wants to try and seduce his way out of an encounter, allow the attempt. Don’t be rigid, don’t say “no that’s stupid”, and don’t let them ever think that you wish they stuck with the plot you spent months writing.
If you play D&D (or GM most tabletop games) and have trouble improvising, you have a set of polyhedral tools to help you. Rely on your dice. You don’t have to dictate the players’ successes and failures, allow the dice to help you. Additionally, get a good feel for the skills, and which one should be used for which situation. I usually DM with a character sheet in front of me, so that I always have a list of what the players can use. Coming up with a good DC is also important, as it is another form of improvisation.
While improvisation is incredibly important, relying on it all the time is a hallmark of an unprepared DM. You need to be at least a little prepared. Personally, I won’t play unless I have a pretty good idea for how the session should go, and I make sure to roll up all my enemies before the session starts. Your players will know when you are improvising all the time, and it can bring some people out of the experience.
Another aspect to preparing is that it without any pre-written history, it is much harder to improvise dialog. As the creator of a campaign setting, you need to prepare some history about the realm, and the characters that show up in it. Give the major NPCs (i.e. the ones that players interact with) at least one motivating factor. Personally, I’m a fan of history, so I typically write a fairly general history going back a couple hundred years. The point is, all you need to do is write a short backstory for the “Major” NPCs, and they’ll write their own dialog.
While preparation is important, I want to stress that you don’t need to have every possible piece of dialog written in advance (unless you want to of course). Your preparation before the first session should include the setting, some history, the characters motivations, and the mechanistic aspects of the game. The mechanistic aspects are all the things that you need to roll for and should be highly fleshed out before the start of each session. When it comes to a session, I usually write a list of all the stuff I would like to happen for the session. This is as far as I will go when it comes to a “script” for an adventure.
So remember, improvisation is good. If you are playing from a script the whole game, you’re not going to be as engaged, and as a result, neither will your players. Again, this doesn’t mean that you should try and start your sessions completely unprepared. D&D is like having people over for a meal. You aren’t going to start cooking when everyone arrives, you prepare with enough food (game content) so that everyone will have enough. The important thing is to strike a proper balance. If you meticulously plan every possible aspect of your campaign, try loosening it up a bit. If your campaign is an unprepared every session, try writing down three to five bullet points that you want to hit next time, and start from there.