It's not strictly wrong to let the story go wherever the players want as a DM, as long as you can keep up. In fact I'd say one of the traits of a good DM is knowing how much to let the story diverge from their plans, and to be able to think quickly and respond to the players on the spot. If the direction is entertaining, most parties are okay with exploring it. How tightly you keep the story on the rails is part of the game. In D&D sometimes you play the DM as much as you do their scenario.

What would be interesting is figuring out if you could configure ChatGPT's tolerance for divergence from the planned scenario. So that parties who prefer high walls can have them, while people who find your story thoroughly entertaining can experience it like you did.