It’s hard to get an idea of just how much of a Hard Problem decent GMing by a computer is. But it maybe helps to consider just a very small slice: a player vs a single NPC foe.
Let’s say they have both been thrust, unarmed, into an Immersive Sim level with a climbable environment, exploding barrels, swing&climbable chains, dousable lights, flammable crates/bridges/buildings, etc, and each knows roughly where the other is.
The AI character, given very good, AAA quality navmeshing and AI, will know where he can run to, where he can jump down safely, where he can jump up or over, where he can duck behind to get cover from, how to circle around the player while maximizing cover, and how to avoid flames. So, he will approach the player as fast as he can, possibly taking a route which maximizes cover, then try to beat the player to death with his fists.
The player will inevitably kill the AI player in short order. Not with his bare hands, but with improvised weapons and the environment. Crate to the face! Burn down the room! Burn both ends of the bridge! Sneak! If you’re taking a beating, make an unsafe jump to relative safety! Lure them by the exploding barrels then throw a torch at the barrel! And so on.
No AI character can navigate a navmesh and create emergent tactics of anything like the same order as a human, not even in AAA games.
But worse, the AI, because he is an AI, does not understand the things that he would if he were a real intelligence. He will not accept surrender. He will not parley. He will not hide, whether to protect himself or to lie in wait. He will not take pity on his foe. He will not beg for mercy himself, nor is he susceptible to threats, bribery, emotion or logic. He is essentially “on rails”. He will do nothing but attack until death. So, in the restriction of the actions he can react to and can perform, he restricts the actions of the players in dealing with him. Even a bardic player cannot ever say “I try to seduce the dragon”.
And that problem can be extended out, from that single NPC, to every single other aspect of the game world. Not only can every aspect of the game world not interact with any other aspect in any way other than those preprogrammed (if you cannot use a dead mouse with a wooden spike and torch to get rat on a stick in UA, it is because those things were not programmed to interact that way), but the player themselves can only perform the pre-defined actions determined by the keys they can press. They cannot try to disguise themselves, or draw on the walls, or even speak to anyone unless the programmers decided they may.
But I’d argue that’s OK, and that for most of my life I have been wrong about this, as I felt that the more perfect the simulation of life, the better the game would be.
Instead, the job of the computer DM isn’t to make the world, or the foes, awesome.
To me this understanding gave a HUGE insight into fantastic UX design in general: the job of any user interface, including a computer game, is to make the user (not their character) feel that they themselves are awesome.
So a game where you can make a rat on a stick does not make you feel awesome, but a game where you figure out which things can be cooked, does. A game where the AI defeats you because it understands the emergent properties of the level better than you does not make you feel awesome, but game where you are the only person who figures that stuff out, is. I’d argue that a game where the NPCs can use any emergent tactic that you have already used, pushing you to get even more creative and to feel awesome even more times, is an even better game, though.