Balance in extremely creative combat encounters

The more creative the encounter is the more ways you have to adjust on the fly and the less CR is accurate. Sure it gives you an idea on which ballpark you are in.

For example you have multiple lair actions and a selection of those activate each round. If the PCs are hurting then you can choose a few soft ones to give then a chance to regroup and recover. If reinforcements are coming perhaps there are a few less. Similarly if the PCs are cruising, hit 'em hard.

Whether or not this crosses the border into DM fiat/fudging I'll leave up to you.

Also remember you don't have to use the highest DPR actions in general.

Start by not really worrying about balance too much. CR is next to useless, even at very low levels. Just feel it out experimentally over a few battles, and if you go too big, withhold some of the nastier enemy abilities.

I do those big creative boss battles. I start by finding the most similar enemies to my custom enemies and running some basic numbers in Kobold Fight Club (maybe 5 minutes). Then I eyeball a rough difficulty multiplier based on their level, classes, and magic items.

My players are currently level 17 with minimal magic items and based on some experimental results I take whatever Kobold gives me for extremely deadly and then multiply the difficulty by about three to actually have a difficult battle. That might mean more enemies, or it might mean more damage, higher HP, higher AC, increased saves, etc. Right now a challenging battle for them is one in which I take a put maybe three CR 20-something monsters on the field against my party of five.

Even with playing just the monsters in the book, high-level encounters in D&D will be incredibly swingy because there are a lot of abilities for both PCs and monsters where one character rolling well or poorly can completely change whether the fight is a near-TPK or a cakewalk. The party succeeded on the save against the cool boss ability, well they're probably gonna be fine. Or they all failed, well now they're fucked. The boss failed the save and now it's paralyzed, guess you've pretty much won. Or it succeeded, now you've wasted your turn. That kinda thing.

That's why making encounters with a bunch of swingy abilities can actually tip things back into being controllable. If your boss is getting whomped harder than expected, he gets desperate and breaks out the super-kill abilities. Or if the party is the ones getting clobbered, maybe the boss gets overcompetent and doesn't use their super-kill ability until it's too late.

I've found that at the end of the day, the PCs are generally expected to win, and they have a major advantage: when they hit zero, they get to roll death saves and can be healed, whereas the enemies usually just die. This is actually a huge factor in their favor, and explains partially why PCs can beat enemies that might seem way above their level. So honestly I don't worry too much about making strong enemies, my parties can usually handle them. And if they handle it too easily, it's not the end of the world.

In D20, every fight is cinematic. If you give them the same feel in your game and have an end goal separate from killing monsters, a huge 3d environment and a ticking clock of tension then you’re emulating it perfectly.

Brennan Lee Mulligan does it by fudging, if I understand things correctly. He improvises very much, too. His group seems to really enjoy it but it's a very different playstyle from the more "gamist" styles where players put care into their character's stats and such. If you sit down to make a character for a Brennan table, never ever think "Ooh, I wanna take lunging attack, that seems like an effective maneuver". That'd be meaningless. Instead, only think about what you would think would be fun or cool to do, like "oh, Rakish Audacity seems like it'd be a fun play pattern".

Colville's approach is similar to Brennan's.

Mercer's official statement, on the other hand, is that he only fudges for new players. He is generous with revivify which solves a lot of problems, too. That said, he rolls secretly and there's no behind-the-screen cam that can tell whether he pads HP or whatever. But as far as how he officially describes his game, he plays to find out. He preps situations, the players play them out, anything can happen. In that kind of game, your choices for your character matter a lot more. It's more akin to my own jam than the Dimension 20 stuff because of that.

Me, who doesn't fudge, I just have made peace with how sometimes the monsters steamroll the PCs and sometimes it's the other way around. That was my "solution". + when making custom monsters, or converting monsters from other games, I've started using the Forge of Foes approach of lower AC and higher HP. Often makes the fights feel a li'l more interactive.

I'm also scared of falling into "MPE"—"my precious encounter" syndrome, where you put so much care into creating an encounter that you end up railroading the characters into it because you don't have the heart to see it go to waste.

My big thing is to avoid hard numbers wherever possible. EG, you've got a big spooky encounter with complicated abilities, passive/lair effects, etc etc; it's health pool is whatever the plot demands. :P Realize you underestimated the raw dps of your players? Bump that health up until it has a chance to show off what it can do and feel like a threat. Realize this thing will just murder the hell out of them? Tune it down until they only need 1-2 good hits to bring it down before they're out. Similar with saving throws, bonuses to hit, ac, etc. I usually don't commit to anything until at least a round or two has passed. Much easier to balance the interplay of complicated features and abilities after you've seen what they can do, and there's no reason you can't do that on the fly, assuming your players trust you to want them to have the best experience possible.