From what I gather, he backed down because all his top men's families were being threatened. Prigozhin's family was kept safe, of course, but I don't think they expected Russia to threaten the families of those further down the ladder.
Mercenaries and auxiliaries are useless and dangerous; and if one holds his state based on these arms, he will stand neither firm nor safe; for they are disunited, ambitious and without discipline, unfaithful, valiant before friends, cowardly before enemies
-- Machiavelli in the Prince
Not to call out Putin's Russia for being amateur hour, but the Prince is probably the first thing to read for an aspiring dictator.
I mean he had to know, right? With guys like Putin if you go hot, you either kill him or he has you killed. There is no in-between. His entire image relies on that. Maybe they had Prigozhin's family or something?