Saying "it's a yes or no question" is not always fair.
In some ways it seems manipulative. Sometimes it is indeed a yes or no question, but most people know that certain answers require further explanation. It gives off the impression that you don’t wanna hear someone’s side of the story/debate. Sometimes “yes, but” or “no, but” is warranted.
The question was presented as yes or no. It's your prerogative to want to provide additional information, it is the prerogative of the asker to want a simple answer. It's inconsistent to champion the desires of the responder while disregarding the desires of the asker. Unless there's some kind of power imbalance (e.g. employer/employee), every party is entitled to engage with others as they see fit.