Saying "it's a yes or no question" is not always fair.

In some ways it seems manipulative. Sometimes it is indeed a yes or no question, but most people know that certain answers require further explanation. It gives off the impression that you don’t wanna hear someone’s side of the story/debate. Sometimes “yes, but” or “no, but” is warranted.