For the same reason that "TGIF" is a great way to remember Toes Go In First when you're putting on your shoes, but it's not the reason that your toes go in first
But the association with the left side of the body goes farther than just patches. Lapel pins go on the left lapel, and when hung vertically from a wall the union goes to the viewers' left.