My birthday was the first day of school, so I was as young as possible - if my birthday was the next day I would have had to wait a year. I was the youngest person in every class I ever had except the occasional person who skipped a grade, and I was younger than some of them, too.
I'd say there's truth there. When you're in early elementary school, your coordination and stuff really changes rapidly, so I sucked at sports compared to kids in my grade who were nearly a year older than me. I ended up hating sports for that reason - still do. It was socially tough, too.
In the middle of elementary school, they told my parents that they thought I was bored and should skip a grade (I didn't find out about this until I was an adult). They decided that being yet another year younger than my classmates would just be too tough, so they declined. Probably the right decision, but I ended up being a crummy student until college.