I remember contemplating the conservation of mass when I was about 4. Except I didn't know what it was called. My mom gave us a piece of Double Bubble. My sister cut hers in half and declared she had more than me. I distinctly remember thinking how are the two smaller pieces bigger than the one big piece.
i guess maybe it depends on the person. i usually remember the moments when i learn things, including words. at the very least I'm pretty confident with whether i knew a certain word while i was in school, before, or after. and i would think a word like fabulous would be more memorable than usual.
I think you missed the part where he wasn't around when she was four years old. How does he know what she said when he left her to be raised by her mom and nannies?
I have trouble remembering things from highschool. But my younger sibling, they have vivid memories of stuff from pre-school onwards. And I know they're not making it up because, well, I was there and was 5 years older.
And many research has shown that long term memory starts forming at, coincidentally, around 4 years of age.
The mind is a wondrous place, and just because you or I can't imagine it happening doesn't mean it's not possible.