Close relatives, yes, but not so much with cousins. And it wasn't until Darwin that they truly started to grasp what was going on.
It's easy to spot the recurring issue of diseased/sickly/malformed offspring with incest between siblings, or between parents and children, because the rate of birth defects is much higher. Fourfold, in fact.
Cousins didn't produce them nearly as much, so it wasn't an obvious enough trend. It's perfectly possible for first cousins to have healthy offspring more than half the time. In fact some studies have found the risk of genetic defeats is not much higher than a regular couple where the women is over the age of 40. Which is to say, it's low, but not low enough to ignore. There's also a lot of other factors involved especially when it comes to closed ethnic groups that tend to only reproduce amongst themselves.
He's the one that put forth the theories about the benefits of crossbreeding and the risks of inbreeding. He studied it in plants and wrote several books on it.
And yes, when his first daughter died at 10, the first of 3 that would die young, he worried a great deal it was because of his marriage to his cousin. He didn't have the facts or the data to prove it, but he had a very good inkling as to why 3 of his 10 kids died young and some of the ones that lived were infertile.
He was married and having kids before his theories had fully taken shape.
But he did know, or at least suspected, he'd made a mistake after the death of his first daughter. But he also had healthy children too, so he couldn't say anything unequivocally.
He ended up being a case study for his own theory, and he was well aware of it.