Why have chestnuts been accused of being lazy?

Because all they do is roast on the open fire all day while we break our backs doing stuff.