It can accumulate into clothes and inside the washing machine. Over time it can start to smell and damage bearings and heating elements.
Those problems probably only come with constantly going over the recommended dose though.
Zeolites replace phosphates in detergent. Is that bad? Aren't phosphates more harmful in wastewater than zeolites?
As I understand it, some zeolites can be aluminum based, and there's some research to show a connection between breast cancer and some aluminum products (deodorant being the largest source). But from what I've read these connections are tenuous, and mostly related to aluminum in deodorant being absorbed and having estrogen-like effects in the body, and the relationship of estrogen to breast cancer.
Why, pray, do I not want to put zeolites in my washer?