control net for xl

I find control net will sometimes not work at all with XL. Some times it will, but sometimes the image will just not be impacted by it at all. I don't know why. Is this me, or a bigger problem with control net?