It probably wouldn't have been annoying if you could just disable it. Getting that little notification with a number (that you compulsively want to clear down to zero) was always a pain.
Hm. I was quite lucky to only have one such encounter. It was some bot inviting people with a certain prefix in their username to a chat, posted one link and then was silent. The group of 50+ people then decided to repurpose the chat to be for an honorable committee, making plans to freeze the world (fighting global warming), throwing a nuke at a bright house of a (at that time) high-positioned person in the USA, overthrowing the danish government (idk why or who started it) and some more stuff.
It even spawned a subreddit :))
It was a nice chat group actually, though with time, many left. I think it might be barely over 20 people left now, and no one's said anything in a few months now.
I don't think that's possible to do using activity pub, but if you have a matrix account you can add it in your settings. When viewing the profile of someone who has a matrix account set on the web ui, there will be a button that says "send encrypted message". You should see it if you look at my profile page, for example.