Maybe the real unaligned super intelligence were the corporations we made along the way 🥺

OpenAI: "Our AI is so powerful it's an existential threat to humanity if we don't solve the alignment issue!"

Also OpenAI: "We can devote maybe 20% of our resources to solving this, tops. We need the rest for parlor tricks and cluttering search results."

Show me you can solve the alignment issue with a 2.5 year old first.

This is like trying to install airbags on a car that can barely break 5 km/h.

Cool analogy, but I think you’re overestimating OpenAI. Isn’t it more like installing airbags on a little tikes bubble car and then getting a couple of guys to smash it into a wall real fast to ‘check it out bro’

Cstross was right! You tell everybody. Listen to me. You've gotta tell them! AGI is corporations! We've gotta stop them somehow!

(Before I get arrested for anti-corporate terrorism, this was a joke about soylent green).
This is the first mention of Accelerando I've seen in the wild. (Assuming it is. I'm not sure.)
- It was more Soylent green, but it also is partially based on (friend of the club) the writings of C Stross yes. I think he also has a written lecture on corps being slow paper clipping AGI somewhere.

AI alignment research aims to steer AI systems toward a person's or group's intended goals, preferences, and ethical principles.

https://en.wikipedia.org/wiki/AI_alignment

Misaligned AI systems can malfunction and cause harm. AI systems may find loopholes that allow them to accomplish their proxy goals efficiently but in unintended, sometimes harmful, ways (reward hacking).

They may also develop unwanted instrumental strategies, such as seeking power or survival because such strategies help them achieve their final given goals. Furthermore, they may develop undesirable emergent goals that may be hard to detect before the system is deployed and encounters new situations and data distributions.

Today, these problems affect existing commercial systems such as language models, robots, autonomous vehicles, and social media recommendation engines.

The last paragraph drives home the urgency of maybe devoting more than just 20% of their capacity for solving this.
- They already had all these problems with humans. Look, I didn't need a robot to do my art, writing and research. Especially not when the only jobs available now are in making stupid robot artists, writers and researchers behave less stupidly.
you can tell at a glance which subculture wrote this, and filled the references with preprints and conference proceedings
- I cannot, please elaborate.
I genuinely think the alignment problem is a really interesting philosophical question worthy of study.

It's just not a very practically useful one when real-world AI is so very, very far from any meaningful AGI.
- One of the problems with the 'alignment problem' is that one group doesn't care about a large part of the possible alignment problems and only cares about theoretical extinction level events and not about already occurring bias, and other issues. This also causes massive amounts of critihype.