And the media isn't helping. The title of the article is "Google’s Search AI Says Slavery Was Good, Actually." It should be "Google’s Search LLM Says Slavery Was Good, Actually."
Unfortunately, people who grow up in racist groups also tend to be racist. Slavery used to be considered normal and justified for various reasons. For many, killing someone who has a religion or belief different than you is ok. I am not advocating for moral relativism, just pointing out that a computer learns what is or is not moral in the same way that humans do, from other humans.