According to the researchers, a major source of the technology’s problems with kids and dark-skinned people comes from bias in the data used to train the AI, which contains more adults and light-skinned people.
this is misleading. if you read the study, you'll see the aggregate miss rate is much worse for dark skinned people and children, but that's weighted by the models that haven't been trained specifically for use with pedestrian detection. the pedestrian detection models, i.e. the ones people actually are going to use, were within 1% as accurate for skin tone, and were about 10% worse at detecting kids. since kids are already harder to see for human drivers, that seems much more similar to human performance.
the other thing it doesn't mention is how bad they are at detecting people in general. the best one misses people roughly 5% of the time, which seems pretty high for something that's supposed to be driving a car
Researchers ran more than 8,000 images through the software and found that the self-driving car systems were nearly 20% better at detecting adult pedestrians than kids, and more than 7.5% better at detecting light-skinned pedestrians over dark-skinned ones. The AI were even worse at spotting dark-skinned people in low light and low settings, making the tech even less safe at night.