Depleted means that the radioactive isotope is lower in concentration. It still is somehow radioactive (it's almost fine if not ingested) and still remains a heavy and toxic metal.

Everyone should worry. Depleted uranium will contaminate large crop production areas and later these grains will be sold all around the world. Everybody will eat some.