Lots of good answers here, but one I haven't seen is that some people have different value systems. They would be the ones that say "yes, human rights would be nice, but at what cost?"
Typically, as everyone here has pointed out, they value their own well being and comfort. "We can't end child slave labor because then a KitKat would cost $20." They might cite economic priorities, national or personal security, religious beliefs, or civic pride (see: China).
It always boils down to the same core belief: “It's OK for some humans to suffer and die in indignity.” The cruelty is the point. Yes, it's a different value system, but usually it's about putting something other than humanity above everything else. Be it money, religion or ideology, it's always about the idea that some material or conceptual object is more valuable than human life and dignity. The other face of that coin is dehumanization, which is the idea that, “yes all humans deserve basic rights, but did you see what they did? They obviously are animals who don't deserve rights.”
Another common argument I've heard is regarding crime and criminals. Some politicians in my country tout that you cannot arrest criminals and we spend a fortune maintaining jails as if they were 5 star hotels. This is of course not true, but it does rail up the tough on crime crew who believe the only reason crime is rampant it is because it somehow allowed by human rights.
Since human rights are international agreements, it makes for an easy scape goat for those who believe in crazy conspiracy theories