But that's more of an economic issue than it is a technical one. It's possible to process the less pure stuff, but it's cheaper to just get the good stuff to start with.
Silicon itself is certainly not rare (even if the purification process is somewhat exigent on the feedstock), but modern chips uses a lot of rare-ish elements in their makeup (hafnium for mosfet gates, gold for chip/package connexion, etc...).
I would be proud of being a warlock, but I'm more like the guy who speaks to the guy who speaks to the guy who speaks to the warlock who speaks to the stones.