But that's more of an economic issue than it is a technical one. It's possible to process the less pure stuff, but it's cheaper to just get the good stuff to start with.
Silicon itself is certainly not rare (even if the purification process is somewhat exigent on the feedstock), but modern chips uses a lot of rare-ish elements in their makeup (hafnium for mosfet gates, gold for chip/package connexion, etc...).