How do you clean your physical game controller?

I would use a damp wrung-out rag with just a tiny bit of soap in it too wipe it down. I might come back with clean water, still wrung-out.

Wooden toothpicks are great for the cracks because they are softer than the plastic.

I would save the IPA on a swab for any electric contacts, assuming you have removed batteries. If you can't remove the batteries, I would wait until you are having physical connection issues or have spilled something on it.