The only way to explain it's faster could be because the case was making the phone overheat. The chipset is made to protect itself by lowering the clockspeed (calculations per second) to avoid damage by heat resulting in being slower at performing tasks.

So maybe the case was isolating the phone so much it overheated constantly and after removing the case it could breath again and keep its original speed.

If you had a chunky case then maybe a sleeker design could offer enough cooling and still some protection. Best of both worlds :)