I think it's because there are important, naturally occuring units of time that simply don't divide well - that is, the day and the year. Having it standardized to metric would still leave us with 1:365.24 conversion. Using metric time would require us to stop being metric beyond the day, or just have a cumbersome conversion number to talk about years.
On the other hand, things like weight, length, and temperature are completely arbitrary and there's no natural standard unit, so changing those to another completely arbitrary unit is easy.
Could you not divide a day into, say 10000, and just call that length of time 1 second? 100 seconds in an hour and 100 hours in a day? At least for the day to day clock.
If you redefine the length of a second, all sorts of bad things would happen because of the transition. And that's even before you remove the standard for what a second means because the length of a day will change over time.
As for weeks and months, check out https://en.m.wikipedia.org/wiki/International_Fixed_Calendar for one such idea.