In an ideal world, yes, we'd keep the carbon tax cranked. But fuck that. People with oil heaters would get off them if they could. If they're still on oil, there's probably a reason.
The current subsidies of heat pumps look like they're structured as loans. And they're complicated. Don't do that. Just install the damn heat pump for free, and call it a day.
I think they should get an exemption as it is literally keeping them alive.
I think a better solution would be to heavily subsidize heat pumps, resistance furnaces, and electric space heaters and make them more cost effective to use. People will switch to save money. Of course the cost of electricity has to be significantly cheaper than the oil.