Refactoring is something that should be constantly done in a code base, for every story. As soon as people get scared about changing things the codebase is on the road to being legacy.
Been with a lot of codebases that had no unit tests at all and everyone was afraid to change anything because the QA process could take weeks to months.
The result is you have a codebase that ages like milk.
Today I removed code from a codebase that was added in 2021 and never ever used. Sadly, some people are as content to litter in their repo as they are in the woods.
Why do you need time to refactor? It is just part of the work you need to do and should be accounted for when doing any other work. IMO a big mistake people make is thinking refactoring is some separate thing they need permission to do. You don't, if you need to make a change in some area refactor it first to make it easier to accept your change, then add your change then refactor to clean up. This is not three separate tasks, just three steps in one task. You should be given enough time to do the whole task, not just part of it.
I guess I need to refactor for readability. What you just explained is the entire point of the comment I posted. Refactoring is part of the job. Don't give your manager a choice on whether or not it needs done.