While it can be a hassle, reducing your attack surface is always recommended
Even though Bluetooth is designed to be secure, vulnerabilities can sneak in. We saw an example of that with the Flipper Zero being able to spam Apple devices, and while that was relatively harmless, that doesn't mean the next vulnerability will be.
I've been team Bluetooth off while not in use since my first smartphone. Now that I run GrapheneOS there's even a super handy Bluetooth setting to turn it off automatically if there hasn't been a device connected for the last 30s.
I recommended people turn off Bluetooth and WiFi when they're outside the house (and not intending to connect to a network) merely as a battery saving measure if nothing else. Several people in the thread told me to eff off and they couldn't be bothered.
Now we know modern iOS products don't even really allow you to turn off your radios without arbitrarily re-enabling them.