Power outage worries
Power outage worries
Hi fellow hosters!
I do selfhost lots of stuff, starting from the classical '*Arrs all the way to SilberBullet and photos services.
I even have two ISPs at home to manage failover in case one goes down, in fact I do rely on my home services a lot specially when I am not at home.
The main server is a powerful but older laptop to which i have recently replaced the battery because of its age, but my storage is composed of two raid arrays, which are of course external jbods, and with external power supplies.
A few years ago I purchased a cheap UPS, basically this one: EPYC® TETRYS - UPS https://amzn.eu/d/iTYYNsc
Which works just fine and can sustain the two raids for long enough until any small power outage is gone.
The downside is that the battery itself degrades quickly and every one or two years top it needs to be replaced, which is not only a cost but also an inconvenience because i usually find out always the worst possible time (power outage), of course!
How do you tackle the issue in your setups?
I need to mention that I live in the countryside. Power outages are like once or twice per year, so not big deal, just annoying.
I replace the batteries in my UPS every 18 months, and don't try to outlast power outages.
I have everything configured to shut down if the power goes down and stays down more than 5 minutes, which is ~20% of the maximum rated runtime. (I'm using repurposed desktop hardware that loves it's watts as a home server.)
I picked the low number for the reasons you've outlined: even if the battery is severely degraded, it's probably not THAT severely degraded and it's a safe time span to ride out short hiccups, but still well under the runtime limits so that a safe shutdown can happen.
That and I've noticed that, typically, if the power is down for 5 minutes it's going to be down for way longer than 5 minutes, so it doesn't matter and I'm not going to have enough batteries to outlast the outage.
Plus lead acid batteries hate being discharged at all. But deep charge cycles are really awful for them.
That said changing batteries every 18 months is a little excessive. Unless your UPS is overcharging the shit out of them they should last way longer than that.
Little bit of A, little bit of B.
I probably go through at least one full discharge cycle a month, if not more because the power around here suuucks. (The NAS goes down, but I leave the network gear up until the UPS dies, because fuck it, why not.)
It's also a ~10 year old UPS that likes to eat a $25 battery every 18 or so months so I just haven't really had any justification to replace the whole thing yet since there's an awful lot of $25 batteries in a new UPS.
The batteries should not degrade that fast. Get a name brand like Eaton or APC or something like that similar to a Power supply, cheap devices can do harm to your expensive devices. Have a couple of setups where the batteries (lead acid) are 5+ years old and they work ok.
The batteries should not degrade that fast.
For real!
I use several refurbished APC UPS', and also use third-party batteries (from the company that refurbishes the UPS') and it's been trouble-free for like 10 years. I replace batteries, it seems, every 4-5 years and only when the self-test says to replace it.
Never had a problem with data loss due to the UPS failure.
APC is garbage. Get Eaton. Lead acid batteries should last 3-5 years.
APC makes some good stuff. It’s just all their consumer stuff is garbage.
If you can get a rack mount unit from a data center those are real nice. Just don’t try to buy them brand new.
Replacing the UPS batteries is required maintenance.
Compare the cost of new batteries to a new UPS and realise it's the cheaper option.make sure it's configured for clean shut downs before your battery runs out, auto power up on restoration, and hope it doesn't happen. you will eventually have an outage that outlasts your batteries.
I have a large string of batteries from an old telco office, that runs my rack for 14hrs (calculated, I shut everything down around this time) and that did not last for the 2-3 day outage we had after a storm. Without a generator, you will inevitably have an outage, but if you are prepared, then you can mitigate any damage. use NUT if you need to shutdown or power multiple devices from one monitored UPS
A UPS is really just for brief interruptions, and to bridge the gap until the generator comes on for extended interruptions. If power is that bad where you live, and uptime is that important, get a generator that comes on automatically when power goes out. Or solar panels and a deep cycle battery array or something.
I was able to adapt my ups to use lithium ion batteries. I bought 2 giant ones that are used for rv solar systems. I tried using the other batteries and even when sitting by idle they loose their effectiveness.
As others have said, changing UPS batteries is required maintenance, and I agree 18-24 months is the typical service life for even high-end UPSs. However, you may want to look into LiFePO4 based UPSs, which can handle many more charge-discharge cycles and often have 5-year warranties. More expensive and potentially not as recyclable as lead acid batteries, but maybe appropriate for your use case.
If your batteries are dying that fast, you're doing something wrong.
You're saying that data centers are replacing batteries constantly...just imagine the labor costs on that (and the down time), not even considering the material cost.
I work in enterprise, and have never heard such a thing. Even my friends in SMB routinely replace UPS's at the 5-year mark, the same time they replace servers. They rarely replace batteries, at all, it's so rare that it's notable.
You're saying that data centers are replacing batteries constantly...just imagine the labor costs on that (and the down time), not even considering the material cost.
I’m the tech doing the battery replacements. The big boy UPSes are typically a 3-5 year replacement cycle. Something like this:
(I just picked the last one on my phone so not a great picture, they’re about the size of a small refrigerator)
On rack mount and desktop style UPSes 18-36 months isn’t unreasonable. Some of the smaller UPSes, like APC 750s, go through batteries even faster. My personal theory is that they just get and stay too hot.
There is typically zero downtime while servicing any of them, every critical system has redundant power supply and battery replacements usually don’t interrupt power output anyway. It would take multiple failures to cause any sort of significant downtime, and if it would, we just do them during scheduled downtime.
If you drain the UPS past 50% it really kills the life of the batteries.
In the Enterprise you probably have generator backup. So the UPS only need to keep things running for 30 second to a minute. So that never happens.
Two UPSs? Or better 3?
Tbh that is the coolest setup I've is the OpenCompute rack with 3 UPSs powering the DC rail for the rack. Otherwise two UPS for servers with two redundant power supplies.
How do you monitor your setup now, have you/can you use NUTS?
My old server is a HP Microliant.
What I love about it, is it turns back on automatically after a power out.
I don't know what magic is in there, but the power goes out, as it can at times, and when power is restored, the old girl just starts up again.
I'm travelling at the moment, and have it at my parents house under a desk. It's been there over 1.5 years and haven't had to have anyone physically touch it.
When it eventually dies, the next machine will definitely have to have that feature.
That's often a BIOS setting.
Any PC can do that, it's called "status after power off" or something like that.