House got hit by a tornado last night. Power for entire city will be out for a week. My homelab is still running strong 24 hours later!!!!

House/city got hit by a 115 mph wind gust, taking out power to most of the city last night.

Knocked down all of my trees, wrecked havoc on the city. Messed up roofs everywhere, and most importantly, no power for anyone!

https://www.koco.com/article/oklahoma-severe-storms-weather-tornado-hail-saturday/44233131

My city/utility estimates it may take up to a week to get power restored. But, thankfully, I have spent a lot of time preparing for this event.

As- my MAIN house batteries were only charged up to 50/60% when the wind hit- AND, I had a misconfiguration on my primary inverter causing them to shutdown at 20% (Rather then 10%), the first night, the entire house ran on battery from 11pm up to 7:20am. This includes, running the A/C, running my rack of servers. All of it.

I have a constant, 500w load from my servers. I have a optiplex micros in my kubernetes farm, I have a r730xd pushing 256g of ram, 32c/64t, Tesla P2 GPU, and a whopping 130TB of raw storage (before redundancy).

So- at 7:20am, the main inverter shuts down due to hitting its battery shut off limit. Of course, this means, my A/C and fan shuts off, causing me to wake up pretty quickly. The majority of the house is out- but, not my rack!

My rack is still plugged into my homemade 2.4Kwh UPS I built a few years back.

• https://xtremeownage.com/2021/06/12/portable-2-4kwh-power-supply-ups/

So, after getting up, grabbing my coffee, I went ahead and plugged in the generator, which got everything turned back on until the sun came back out. Once the sun came out, the steady 3-5kwh of solar PV power kept everything running, and put its extra juice back into the batteries for later.

During the day- the entire house, and rack of servers was able to run off of pure sunshine without issue.

Around an hour or ago, when the sunlight went away, I went ahead and plugged in the generator to get the batteries topped off for another night without grid power.

Home assistant automation yells at me when the batteries are full, to tell me to go and turn off the generator. When the batteries get low, it yells at me to go start the generator. (Literally- it talks to me via TTS).

That being said, I have 5 gals of gas in the generator, another 5 gals on standby. And, can always go and get more if needed. But- that should be enough to keep my entire lab running for the rest of the week. .. between generator power, and PV/Solar energy.

If- you are interested in an overview of how my solar setup works, its all documented here:

https://static.xtremeownage.com/pages/Projects/Solar-Project/

Edit- also, if https://xtremeownage.com/ and https://lemmyonline.com/ are still working- my Lab is still powered.