After almost half a century, I'm still doing it...

I knew a guy who did this and had to fly to Germany to fix it because he didn’t want to admit what he’d done.

Use cockpit by Red Hat. It gives you a GUI to make networking changes*, and will check if the connection still works before making the change. If the connection doesn't work (like the ip addresses changed), it will undo the change and then warn you. You can then either force the change through or leave it be.

*via NetworkManager only.

I have a failsafe service for one of my servers, it pings the router and if it hasn't reached it once for an entire hour then it will reboot the server.

This won't save me from all mistakes but it will prevent firewall, link state, routing and a few other issues when I'm not present.

I've done this kind of thing remotely in screen with ifdown eth0 ; sleep 10 ; ifup eth0 ;

I formated an OS drive by mistake last night, thought it was my flash drive...

Lol I've locked myself out of so many random cloud and remote instances like this that now I always make a sleep chain or a kill timer with tmux/screen.

Usually like:

./risky_dumb_script.sh ; sleep 30 ; ./undo.sh

Or

./risky_dumb.script.sh

Which starts with a 30 second sleep, and:

(tmux) sleep 300 ; kill PID

A decade and change ago, in a past life, I was tasked with switching SELinux to permissive mode on the majority of systems on our network (multiple hundreds, or we might have gotten above one thousand at that point, I don't recall exactly). This was to be done using Puppet. A large number of the systems, including most of our servers, had already been manually switched to permissive but it wasn't being enforced globally.

Unfortunately, at that point I was pretty familiar with Puppet but had only worked with SELinux a very few times. I did not correctly understand the syntax of the config file or setenforce and set the mode to ... Something incorrect. SELinux interpreted whatever that was as enforcing mode. I didn't realize what I had done wrong until we started getting alerts from throughout the network. Then I just about had a panic attack when I couldn't login to the systems and suddenly understood the problem.

Fortunately, it's necessary to reboot a system to switch SELinux from disabled to any other mode, so most customer facing systems were not impacted. Even more fortunately, this was done on a holiday, so very few customers were there to be inconvenienced by the servers becoming inaccessible. Even more fortunately, while I was unable to access the systems that were now in enforcing mode, the Puppet agent was apparently still running ... So I reversed my change in the manifest and, within half an hour, things were back to normal (after some service restarts and such).

When I finally did correctly make the change, I made sure to quintuple check the syntax and not rush through the testing process.

edit: While I could have done without the assault on my blood pressure at the time, it was an effective demonstration of our lack of readiness for enforcing mode.

Not SysAdmin but about a year into my first software engineer job I was working on the live DB in SQL without using BEGIN TRAN ROLLBACK TRAN.

Suffice to say I broke the whole system my making an UPDATE without a WHERE clause. Luckily we have regular backups but it was a lot of debugging with the boss before I realised it was me that caused the issue the client was reporting.

I was on-call and half awake when I got paged about a cache server’s memcached being down for the third time that night. They’d all start to go down like dominoes if you weren’t fast enough at restarting the service, which could overwhelm the database and messaging tiers (baaaaad news to say the least). Two more had their daemon shit the bed while I was examining it. Often it was best to just kick it on all of them to rebalance things. It was… not a great design.

So I wrote a quick loop to ssh in and restart the service on each box in the tier to refresh them all just in case and hopefully stop the incessant pages. Well. In my bleary eyed state I set reboot in the variable instead of restart. Took out the whole cache tier (50+) and the web site. First and only time I did that but that definitely woke me up. Oddly enough the site ran better after that for months as my reboots uncovered an undiscovered problem.

Every network engineer must lock themselves out of a node at some point, it is a rite of passage.

Why don't you use chained commands, or better yet simply create an alias that chains down/up, then use the alias instead?

I still prefer net-tools and use ifconfig eth0 up That ip mess I'd rather do without, and those funky UU device/interface names I wish them out of my system

By the way, what system/init/svc manager are you using? With 50y in your back, cron job to check if it is up and resetting it while you are away. You can always remotely cancel the cronjob ... but it will be a new mistake not the old one :)

I started on Irix and ultrix if you remember those, what would I know :)

This is why IPMI is so important.

It's not Unix, it's you.

A few months ago I accidentally dd'd ~3GiB to the beginning of one of the drives in a 4 drive array... That was fun to rebuild.

Good to know that in another 30 years, I will still be doing the dumb shit I've been doing for the last 20.

If it makes you feel any better, I did something just as infuriating a few years ago.

I had set up my home media server, and had finally moved it to my garage with just a power cable and ethernet cable plugged in. Everything was working perfectly, but I needed to check something with the network settings. Being quite new to Linux, I used a remote desktop tool to log in and do everything through a gui.

I accidentally clicked the wrong item in the menu and disconnected the network. I only had a spare ps/2 keyboard and mouse, and as the server was an old computer, it would crash if I plugged a ps/2 device in while it was running*.

The remote desktop stayed open but frozen, mocking me for my obvious mistake and lack of planning, with the remote mouse icon stuck in place on the disconnect menu.

*I can't remember if that was a ps/2 thing, or something specific to my server, but I didn't want to risk it

Did this once on a router in a datacenter that was a flight away. Have remembered to set the reboot in future command since. As I typed the fatal command I remember part of my brain screaming not to hit enter as my finger approached the keyboard. 🤦‍♂️

If I understand, this is less a complaint about how UNIX works and more a story about the consequences of careless mistakes.

This is why we have KVMs I guess. Though not every server has one of those.

At $DAYJOB, we're currently setting up basically a way to bridge an interface over the internet, so it transports everything that enters on an interface across the aether. Well, and you already guessed it, I accidentally configured it for eth0 and couldn't SSH in anymore.

Where it becomes fun, is that I actually was at work. I was setting it up on two raspis, which were connected to a router, everything placed right next to me. So, I figured, I'd just hook up another Ethernet cable, pick out the IP from the router's management interface and SSH in that way.
Except I couldn't reach the management interface anymore. Nothing in that network would respond.

Eventually, I saw that the router's activity lights were blinking like Christmas decoration. I'm guessing, I had built a loop and therefore something akin to a broadcast storm was overloading the router. Thankfully, the solution was then relatively straightforward, in that I had to unplug one of the raspis, SSH in via the second port, nuke our configuration and then repeat for the other raspi.

I was scared to move the cloud for this reason. I was used to running to the server room and the KVM if things went south. If that was frozen, usually unplugging the server physically from the switch would get it calm down.

Now Amazon supports a direct console interface like KVM and you can virtually unplug virtual servers from their virtual servers too.

There, but for the grace of god....

@[email protected] You're doing it wrong. Just setup a KVM behind your server. So then you never need to leave home again.

I hope you don't admin any mission critical servers. That's a first year mistake.

Edit:

Hi salty kiddos still making year one mistakes! Down voting this comment won't improve your skill set. You will get walked out at a serious enough outfit for doing something like that to a prod system.