Many people need to shift away from this blaming mindset and think about systems that prevent these things from happening. I doubt anyone at CrowdStrike desired to ground airlines and disrupt emergency systems. No one will prevent incidents like this by finding scapegoats.
When anywhere from 8.5 million to over a billion systems went down, numbers I've read so far vary significantly, still that's way too much failure for a simple borked update to a kernel level driver, not even made by Microsoft.
That means spending time and money on developing such a system, which means increasing costs in the short term.. which is kryptonite for current-day CEOs
Right. More than money, I say it's about incentives. You might change the entire C-suite, management, and engineering teams, but if the incentives remain the same (e.g. developers are evaluated by number of commits), the new staff is bound to make the same mistakes.
I strongly believe in no-blame mindsets, but "blame" is not the same as "consequences" and lack of consequences is definitely the biggest driver of corporate apathy. Every incident should trigger a review of systemic and process failures, but in my experience corporate leadership either sucks at this, does not care, or will bury suggestions that involve spending man-hours on a complex solution if the problem lies in that "low likelihood, big impact" corner.
Because likely when the problem happens (again) they'll be able to sweep it under the rug (again) or will have moved on to greener pastures.
What the author of the article suggests is actually a potential fix; if developers (in a broad sense of the word and including POs and such) were accountable (both responsible and empowered) then they would have the power to say No to shortsighted management decisions (and/or deflect the blame in a way that would actually stick to whoever went against an engineer's recommendation).
Edit: see my response, realised the comment was about engineering accountability which I 100% agree with, leaving my original post untouched aside from a typo that's annoying me.
I respectfully disagree coming from a reliability POV, you won't address culture or processes that enable a person to make a mistake. With the exception of malice or negligence, no one does something like this in a vacuum; insufficient or incorrect training, unreasonable pressure, poorly designed processes, a culture that enables actions that lead to failure.
Example I recall from when I worked manufacturing, operator runs a piece of equipment that joins pieces together in manual rather than automatic, failed to return it to a ready flag and caused a line stop. Yeah, operator did something outside of process and caused an issue, clear cut right? Send them home? That was a symptom, not a cause, the operator ran in manual because the auto cycle time was borderline causing linestops, especially on the material being run. The operator was also using manual as there were some location sensors that had issues with that material and there was incoming quality issues, so running manually, while not standard procedure, was a work around to handle processing issues, we also found that culturally, a lot of the operators did not trust the auto cycles and would often override. The operator was unlucky, if we just put all the "accountability" on them we'd never have started projects to improve reliability at that location and change the automation to flick over that flag the operator forgot about if conditions were met regardless.
Accountability is important, but it needs to be applied where appropriate, if someone is being negligent or malicious, yeah there's consequences, but it's limiting to focus on that only. You can implement what you suggest that the devs get accountability for any failure so they're "empowered", but if your culture doesn't enable them to say no or make them feel comfortable to do so, you're not doing anything that will actually prevent an issue in the future.
Besides, I'd almost consider it a PPE control and those are on the bottom of the controls hierarchy with administrative just above it, yes I'm applying oh&s to software because risk is risk conceptually, automated tests, multi phase approvals etc. All of those are better controls than relying on a single developer saying no.
If you were a developer that knew you were responsible for developing ring zero code, massively deployed across corporate systems across the world, then you should goddamned properly test the update before deploying it.
This isn't a simple glitch like a calculation rounding error or some shit, the programmers of any ring zero code should be held fully responsible, for not properly reviewing and testing the code before deploying an update.
Edit: Why not just ask Dave Plummer, former Windows developer...
If you system depends on a human never making a mistake, your system is shit.
It's not by chance that for example, Accountants have since forever had something which they call reconciliation where the transaction data entered from invoices and the like then gets cross-checked with something else done differently, for example bank account transactions - their system is designed with the expectation that humans make mistakes hence there's a cross-check process to catch those.
Clearly Crowdstrike did not have a secondary part of the process designed to validate what's produced by the primary (in software development that would usually be Integration Testing), so their process was shit.
Blaming the human that made a mistake for essentially being human and hence making mistakes, rather than the process around him or her not having been designed to catch human failure and stop it from having nasty consequences, is the kind of simplistic ignorant "logic" that only somebody who has never worked in making anything that has to be reliable could have.
My bet, from decades of working in the industry, is that some higher up in Crowdstrike didn't want to pay for the manpower needed for the secondary process checking the primary one before pushing stuff out to production because "it's never needed" and then the one time it was needed, it wasn't there, thinks really blew up massivelly, and here we are today.
sure it is the dev who is to blame and not the clueless managers who evaluate devs based on number of commits/reviews per day and CEOs who think such managers are on top of their game.
[…] THE OFFERINGS AND CROWDSTRIKE TOOLS ARE NOT FAULT-TOLERANT AND ARE NOT DESIGNED OR INTENDED FOR USE IN ANY HAZARDOUS ENVIRONMENT REQUIRING FAIL-SAFE PERFORMANCE OR OPERATION. NEITHER THE OFFERINGS NOR CROWDSTRIKE TOOLS ARE FOR USE IN THE OPERATION OF AIRCRAFT NAVIGATION, NUCLEAR FACILITIES, COMMUNICATION SYSTEMS, WEAPONS SYSTEMS, DIRECT OR INDIRECT LIFE-SUPPORT SYSTEMS, AIR TRAFFIC CONTROL, OR ANY APPLICATION OR INSTALLATION WHERE FAILURE COULD RESULT IN DEATH, SEVERE PHYSICAL INJURY, OR PROPERTY DAMAGE. […]
It's about safety, but truly ironic how it mentions aircraft-related twice, and communication systems (very broad).
It certainly doesn't impose confidence in the overall stability. But it's also general ToS-speak, and may only be noteworthy now, after the fact.
That's just covering up, like a disclaimer that your software is intended to only be used on 29ᵗʰ of February. You don't expect anyone to follow that rule, but you expect the court to rule that the user is at fault.
Luckily, it doesn't always work that way, but we will see how it turns out this time
Lawful Masses with Leonard French covered this yesterday. He is a copyright attorney. He starts the video with the opinion that the ToS wouldn't protect CrowdStrike.
Yeah exactly. You'd think they'd have a test suite before pushing an update, or do a staggered rollout where they only push it to a sample amount of machines first. Just blaming one guy because you had an inadequate UAT process is ridiculous.
I hope this incident shines more light on the greedy rich CEOs and the corners they cut, the taxes they owe, the underpaid employees and understaffed facilities, and now probably some hefty fines, as just a slap on the wrist of course..
You can bet that they have a KPI that says they can deliver a patch in under 15m; that can preclude testing.
Although that would have caught it, what happened here is that 40k of nuls got signed and delivered as config. Which means that unparseable config on the path from CnC to ring0 could cause a crash and was never covered by a test.
It's a hell of a miss, even if you're prepared to accept the argument about testing on the critical path.
(There is an argument that in some cases you want security aystems to fail closed; however that's an extreme case - PoS systems don't fall into that - and you want to opt into that explicitly, not due to a test omission.)
That's the crazy thing.
This config can't ever been booted on a win10/11 machine before it was deployed to the entire world.
Not once, during development of the new rule, or in any sort of testing CS does.
Then once again, never booted by MS during whatever verification process they (should) have before signing.
The first win11/10 to execute this code in the way it was intended to be used, was a customer's machine.
The simplest, least effort thing that could have prevented the scale of issues is not automatically installing updates, but waiting four days and triggering it afterwards if no issues.
Automatically forwarding updates is also forwarding risk. The higher the impact area, the more worth it safe-guards are.
Testing/Staging or partial successive rollouts could have also mitigated a large number of issues, but requires more investment.
The update that crashed things was an anti-malware definitions update, Crowdstrike offers no way to delay or stage them (they are downloaded automatically as soon as they are available), and there's good reason for not wanting to delay definition updates as it leaves you vulnerable to known malware longer.
Four days for an update to malware definitions is how computers get infected with malware. But you're right that they should at least do some sort of simple test. "Does the machine boot, and are its files not getting overzealously deleted?"
Reading between the lines, crowdstrike is certainly going to be sued for damages, putting a Dev on the hook means nobody gets - or pays - anything so long as one guy's life gets absolutely ruined. Great system
Crowdstrike CEO should go to jail. The corporation should get the death sentence.
Edit: For the downvoters, they for real negligently designed a system that killed people when it fails. The CEO as an officer of the company holds liability. If corporations want rights like people when they are grossly negligent they should be punished. We can't put them in jail so they should be forced to divest their assets and be "killed." This doesn't even sound radical to me, this sounds like a basic safe guard against corporate overreach.