If you do business on Microsoft computers with CrowdStrike software, you had one hell of a day today. Millions of people, and thousands of major businesses were impacted.
Major IT Outage Grounds Flights, Hits Banks and Businesses Worldwide
The Wall Street Journal reports Major IT Outage Grounds Flights, Hits Banks and Businesses Worldwide
A massive tech outage swept the globe, knocking out operations for banks, media companies and emergency services and forcing airlines to ground flights, exposing the fragility and interdependence of global digital technology.
A single update from cybersecurity-software company CrowdStrike (CRWD), a major provider of malware and virus protection to a large array of companies, caused outages for millions of users of Microsoft (MSFT) Windows devices worldwide. Major airlines including Delta Air Lines, United Airlines and American Airlines halted departures overnight.
The outage touched almost every industry. Multiple financial institutions, government entities and corporations reported tech issues. Some hospitals and school districts said computers were down. Still, financial markets were largely operating as normal, and many companies said systems were starting to be restored.
That one update from a single provider could plunge so many companies—from airlines’ check-in desks to consultants’ conference rooms—into a digital dark age serves as fresh warning of the world’s technological dependence.
One of CrowdStrike’s main services is called Falcon, which monitors a company’s machines for hacking attempts, viruses and other threats. The Austin-based company has about 29,000 customers and went public in 2019.
CrowdStrike told customers in a status update seen by The Wall Street Journal that the problem was with a software change it had pushed via Falcon out to clients’ computers. The company said its engineers had undone the change but clients would need to use a workaround to download a fix to affected computers.
Some affected users may be back up and running soon, but for others it could take weeks depending on the system in use, said Simo Kohonen, founder of Finland-based network security company Defused. “The fix CrowdStrike has given is quite manual and may be difficult, in some cases, to deploy at large scale,” he said.
At Britain’s National Health Service, the outage disrupted the appointment and patient-record system at most doctor’s practices—leading to handwritten prescriptions. In India, at least some passengers received handwritten boarding passes.
In the U.S., many 911 and nonemergency call centers were disrupted. In New York City, some screens showing subway arrival times weren’t working.
Visa said it was aware of reports of people unable to make payments. Some JPMorgan staff had trouble logging on. Charles Schwab said certain online systems may be unavailable.
More than 27,000 flights around the world were delayed and 2,800 were canceled by midmorning Friday, according to flight-tracking website FlightAware.
CrowdStrike Statement
Sorry
Sorry is what you say when you accidentally kick the cat. Sorry does not cut it. Expect lawsuits.
Blue Screen of Death – What to Do
As IT departments, Microsoft and CrowdStrike work to fix the outage, many people are still struggling to get their systems going and get rid of that “blue screen of death.”
Unfortunately, there may be no easy fix for most of us. It requires computer savviness or help from your IT department.
If you want to tackle this yourself, you can first try to reboot your computer. Microsoft says that sometimes helps but could require as many as 15 restarts before it takes.
If that doesn’t work, next check (or remember) if your hard drive has been encrypted with Microsoft’s BitLocker technology. If so, you’ll need to access your BitLocker recovery key to unlock it, says Grant Geyer, chief strategy officer of Claroty, a cyber defense company that recently raised $100 million in funding.
Once your hard drive is unencrypted, you can try to take your computer back to the version before the faulty update. Reboot your computer in safe mode and open up a command prompt in the search bar of your Windows screen. Type c:windowssystem32driverscrowdstrike and then type del C-00000291 to delete it. Hit Tab, then Enter and you should be back up and running, says Geyer.
If you’re not able to try the workaround yourself, it could take some time for your IT department to do it.
CrowdStrike Incompetence
I was in IT for over 20 years. We had a rollback plan for every update.
I was not in networking. I was in major bank applications on the technical end of things. My users were not end users but rather programmers who needed technical assistance to implement business needs.
Even a two-hour outage rollback was a disaster. We did changes on weekends, late at night, to minimize impact.
Answers Please
- Are system changes now that complex that they can’t be rolled back for weeks?!
- Did anyone test this?
The answer to number two is obvious. It’s either no, or the testers were grossly incompetent.
The answer to number 1: It’s idiotic to knowingly run software that cannot be rolled back for days, perhaps weeks if there are issues.
CrowdStrike offers protection against malware. Now we see that it needs to protect computers from its own updates.
Update, as of Monday july 22nd, Delta is still having systemwide problems. Certainly banks and others as well, but most are no longer reporting publicly problems as they don’t want to be seen as “weak” by their customers and the public at large.
as a reminder paper systems were robust and though they required lots of manpower (btw manpower actually buys products from companies, unlike computers) – paper systems were much more difficult to hack and front run.
So the three letter agencies, primarily led by the NSA worked to institute hackable digital systems throughout the world. Often you reap what you sow.
“Regression testing
Regression testing is re-running functional and non-functional tests to ensure that previously developed and tested software still performs as expected after a change. If not, that would be called a regression. Changes that may require regression testing include bug fixes, software enhancements, configuration changes, and even substitution of electronic components. Wikipedia”
In addition it is apparent that with the current state of software production, no one has explained to the current crop of system administrators that inside every update there are typically one or more NEW BUGS just waiting.
With friends like these….
YIKES!
Hey you. Get off of my cloud.
And they can’t wait to implement a 100% digital currency.
What happened to the “system restore” function? Is it gone from my windows laptop with office 365 also?
an antimalware company, writes a malware payload and sends it globally.
Shutting down computers worldwide, thus proving the viability of the packet for causing mayhem.
Any 2 bit hacker can now take this “update” and incorporate it into the payload of a virus or trojan etc.
They have saved hackers millions of manhours by designing a windows 10/11 payload.
Truly the name should be KlownStrike, not KrowdStrike.
You are forgetting to reference the biggest virus/malware of all: Windows
crowdstrike(sic) are the same folks who misled a nation into thinking the ruskies “hacked” the DNC 8 years ago after it was revealed that the DNC conspired to bestow the nomination upon Hillary. Binney was much more convincing than the crowdstrike folks in laying out what happened, using hard facts rather than inference. How that firm became an integral part of so many things makes one wonder…
https://jason-ross.medium.com/crowdstrike-and-russiagate-another-case-of-enormous-evidence-f53fd5fcc1c
Crowdstrike is run by Atlantic Council neocons. It made its name by overpromoting the idea that Russian hackers were a deadly threat to our IT infrastructure. It has never provided actual evidence that the so-called hack of DNC servers was a Russian state attack. This narrative was foundational to the spreading of a malicious lie about Donald Trump and his “Russian collusion”. It has now produced so-called protective software that is worse than any malware it was meant to prevent. Why do we allow Crowdstrike to exist?
Hopefully now they will be sued out of existence.
Trial Run, would be my best guess.
All we hear about is A Digital Currency, Balloons flying over our Country, story after story of electronic surveillance, Hackers breaking into Banks, private data, Cars, Supply Chain Disruptions, Identity Theft, Etc.
It’s coming…
Hey Mish, I manage an IT infrastructure group. We run Crowdstrike and spent the last 24hrs working on bringing our Windows systems up. Our Linux systems were fine. We have CS running on over 2500 endpoints. A flawed update rendered Windows clients and servers unavailable. Once an endpoint took the update it rebooted into the BSOD, which then required manual intervention. Basically, a single file had to be deleted, but with a BSOD it was challenging on a large scale. We have servers in our data center, servers on 2 cloud providers, local clients, and remote clients. Our operations were crippled but slowly recovered as each endpoint was addressed. Our VMware workloads were easy to repair, our local clients were addressed rather quickly but our Azure workload and remote clients have been challenging. They all require manual intervention.
Why use something like Crowdstrike? Organizations don’t often have time to react to malicious activity. Malware can spread throughout the world like wildfire and take control of organizations quickly. Crowdstrike has had a solid reputation so customers would rather allow the CS service to constantly update itself to protect large quantities of endpoints than face the consequences of an attack. Fighting bad actors is now real time, which is the biggest change from 20 years ago. Organizations no longer have the luxury of time to act and plan endpoint protection updates on designated weekends. Updates need to roll out constantly and quickly.
As far as a rollback plan, this is not as easy as it sounds. Any endpoint can be rolled back to the point of the last backup. Are all the clients backed up daily? Not our clients. Are all the servers backed up? Yes, but many are servers running databases and rolling back a database means lost transactions. Then there is the time to rollback. Manual intervention of a file deletion can often be faster than rolling back a database. So yes, systems can be rolled back with a restore but there may be consequences. We rolled a handful back but found it was easier to gain access to each endpoint and delete the single file.
I’m not a conspiracy theorist but I do not believe this was a result of an untested update. No way does something like this get released globally without thorough testing. I wouldn’t be surprised if they had a disgruntle employee responsible or Crowdstrike themselves were hacked.
The most naive system administrator I’ve ever heard.
“Organizations no longer have the luxury of time to act and plan endpoint protection updates on designated weekends. Updates need to roll out constantly and quickly.”
With that excuse you deserve what you got.
“No way does something like this get released globally without thorough testing.”
In other words you blew off your responsibility for testing by foisting it onto an external vendor.
Now who would ever have thought that system administration would be hard.
You are hereby sentenced to two years of actually having to write test scaffolding for the developers and reviewing regression test results.
Instead of pushing updates, software companies need to alert customers to download them.
Looks like DEI has struck once again, hiring people for just about anything other than being compident. things will change when the country goes through enough pain.
The word is spelled “competent” by persons eligible for DEI employment.
Let’s back off this particular instance and remember this: If you have a “security” update for your software, the moment you start your update you start a race. The race is between you and the bad guys. The finish line for you is when all of those systems are updated. The bad guys’ finish line is when they have examined your update to find the security flaw, and then exploited that flaw.
Life will get unpleasant when, if you produce a critical piece of technology, you are required to make it fail random ways at random times for some significant subset of your customers. Call that testing your real world for robustness.
And/or you are required to produce multiple versions of your system under “clean room” discipline so that no particular system becomes too critical, too important.
And/or you are forced not to sell to too many important customers. Call it anti-trust.
The world is filled with people who want failures to be predictable. Expect them to make such laws.
: If you have a “security” update for your software, the moment you start your update you start a race. The race is between you and the bad guys.
if your systems are that porous, you should un-network them to reduce your points of failure and systems cascading into chaos.
every computer doesn’t need a network interface, and every network doesn’t need to connect to the internet.
I own 50 computers, only this one I’m typing on is connected to the internets. that is deliberate and for what should be obvious reasons by now.
!!! Dude. For your personal use? I gotta tell my brother about you. I kid him about having a computer for each program he runs. And he’s only got a dozen or so.
Presuming you’re not including “computers” such as they were some decades ago. Counting that sort would get 50 before looking outside a baggie of audio book thumb drives in the glove box of my exoskeleton.
But thanks for describing your set-up. Gets me thinking. Maybe I should split in half one of my three computers – the one that has both post office and web site.
All this doesn’t change the fact that organizations need lots of connected computers.
Crowdstrike are the people who on December 5 2016 testified in Congress that there was NO evidence of Russians hacking the DNC email server. Which Adam Evil Monster Schiff then hid until May 2020. Had to remove some words that I cant use here, thought for a second I was commenting on ZeroHedge
Gee, is that the same chairman Adam Schiff that denied virtually all the Trump attorney’s witness requests during the impeachment hearings?
If it was Bill Gates instead of Henry Ford 120 years ago we’d still be riding horses.
Is there any connection between Trump assassination and today “event”. Did the CIA get an upper cut.
Remember (2020) the SolarWinds update that was used to breach all kinds of US government agencies. Password for updates was “solarwinds”, and that is how the hack installed code in the systems.
These people never learn. All of these big story computer glitches reveal people that should have expertise doing criminally stupid things with remote computers (Ukraine electrical grid) or public internet (Colonial Pipeline), etc., time and again.
First rule for IT administrators: Always make sure you can back out of any change.Second rule: One change at a time.
Anything else makes you a rank amateur.
The companies that implemented the update are apparently doing this on auto-pilot.
As for Crowd Strike, obviously didn’t test properly, and certainly with software this privileged in the system, you would expect testing & staggered update releases to avoid catastrophes like this. This is as bad as the Secret Service.
“This is as bad as the Secret Service.”
Are you saying it was staged?
No, I said initial staggering should be routine.
Whether the shooting was staged I do not know.
The degree of incompetence and the lies and inane excuses by the director do give pause.
Government agencies generally admit to incompetence only when the alternatives are worse. Incompetence remains the best form of plausible deniability. But incompetence also seems to be a societal trend.
I don’t think we’ll ever know, because investigation will fall to officially neutral competent authorities like the FBI and the DNI and the SS itself. Investigations such as this have never been delegated by committees of competent independent experts granted blanket subpoena powers.
Yes.
There is a movie about this.
It is called Idiocracy.
It’s only funny as a movie.
Took a family member to TIA today late afternoon to fly to South America. Kinda deserted. Must have hit the sweet spot.
CRWD closed down over 11% today.
Cue the news stories of peoples’ surgeries canceled (heaven forbid, no deaths because of it!) and other wrenches in plans and you have to think the stock will keep plumbing.
An amazingly complex but dangerously fragile online everything system with multiple single points of failure. What could go wrong? /sarc Perhaps the future will be simple reliable DUMB commerce over easily-broken “Smart” everything? Reading this post is like a little mini version of Forschten’s “One Second After”.
Entire Microsoft Network Goes Down After Greg Removes USB Device Without Clicking ‘Eject’ First | Babylon Bee
Maybe NSA errr I mean Crowd Strike has gone too far into hiring DEI coders … Helen Keller would have been a great software engineer.
Actually, the coder responsible was a white guy in a MAGA hat, hired because an old pal of the CEO.
DEI coders have been feverishly working all day to correct the error.
^^^^
Warning DEI coder alert!
Why do they push out all at once? Arogance? Small push out and wait for feedback. Than pushout every hour until all customers are updated.
I can only speak for myself, and never allow automatic updates. Usually wait for a few days before installing. Same thing with upgrading to new versions (beta’s) let someone else do the leg work on the release issues.
Being responsible and thinking things through is so blasé these days.
It is a new generation.
This is unbelievable! My how things have changed since I was an IT manager for a large company. We never allowed software updates to automatically roll out en masse. We tested and did our own deployment in stages. This is insane that these large institutions let vendors auto deploy software updates to mission critical platforms in this fashion.
My internet access showed signs of traffic congestion at times all the way until about a half hour ago beyond the blue screened systems.
Reasonable approach when your own IT team rolling out updates. In the rush-to-the-cloud mantra, the cloud provider rolls out updates, in this case Microsoft. Both are at fault.
This is/was a CrowdStrike caused problem that only affected MS Windows OS clients.
For once it wasn’t a directly MS initiated update that crapped out.
IT isn’t like it was 30 years ago or even 10 years ago. And I can say that as someone that’s been in the biz since 1984. Heavily interconnected systems via the “cloud” can ruin your day in a flash obviously. “Automatic” updates really shouldn’t be enabled as it removes your opportunity to test on your in house machines configurations.
Given the wide variations of hardware, versions of software, and levels of sysadmin competence these days, I can see how it happened. Never mind the actual skills of the widely dispersed and often outsourced ‘programmers’ today.
I’m just glad it wasn’t me that clicked on the “update” button to roll it out world wide.
What’s amazing about it that the previous version was already c.r.a.p. The only excuse for them would be if they filled the diversity quotas.
Being global affected over a billion people and millions of companies. My son couldn’t buy an auto part this morning as the various parts suppliers couldn’t access their inventory or make any sales even for cash. My worksite had all their computers down so they had no access to construction plans. This is in Sydney Australia, and that is just one country in over 100 and one city in tens of thousands. now apply that disruption to millions of companies and all their managers, employees and customers and the lost productivity must be in the hundreds of billions of dollars.
“I was in IT for over 20 years. We had a rollback plan for every update.”
You worked on (still sort of-) sane-complexity systems, You never rolled out an update to a billion computers, each and every one configured differently, in every way from hardware to the latest silly animated emoji fonts. While having to make sure your “security” software still allowed backdoors for at least half a dozen different clownish “State Actors” to do as they please. Becasue, like, you know, like ehh! The Hobgoblins are scariii! And, like stuff!!!
Also, back then: “Compute” was expensive. “Users” hence had to generally at least pretend to be sentient before being given wide access (COBOL, RPG and SQL were all intended for direct use by executives. APL was an actuary language.. Good luck with that, in today’s age of two numbers and a two-column bar graph worth of ‘Key Performance Metrics’ maxing out the empty skull space of that crowd; in between getting bailouts ). “Management”; both at computing vendors, and at clients; were not specifically selected, at all levels, for maximum stupidity.
IOW: Compared to today: “Yours” were the good ol’e days…. Very different from today, when the idiots are dumb enough to blindly “believe” in doozies like “self driving cars” and “AI” and who knows what else. Give them a rubber mallet, and they’ll find a way to ruin it and themselves. And when hurt, they’re too dumb to call the doctor. Instead calling an ambulance chaser. To “hold someone accountable.” Or something similarly dumb.
Of Note: Big Japanese industrials; who got their crash done-with-and-over; and who hence are to a much lesser degree beholden to credit-bubble-enabled rank retards making decisions about things which they’ll never comprehend and which really should be decided on by sober adults; STILL to a shockingly large degree run on “your” kind of systems. Those things; for a lot of core functionality; mostly work. For decades uninterrupted. Predictably. They do what they need to and; MOST IMPORTANTLY; not a billion other things which they don’t need to and which only serve to complicate things, especially security related things. Those systems haven’t been meaningfully improved upon for a whole host of core workloads. Replacing them with today’s clusterF of pointless and unconstrained complexity, is simply going backwards.
A wonderful thing about writing codes as objects is inheritance.
The disastrous thing about writing codes as objects is inheritance.
Regression testing is too expensive, until …
The initial product is good enough for most of us. If it ain’t broke, leave it alone.
More of a Crowdstrike for Windows update than a Windows update. Puts the computer in a endless reboot loop. Have to boot in safe mode and then delete the bad file. See https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/
What many of us expected when the Cloud became popular. As the more centralized control becomes the norm, we will have more giant errors.
Probably DEI employees. This is what happens when you don’t hire quality people.
More likely, management cutting corners with testing.
Employees do what they’re told.
It was the illegals.
Russia and or Iran
No way. It had to be climate change!
Updates these days are pushed out, often whenever a company like Crowdstrike wants. I think the problem with rolling back this update is that the process is manual, as opposed to the automatic nature of the update installation, and if you have 1000’s of PC’s affected, well, that will take weeks.
always have a few bucks on you. internet, sales terminals, wifi, cyber attacks happen frequently enough that carrying some green dollars in your wallet makes sense. Cash is king.
Unfortunately, cash will be a peasant soon enough once they push through digital cash.
After 35 yeard in IT … rollback plans were never an option. This seems to have had downstream affects as well. I pulled into a gas station this morning and they were only taking cash payments.
“CrowdStrike” was very aptly named… that’s exactly what they did!
How about “ClownStrike”?