“This is basically what we were all worried about with Y2K, except it’s actually happened this time.”
What people were worried about with Y2K was nuclear weapons being launched and planes falling out of the sky. And it was nonsense, but bad things could have happened.
The good part is that the harm was mitigated for the most part through due diligence of IT workers.
This is similar to what would have actually happened if not for the dilligence of IT workers fixing the Y2K code issues globally. Uninformed people were worried about missiles and apocalyptic violence, but IT workers withdrew some cash and made sure not to have travel plans.
The difference here is that this was caused by massive and widespread negligence. Every company affected had poor IT infrastructure architecture. Falcon Sensor is one product installed on Windows servers. Updates should go to test environments prior to being pushed to production environments. Dollars to donuts, all of the companies that were not affected had incompetent management or cheap budgets.
Millions of man hours spent making sure Y2K didn’t cause problems and the only recognition they got was the movie Office Space.
There isn’t a single one of them who was working at that time I have spoken with who didn’t think Office Space was exactly the correct tribute
I’ll take it. I identified so hard with that movie. When I eventually die, I’ll do so knowing I’ve been seen.
I wonder if there would be any way to work it so that a dry concept like that could be made into a decent movie based on the actual events. They did it for Tetris.
Sure, but even the worst Y2K effects wouldn’t have had what lots of people were worried about, which was basically the apocalypse.
People who really should have known better were telling me that Y2K would launch the missiles in the silos.
We knew. However we knew there would be problems so we emphasized extremely unlikely scenarios to get the budgets to prevent the really annoying shit that might’ve happened.
We rarely disagree, but I’m gonna pull the “I work in the industry” card on you. A lot of hardworking people prevented bad things from happening whether big or small. We only look back at it as overblown because of them.
Are you really going to claim that we would have had a global thermonuclear armageddon if Y2K mitigation was a failure?
You’re focusing on the extreme unrealistic end of what people were worried about with Y2K, but the realistic range of concerns got really high up there too. There were realistic concerns about national power grids going offline and not being easily fixable, for example.
The huge amount of work and worry that went into Y2K was entirely justified, and trying to blow it off as “people were worried about nuclear armageddon, weren’t they silly” is misrepresenting the seriousness of the situation.
I literally said in my first comment:
The good part is that the harm was mitigated for the most part through due diligence of IT workers.
What more should I have said?
It’s not what more you should have said, but what less. It’s the “people were worried about nuclear armageddon” thing that’s the problem here. You’re making it look like the concerns about Y2K were overblown and silly.
No. I’m saying that something like today would have happened only it would have been much worse in that it couldn’t be fixed in the space of hours / days.
Sure, but that’s not what people were worrying about at the time, which was my point.
Work is borked so I get to paint Warhammer today.
Minis are for painting at unspecified times in the future, not now
My Mountain of Shame must be mined.
I love this phrase and I will use it.
I fully support your sacrifice o7
Be sure to post the results to the corresponding communities.
Meanwhile, friends at my old company run sites with CS and my current company doesn’t. I’m kicking back and having a great friday
So the hindsight is always 20/20 but was there like warning signs or red flags which should have been obvious this is going to happen or are you just lucky in hindsight?
Red flags? Yeah don’t use “security Software” that just increases your attack surface. Why the fuck would you want to install a root kit on your critical infrastructure?
The second one, as far as I can tell. But also, those calls are made above me and I have no insight into the decision-making. It could have been keen foresight by someone else.
Same. Had time for my trainees and used this for an extra learning session. :)
My office sent out this big message about people not being able to log in this morning. And I had absolutely no issues, and all of my tools are working. So I guess I’m stuck actually doing work.
Your work and their work, since they can’t log in.
Look at this “team” player hehe
Bro, why didn’t you lie 😭
Nothing like getting a full work day in before the office opens
Will this change how companies run their IT? Absofuckinglutelynot!
It kinda sounds like this ones more on the developers than sysadmins…
Well its really on the people deciding to use garbage software from some random, clearly incompetent company.
Literally one of the largest enterprise grade endpoint protection packages. This isn’t an issue of a bad sysadmin, or even developer, so much an issue bigger than the industry itself. Up until now, as far as I knew, crowd strike has been recommended as a solid choice for endpoint protection.
Who else are you going to trust? Fucking Symantec? Ask VMware how being owned by Broadcom is, then get back to me.
No one gives a shit about their job anymore because they have no reason to. I hate to sit here and chalk everything in the world up to late stage capitalism, but jfc if it doesn’t seem like the recurring theme from hell. Something tells me the guys who work at Crowdstrike are no different.
So was this Crowdstrike’s fuck up and not Microsoft’s?
Probably, but the issue is in the interface between Windows and the CrowdStrike software causing Windows to go into a crashing bootloop.
Closed source is great, I tell you. /s
It has nothing to do with closed source, this is entirely about a privileged application fucking around and not testing shit before pushing it globally. You can have the same issues with Linux too. My org has stated that our AV product is never to be installed on Linux machines because they hosed tons of machines years back doing something similar.
High privilege security applications are always going to carry this risk due to how deeply they hook into the OS to do their job.
That is true. An obvious failure is that the update that broke everything was pushed everywhere simultaneously.
That’s what has me confused. I haven’t even stepped into an office in 20 years so I have no modern experience there, but I would have thought that a company with such a massive user base would release things at different times based on region. Is it because with security based applications they don’t want to risk someone having time to exploit a new vulnerability?
Is it because with security based applications they don’t want to risk someone having time to exploit a new vulnerability?
Pretty much. Given how fast the malware scene evolves and implements day 1 exploits, and how quickly they need to react to day 0 exploits, there’s kind of an unwritten assumption (and it might actually be advertised as a feature) that security software needs to react as fast as possible to malicious signatures. And given the state of malware and crypto shit, it’s hard to argue that it isn’t needed, considering how much damage you’ll suffer if they get through your defenses.
That being said, this kind of a fuck up is damned near unacceptable, and these updates should have been put through multiple automated testing layers to catch something like this before this got to the end user devices. I could see the scale of it taking them out of business, but I also wouldn’t be surprised if they managed to scrape by if they handle it correctly (though I don’t see the path forward on this scale, but I’m not a c-suite for many reasons). Like I said above, we had an incident years back that hosed a bunch of our Linux boxes, but the vendor is still around (was a much smaller scale issue) and we even still use them because of how they worked with us to resolve the situation and prevent it from happening again.
Hard to tell, fake news running both of their names, looks like both?
I like how it’s the biggest IT issue and the best solution is to turn it off and on several times
They are saying “up to 15 times” now.
Ftfy: ‘Largest
ITWindows outage in history’I learned of the problems from the radio news on my way back home.
CrowdStrike, not Microsoft, is responsible. Let’s put blame where blame is due.
This could happen to any OS that has cybersecurity where permissions are needed at deeper levels to protect systems.
laughs in linux
Work was supposed to be slow today. D’:
