I am fascinated by a story of how a Microsoft engineer discovered a major, heavily disguised, backdoor security breach that was years in the making, and nearly implemented.
Background
Hidden in a widely use compression utility was a software backdoor that would allow someone remote access to entire systems.
This was a multi-year endeavor by user named Jia Tan, @JiaT75 who gained trust over many years. His account is now suspended everywhere.
HackerNews has this interesting snip.
Microsoft security researcher Andres Freund has been credited with discovering and reporting the issue on Friday.
The heavily obfuscated malicious code is said to have been introduced over a series of four commits to the Tukaani Project on GitHub by a user named JiaT75.
The Long Game
These opensource projects are volunteer work. They pay nothing.
The person normally responsible for the code, Lasse Collin (Larhzu), maintained the utility since 2009 but was suffering burnout.
Jia Tan started contributing in the last 2-2.5 years and gained commit access, and then release manager rights, about 1.5 years ago.
Backdoor Uncovered in Years-Long Hacking Plot
Much of this story is extremely geekish and difficult to understand. An article on Unicorn Riot is generally readable.
Please consider Backdoor Uncovered in Years-Long Hacking Plot
A fascinating but ominous software story dropped on Friday: a widely used file compression software package called “xz utils” has a cleverly embedded system for backdooring shell login connections, and it’s unclear how far this dangerous package got into countless internet-enabled devices. It appears the persona that injected this played a long game, gaining the confidence of the legitimate main developer, and thus empowered to release new versions themselves.
Andreas Freund reported this Friday morning on an industry security mailing list, leading many experts to spend the day poking under rocks and peering into the abyss of modern digital insecurity: “The upstream xz repository and the xz tarballs have been backdoored,” Freund wrote. It cleverly pokes a hole in the SSH daemon (sshd), which is essential to modern-day computing at the most fundamental level.
The risks if this hadn’t been discovered were extreme: as noted expert @thegrugq put it: “The end game would be the ability to login to every Fedora, Debian and Ubuntu box on the internet. If it isn’t a state actor it should be…”
Cryptographer Filippo Valsorda said, “This might be the best executed supply chain attack we’ve seen described in the open, and it’s a nightmare scenario: malicious, competent, authorized upstream in a widely used library.”
The problem was uncovered after Freund noticed that the new version slowed down their PostgreSQL database tests, and they started debugging why this happened. It turns out the backdoor causes a tiny but noticeable slowdown in performance, a big win for picky benchmarking types everywhere.
As Minneapolis security expert Ian Coldwater noted, “Open source maintainer burnout is a clear and present security danger. What are we doing about that?”
This June 2022 message from the original developer confessing to burnout illustrates how Jia Tan gained control over the software:
“I haven’t lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. Recently I’ve worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we’ll see.
It’s also good to keep in mind that this is an unpaid hobby project.
Anyway, I assure you that I know far too well about the problem that not much progress has been made. The thought of finding new maintainers has existed for a long time too as the current situation is obviously bad and sad for the project.
A new XZ Utils stable branch should get released this year with threaded decoder etc. and a few alpha/beta releases before that. Perhaps the moment after the 5.4.0 release would be a convenient moment to make changes in the list of project maintainer(s). Forks are obviously another possibility and I cannot control that. […]”Lasse Collin, xz-devel mailing list, June 8, 2022
Some observers suspect the personas badgering Collin by email may have also been sockpuppets trying to shake control away from him. In a detailed report ars technica warned that even older versions could have security problems since the bad actor made many binary test file changes over the years.
Backdoor Story Unfolding Now
Upstream Backdoor
“Very annoying – the apparent author of the backdoor was in communication with me over several weeks trying to get xz 5.6.x added to Fedora 40 & 41 because of it’s “great new features. We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added). We had to race last night to fix the problem after an inadvertent break of the embargo.”
“He has been part of the xz project for 2 years, adding all sorts of binary test files, and to be honest with this level of sophistication I would be suspicious of even older versions of xz until proven otherwise.”
USA Security Alert
The US Cybersecurity & Infrastructure Security Agency (CISA) issued an alert on a Supply Chain Compromise Affecting XZ Utils Data Compression Library, CVE-2024-3094
CISA and the open source community are responding to reports of malicious code being embedded in XZ Utils versions 5.6.0 and 5.6.1. This activity was assigned CVE-2024-3094. XZ Utils is data compression software and may be present in Linux distributions. The malicious code may allow unauthorized access to affected systems.
Industry-Wide Reckoning Needed
“I really hope that this causes an industry-wide reckoning with the common practice of letting your entire goddamn product rest on the shoulders of one overworked person having a slow mental health crisis without financially or operationally supporting them whatsoever,” commented Mastadon user @glyph.
Here is an Interesting Timeline on how this was nearly implemented.
Here’s the Hero
Wow, just wow.
We were perhaps days away from this code being implemented.
link to bleepingcomputer.com
“PLEASE IMMEDIATELY STOP USAGE OF ANY FEDORA 41 OR FEDORA RAWHIDE INSTANCES for work or personal activity,” Red Hat warned on Friday.
Hi Mish thanks for sharing the story I wrote for Unicorn Riot .. I would just say, it was already in the wild in “testing” distributions, just fortunately was not on the “stable” (in Debian’s case).
You are welcome
Thanks for sharing
The very nature of proprietary software allows this to happen.
Not really. This was more of a problem of open source software, remote developers no one knows personally, etc. Both modes of development have risks, though.
A company that makes billions of dollars with high profit margins is too cheap to pay a few guys to program? If management wants to cut corners to get their million dollar bonuses, why not look at the human resource department and DEI.
This isn’t a Microsoft program, it is open source Linux. People work on this stuff in their spare time as hobbiests, even though it gets incorporated into a lot of critical systems..
MS has plenty of paid programmers.
For most widely used software, incorporating open source components for lots of standard functionality is just an overall better and more reliable way. If this had been an internal closed source project, many fewer eyes would have ever looked at it. Hence it would never have been discovered. Also; even if a vulnerability had been found in the code MS had written and was using; similar vulnerabilities could still be around in everybody elses software which also incorporated a similar library/similar functionality.
I suppose a good case could be made for paying people to very seriously vet all libraries, all open source code, someone as big and central as MS decides to rely on. Despite “everyone” being theoretically being able to read, hence vet, open source libraries; in practice very few do.
It’s complex stuff, though. Software is how “all” things of any complexity is specified and encoded today. The complexity of even the simplest products, are far beyond what is possible to deterministically verify.
And almost all currently popular technologies, descend from an era where “functionality” and “performance” was what was valued. Not resilience against malicious attacks. Example: There is no good reason why a device for reading and commenting on blogs, needs a camera, a microphone, gps etc. Nor why a device used to control a CNC router; nor perform inventory lookup for a warehouse; needs even a fraction of the functionality bundled with the Windows or Linux or Android OSes it is built on. Nor even why it needs access to the internet.
Heck; there’s not any good reason why extremely complex rendering engines; nor any ability to run active content, nor load content from multiple sources; are needed to operate a discussion board nor blog.
The biggest security threat, is simply that almost all applications are executed by far-too-complex-and-general-purpose devices: If all you need to do is pay someone face to face; a wad of paper in your pocket is an awful lot less likely to get hacked by some dude in Russia, than some weird “P to P” payment “app.”
It scares me whenever a field app engineer or a consultant tells me to find answers in the product’s community group or minimally regulated website. Even though these groups of collaborators have their hearts in the right place, They don’t have code reviews and independent verification of their work. In the search for faster, cheaper software development, engineers go to user groups and download code. They are supposed to test it, but put way too much trust in it. Human nature makes humans vulnerable to malicious people and organizations.
Search – ‘Bill Binney’
Lass Collin, the first and the last volunteer standing, got tired. He left the job, to put the fire, in the hands of an arsonist. Where is Satya Nadella fire department ==> smoking Hash.
What kind of management MSFT has.
Not a Microsoft program..it is Linux.
“Fedora, Debian and Ubuntu box on the internet.”
These are all Linux-type operating systems, correct? So this is not about Windows, but the non-corporate (Mac, Windows) alternative. Darn it! I wanted to get a Linux-based machine to prevent to type of thing.
I guess the blob has its fingers in everyone’s pies. This is called a police state.
Macs don’t use Linux.
I do use Linux, so this is a concern. Good thing someone caught it.
OS X may be vulnerable.
”
For the curious, if you run the command: xz –version on Linux, OS X or other Unix-based computers, it will report if the versions linked to malicious activity are present”
link to unicornriot.ninja
Apple products are already subservient to the blob. I wouldn’t own any such product.
Look into Qubes.
It’s Linuxes. Running on top of a virtual machine “hypervisor.” (The hypervisor is orders of magnitude less complex than most OSes. Hence there are fewer places for vulnerabilities to hide.) LinuxES, as in plural. Which means you can run each application, or keep unrelated roles, in a separate Linux VM. Such that if one gets compromised, the others are pretty well fire walled away.
It also benefits from focus on making non-persistent, “disposable”, VMs usable. Such that even if you should experience a breach, everything you have done in that disposable VM is erased, and you again start fresh and clear, next time you start it up. Over time, you recognize that less and less of the tasks you do, really needs to persist arbitrary data between invocations.
Qubes used to be a bit of a bear to install and use. As well as a resource hog. But on (higher end…) modern hardware, it performs well. And it’s less finicky about what it needs to run these days.
Edward Snowden at least used to speak highly of it, and he is someone who has rational reason to be more paranoid about such things than most.
For phones and tablets, look into the Graphene OS version of Android, on Pixel devices. Which are also designed with more of a focus on resilience and privacy than is common.
Stuki,
Great info! i’ll check it out.
It’s more secure, but a PITA to setup and slows things a lot.
So, what is the solution to this problem with open source software? How do you prevent “bad actors” from exploiting open source? How do you monitor all the updates for malicious code?
Currently we are relying on what? The hope that other developers will “catch” these bugs?
Yes, open source relies on many eyes being able to examine the source code on demand. Microsoft and Apple only have a few people able to review the source code. A foreign agent or gang member could infiltrate their system too and there would be fewer people to catch it.
Risk management principles should be applied even to open source, and esp. by large commercial outfits that use the open source components. They should put more dollars and developers into testing key components that they re-use, or to make sure the ‘community’ of testers are diverse and vetted enough to catch such bugs. Perhaps a better QC/testing framework should be involved, even if it, too, is open.
paraphrasing:
the only thing that stops a bad guy with a code is a good guy with a code.
A.I. to the RESCUE!
What worries me is what about all the other bad actors that have already inserted such code into telecom systems all over the world?
This is a Linux issue. How many people does it affect?
An incredible number of data centers and industrial applications where the $25-50 bulk cost for Windows is not worthwhile plus Windows has too much overhead and instabilities.
Ever heard of Red Hat?
NYSE: RHT
It could affect anyone that loads a web page from a server in the cloud, the majority of which are on Linux.
I.E. just about everyone is possible.
But not probable.
This is one instance of a broader problem with our entire economic and political system. The people who do the real work that needs to get done to keep society functioning are being pushed harder and harder. The ‘bloat’ of government and government-sponsored proxies such as the DEI crowd are sucking away more and more of the money and resources that are available. The people at the bottom can no longer repair things faster than the people at the top are breaking them.
This leaves little time and few resources for the overworked and overburdened ‘grunt’ workers to do a thorough job. As a result, the entire system is fragile and inflexible, and whenever any stress is applied from random events or malicious actors, things are breaking instead of bending.
In this instance, one guy had just barely enough time to do his job and stop this bad actor. That’s kind of frightening – one good guy was the last line of defense, and he stopped the bad guy just a few days before he won. There are too many people stealing slices of the economic pie, and too few people baking more pies.
So, this open source problem is because of government?
The solution then is to eliminate government, right?
Then the problem will magically disappear.
Got it.
I think he means less focus on bullshit in general, and more focus on merit and actual productivity.
This guy right here is a genius on all things internet security if you want to read more:
link to schneier.com
A very recent article of his on secure voting systems recommends (you’ll never believe!) paper ballots.
Right on! I thought I was reading Bruce for a sec on this one. Good summary from Mish.
Much ado about nothing. This on the other hand isn’t.
link to newsweek.com
Thanks. I though Newsweek went under in the ‘90s.
Some people just want to watch the world burn. Not me.
In English, please?
What happens when AI takes over…will we ever know? Will one AI system collude with others? My brain is hurting.
The only winning move is not to play.
That means GET OFF LINE NOW, right C.O.?