Report - How Andres Freund saves the Internet
The backdoor had been planted over 2.5 years by a nation-state actor who earned the trust of the maintainer, contributed legitimate code, and then injected malware into the build process itself.
March 2024. Andres Freund, a Microsoft engineer and PostgreSQL developer, is working from home when he notices something odd.
His SSH logins are taking 500 milliseconds longer than normal.
Most engineers would ignore this. Network latency. A busy server. Maybe restart the daemon and forget about it.
Freund didn’t. He broke out the profiler and debugged it.
He traced the micro-delay to liblzma. A ubiquitous compression library used by OpenSSH (via systemd).
He found that recent versions contained a backdoor so sophisticated that it had bypassed
Automated security scanners
Code reviews from major Linux distributions
Penetration testing
Static analysis tools
The “many eyes” of the open-source community
The backdoor had been planted over 2.5 years by a nation-state actor who earned the trust of the maintainer, contributed legitimate code, and then injected malware into the build process itself.
If Freund had ignored that 500ms delay, every major Linux distribution would have shipped SSH servers with a pre-installed, pre-authentication remote access backdoor.
Every CISO in the world relies on million-dollar security budgets, automated scanners, and penetration tests. A backdoor that took 2.5 years to plant was caught by one engineer who thought, “Huh, that CPU spike is weird.”
This is the story of why your security budget missed what one curious engineer caught by accident. And why “free” open-source software might be the most expensive dependency in your infrastructure.
The Myth
Let’s challenge the core religion of modern software engineering.
Linus’s Law states
“Given enough eyeballs, all bugs are shallow.”
This has been gospel in the tech industry for 30 years. It’s why enterprise companies confidently build trillion-dollar infrastructures on open-source software.
The assumption is that open source is inherently more secure than proprietary code because anyone can audit it. Bad actors can’t hide in plain sight when millions of developers are watching.
Here is the reality of XZ Utils
Downloads - 1.4 million per day.
Usage - Core dependency for every major Linux distribution (Debian, Ubuntu, Fedora, Red Hat).
Criticality - Required for SSH authentication on virtually every server on the internet.
Maintainers - ONE unpaid volunteer (Lasse Collin).
Annual Budget - $0.
Last Security Audit - Never.
According to the Open Source Security and Risk Analysis (OSSRA) report, 87% of commercial applications contain open-source components.
Yet, less than 3% of organizations perform regular security audits of their dependencies, and the average application relies on 500+ transitive dependencies.
Linus’s Law assumes the eyes are looking. But nobody audits compression libraries. Nobody reviews arcane build scripts. Nobody sponsors the maintainer working nights and weekends for zero pay.
Your production infrastructure runs on code that hasn’t been reviewed since it was written. The “many eyes” aren’t watching. They’re assuming someone else is. This collective delusion is exactly what made the XZ attack possible.
How many dependencies does your main application have? How many of those have you personally reviewed in the last year? If the answer is ‘zero,’ you’re trusting strangers with your production environment.
How Jia Tan Bypassed the Eyeballs
To understand why your scanners failed, you have to understand the mechanics of the attack. It was a masterpiece of both social and software engineering.
Phase 1 - Social Engineering the Maintainer
The attack didn’t start with code. It started with economics.
Lasse Collin had maintained XZ Utils alone, unpaid, for over 15 years. He had a full-time job. XZ was his nights-and-weekends project. In 2021, a coordinated pressure campaign began. Multiple sock-puppet accounts created hostile issues on the XZ mailing list
“Lasse is unresponsive to patches.”
“This project is effectively unmaintained.”
“We need new leadership or XZ will die.”
The manufactured urgency worked. Lasse, burnt out and dealing with personal health issues, accepted help.
Enter “Jia Tan”
A seemingly legitimate open-source contributor who had been making small, helpful commits for months. Jia earned trust slowly. He fixed real bugs. He responded to issues professionally. He became indispensable. After two years of free labor, Lasse granted Jia Tan co-maintainer status and release access.
The initial vulnerability wasn’t a buffer overflow. It was the economics. One unpaid volunteer maintaining critical infrastructure for billions of users. The attackers didn’t need a zero-day. They just needed patience.
Phase 2 - Hiding the Payload
Traditional malware injection is obvious: You add malicious code to .c source files. But Jia Tan knew code reviewers read source files. Even casual contributors glance at GitHub diffs.
Jia Tan’s solution was brilliant. Don’t hide the backdoor in the source code. Hide it where nobody looks.
The payload was concealed inside binary test files, specifically, .xz compressed blobs used for the test suite to verify the compressor was working. Developers assume test data is benign. Nobody reviews binary blobs. The malicious test files sat in the repository for months, completely invisible to the human eye.
Phase 3 - Build-Time Injection
The source code remained clean. Your Veracode or Snyk scanners found nothing suspicious in the .c files. But Jia Tan heavily obfuscated and modified the build scripts (configure.ac and Makefile.am).
During the make compilation process on a Debian or RPM build serve
The build script extracted the hidden payload from the binary test files.
It decrypted the payload and injected the backdoor directly into the compiled binary (
liblzma.so).
The source code never contained the malware, it only existed in the build artifact.
Your build process is a black box.
You review code, but do you verify the compiled binary matches the source? If not, how would you detect a build-time injection?
Phase 4 - The IFUNC/GOT Hijacking
Once the malicious liblzma.so was loaded into memory by the SSH daemon, it executed a sophisticated Linux exploitation technique
IFUNC (Indirect Function) Resolvers and GOT (Global Offset Table) Hijacking.
Linux supports IFUNC, a mechanism where a function’s implementation is chosen dynamically at runtime based on CPU capabilities (e.g., choosing an AVX-512 optimized version if the hardware supports it).
The backdoor registered a malicious IFUNC resolver that executed during the dynamic linking phase, before main() even runs.
This resolver modified the Global Offset Table, replacing the pointer to OpenSSH’s RSA_public_decrypt() function with a pointer to the attacker’s own code.
The backdoor allowed remote code execution using a specific Ed448 cryptographic key controlled by the attacker. No valid SSH credentials were required.
It was entirely stealthy, it checked if the process was /usr/sbin/sshd so it wouldn’t crash normal tarball extractions, and it self-destructed if it detected profiling or debugging tools. (It only failed to hide from Freund’s profiler because of a tiny CPU cycle overhead bug).
Modern security assumes you can trust the build process. Code review, static analysis, and fuzzing all operate on source code.
But if the build itself is compromised, none of that matters. This was an attack on the software supply chain, not the software itself.
The Root Cause
Let’s shift from the technical to the economic. The brutal math of open-source maintenance is the actual root cause of CVE-2024-3094.
Let’s quantify what Lasse Collin was managing
Impact - Billions of devices worldwide.
Critical dependencies - OpenSSH, systemd, dpkg, rpm.
Maintainer compensation - $0/year.
Support burden - Thousands of emails, bug reports, and feature requests from corporations demanding free support.
The Fortune 500 runs mission-critical infrastructure on a foundation of unpaid labor.
OpenSSL - Maintained by ~10 people with an Annual Budget of ~$1M (via grants) and used by Billions of users.
curl - Maintained by Daniel Stenberg (mostly solo) with an annual budget of ~100k (sponsors) and used by Billions of users.
SQLite - Maintained by D. Richard Hipp + 2 contractors with a Self-funded budget and used by Billions of users.
XZ Utils (pre-attack) - Maintained by 1 unpaid volunteer with $0 budget and used by Billions of users.
The sock-puppet pressure campaign worked because Lasse was genuinely overwhelmed. The complaints about slow response times were real. “Helpful” contributors offered relief, and no one else (not Amazon, not Google, not Microsoft) was stepping up to help him. Handing off maintenance seemed like the responsible thing to do.
If XZ Utils disappeared tomorrow and you had to build a replacement in-house, how much would it cost?
A conservative estimate
3 senior engineers × 2 years × $200K = $1.2 million.
That is the real enterprise value Lasse Collin was providing. For free. While working a separate full-time job.
Companies will happily pay $50,000 annually for a Datadog license to monitor their infrastructure.
But they won’t pay $50,000 to sponsor the unpaid maintainer of the compression library that every server in their data center depends on. Until the maintainer burns out. Hands over access. And a nation-state backdoor ends up in production.
If your most critical open-source dependency disappeared tomorrow, how much would it cost to replace it? That’s what the maintainer is saving you. Are you paying them anything?
What you should do now?
The era of blind trust in npm install and apt-get upgrade is over.
Here is the framework for securing your supply chain Monday morning
Audit Your Dependency Tree
Don’t just look for known CVEs. Identify maintenance risk.
Run dependency analysis (
npm audit,pip-audit,cargo audit).Find dependencies with fewer than 3 active maintainers.
Check the last commit date (>1 year without activity is a massive red flag).
Red Flags - A single maintainer, hundreds of unresolved issues, or a maintainer posting “looking for new ownership.”
Pin and Vendor Critical Dependencies
Stop trusting the internet to compile your software.
Stop using latest or ^1.0.0 version ranges in production. Stop auto-updating dependencies without review.
Pin exact versions with SHA checksums. Use lock files (package-lock.json, Cargo.lock). For ultra-critical infrastructure libraries, vendor them (copy the source into your own repository) and build them from source.
Zero Trust for the Build Process
If the build is compromised, the code review doesn’t matter.
Implement Reproducible Builds (the exact same source code must produce the exact same byte-for-byte binary).
Isolate your build environments in ephemeral containers or VMs.
Generate a Software Bill of Materials (SBOM) for every release.
Verify that downloaded packages match published checksums and ensure no unexpected network requests happen during your CI/CD pipeline.
Fund the Maintainers
Funding maintainers isn’t charity. It’s cheaper than breach response, cheaper than in-house development, and cheaper than explaining to your board why you ignored supply chain risk.
If a library is critical to your business, sponsor it via GitHub Sponsors, Tidelift, or Open Collective.
The Goal - $50K–$100K/year for infrastructure-level libraries.
This reduces maintainer burnout, creates accountability, and gives you input on roadmap and priority bug fixes.
Treat Open Source Like Vendors
Apply the exact same due diligence to your open-source libraries that you apply to a SaaS vendor. Monitor dependencies like production services. Alert on new releases, review changelogs before upgrading, and test updates in staging.
You cannot audit every line of every dependency. You have 500+ transitive dependencies, and they change constantly. But you CAN control your build process, fund the maintainers of critical libraries, and treat “free” software as unpaid labor with systemic risk.
Conclusion
The XZ backdoor wasn’t a failure of code. It was a failure of economics.
Jia Tan didn’t find a zero-day or exploit a race condition. He exploited the fact that trillion-dollar companies depend on unpaid volunteers, and then act surprised when those volunteers burn out and hand over the keys.
Linus’s Law ”with enough eyeballs, all bugs are shallow” assumes the eyeballs are actually looking. But nobody reviews compression libraries. Nobody audits build scripts. Nobody funds the maintainer. Until something breaks.
The next time your CFO questions a $500,000 budget for open-source sponsorships, show them the XZ timeline.
Show them the 2.5 years of patient infiltration. Show them that one burnt-out maintainer was all that stood between production systems and a global, nation-state backdoor.
“Free” software isn’t free. You’re just deferring the payment. The only question is
Will you pay before the breach, or after?
Every dependency in your stack is either funded, or it’s a countdown timer.
Credit to Veritasium for their exceptional video breakdown of the Jia Tan timeline, which served as the foundation for this architectural teardown. Watch it here