{"id":2540,"date":"2026-03-03T08:30:00","date_gmt":"2026-03-03T08:30:00","guid":{"rendered":"https:\/\/tekhead.it\/blog\/?p=2540"},"modified":"2026-03-02T22:31:03","modified_gmt":"2026-03-02T22:31:03","slug":"half-a-second-saved-the-internet","status":"publish","type":"post","link":"http:\/\/tekhead.it\/blog\/2026\/03\/half-a-second-saved-the-internet\/","title":{"rendered":"Half a Second Saved the Internet"},"content":{"rendered":"\n<p>One of my favourite activities on a relaxed weekend morning is watching a couple of <a href=\"https:\/\/www.youtube.com\/veritasium\" type=\"link\" id=\"https:\/\/www.youtube.com\/veritasium\" target=\"_blank\" rel=\"noreferrer noopener\">Veritasium<\/a> and <a href=\"https:\/\/www.youtube.com\/@TheB1M\" type=\"link\" id=\"https:\/\/www.youtube.com\/@TheB1M\" target=\"_blank\" rel=\"noreferrer noopener\">B1M<\/a> videos. Last week Veritasium published a <a href=\"https:\/\/www.youtube.com\/watch?v=aoag03mSuXQ\" type=\"link\" id=\"https:\/\/www.youtube.com\/watch?v=aoag03mSuXQ\" target=\"_blank\" rel=\"noreferrer noopener\"><em>superb<\/em> video<\/a> on a [not so] simple Linux exploit, that could have had HUGE ramifications. If you haven&#8217;t seen it, go watch it, I&#8217;ll wait. It&#8217;s actually one of the most fascinating, yet little known stories in recent tech history, and it sits right at the intersection of many of the things that interest me; open source, trust, the humans behind the software, and just how fragile much of it really is.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<span class=\"embed-youtube\" style=\"text-align:center; display: block;\"><iframe loading=\"lazy\" class=\"youtube-player\" width=\"590\" height=\"332\" src=\"https:\/\/www.youtube.com\/embed\/aoag03mSuXQ?version=3&#038;rel=1&#038;showsearch=0&#038;showinfo=1&#038;iv_load_policy=1&#038;fs=1&#038;hl=en-GB&#038;autohide=2&#038;wmode=transparent\" allowfullscreen=\"true\" style=\"border:0;\" sandbox=\"allow-scripts allow-same-origin allow-popups allow-presentation allow-popups-to-escape-sandbox\"><\/iframe><\/span>\n<\/div><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">A Lone Maintainer<\/h2>\n\n\n\n<p>If you CBA watching the video (seriously though, you really should, just put it on 1.5x!), here&#8217;s the short version. A lone developer called <a href=\"https:\/\/github.com\/Larhzu\" type=\"link\" id=\"https:\/\/github.com\/Larhzu\" target=\"_blank\" rel=\"noreferrer noopener\">Lasse Collin<\/a> maintained a compression library called &#8220;XZ Utils&#8221;, quietly and unpaid, for roughly twenty years. You&#8217;ve almost certainly never heard of it, which is basically the point. It sits underneath an <em>enormous <\/em>amount of critical infrastructure, including (most importantly) SSH, that millions of servers rely on every single day. Nobody thinks about it, nobody talks about it, and for most of its life exactly one person was keeping the lights on&#8230;<\/p>\n\n\n\n<p>Lasse, who was already burned out and struggling with his mental health, was being hounded by &#8220;accounts&#8221; to make more progress on the project, some messages encouraging him to accept help and hand over responsibility to other devs. Then someone calling themselves &#8220;Jia Tan&#8221; showed up, which we now know was almost certainly a nation state operation, and spent two and a half years patiently social engineering their way into becoming a trusted maintainer of the project. They were helpful, responsive, wrote good code, etc. All seemed peachy! Enter <a href=\"https:\/\/rwmj.wordpress.com\/about\/\" type=\"link\" id=\"https:\/\/rwmj.wordpress.com\/about\/\" target=\"_blank\" rel=\"noreferrer noopener\">Rich Jones<\/a>, who works at RedHat, packaging Fedora. He began to trust Jia because&#8230; well, because Jia behaved exactly like the kind of person open source desperately needs. That&#8217;s what makes social engineering so effective; the good behaviour <em>is<\/em> the attack.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">A Backdoor to the Internet<\/h2>\n\n\n\n<p>The backdoor they slipped in was technically brilliant and horrifying in equal measure, a hidden compromise buried in the compression library that would have given someone a backdoor to a significant chunk of the world&#8217;s servers if it had made it into stable releases across major Linux distros. The whole thing eventually unravelled because a single Microsoft developer called <a href=\"https:\/\/x.com\/AndresFreundTec\" type=\"link\" id=\"https:\/\/x.com\/AndresFreundTec\" target=\"_blank\" rel=\"noreferrer noopener\">Andres Freund<\/a> noticed that his SSH logins were taking <em>half a second longer<\/em> than they should and decided to dig into why. Five hundred milliseconds stood between us and a catastrophic supply chain compromise, and one curious engineer is the reason we caught it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Open Source Matters<\/h2>\n\n\n\n<p>I&#8217;m pro open source, always have been. I&#8217;ve been using Linux since the 90s, which probably gives you an idea of what colour my beard is. The concept of &#8220;Software should be free and we&#8217;ll prove it works&#8221; is unarguably one of the greatest ever human endeavours. When geopolitical tensions seem to ratchet up weekly, where we&#8217;re supposedly retreating into blocs and borders, the fact that open source <em>still works<\/em> is genuinely remarkable and something to make you proud of bring an ape descendant! It&#8217;s proof that like-minded humans can can continue to collaborate on a global scale.<\/p>\n\n\n\n<p>But here&#8217;s where we have a challenge&#8230; <a href=\"https:\/\/en.wikipedia.org\/wiki\/Linus%27s_law\" type=\"link\" id=\"https:\/\/en.wikipedia.org\/wiki\/Linus%27s_law\" target=\"_blank\" rel=\"noreferrer noopener\">Linus&#8217;s Law<\/a> says that given enough eyeballs, all bugs are shallow. That&#8217;s true for projects that actually <em>have<\/em> enough eyeballs, but popularity does not equal scrutiny. XZ Utils had <em>millions <\/em>(perhaps billions) of installs, but for most of its life basically one person was reading the code, and then there were two, and the second one was the attacker. (Yes, I know, hyperbole, but you get the point!). <strong>Downloads are not eyeballs<\/strong>, installs are not audits, and I think we&#8217;ve been confusing usage with oversight for a long time. The XZ story is a great example of where it went both brilliantly right and <em>very <\/em>wrong.<\/p>\n\n\n\n<p>The real vulnerability here wasn&#8217;t technical; it was a system that let a single unpaid maintainer carry critical infrastructure on his back for two decades without meaningful support. We collectively built our digital world on top of someone&#8217;s volunteer labour and then acted surprised when that became an attack surface!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Will AI Make This Better or Worse?<\/h2>\n\n\n\n<p>Which brings me to the point of this article.<\/p>\n\n\n\n<p><em>Can AI help with this?<\/em><\/p>\n\n\n\n<p>[&#8230; and yes, it&#8217;s AI again! You now have permission to roll your eyes about having read YAAIA, aka <em>yet another AI article<\/em>]<\/p>\n\n\n\n<p>In theory, AI-powered code review could potentially spot the kind of obfuscated changes that slipped through here. Automated analysis tooling that never gets tired, never gets burned out, never feels social pressure to approve a commit because the submitter has been so helpful lately. That sounds very promising, but there&#8217;s another side to it. As more code gets written by AI and reviewed by AI, do we end up with even fewer human eyeballs on critical paths? Do we create a new kind of &#8220;enough eyeballs&#8221; fallacy where the eyeballs are all artificial and share the same blind spots?<\/p>\n\n\n\n<p>I genuinely don&#8217;t know the answer, and I suspect the truth is that AI will simultaneously make some attacks harder and others easier [really helpful insight, I know!].<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"583\" src=\"https:\/\/tekhead.it\/wp-uploads\/www.tekhead.org\/2026\/03\/article-1200-2-1024x583.jpg\" alt=\"Interesting times in Security\" class=\"wp-image-2546\" style=\"aspect-ratio:1.7564609403077727;width:433px;height:auto\" srcset=\"http:\/\/tekhead.it\/wp-uploads\/www.tekhead.org\/2026\/03\/article-1200-2-1024x583.jpg 1024w, http:\/\/tekhead.it\/wp-uploads\/www.tekhead.org\/2026\/03\/article-1200-2-300x171.jpg 300w, http:\/\/tekhead.it\/wp-uploads\/www.tekhead.org\/2026\/03\/article-1200-2-150x85.jpg 150w, http:\/\/tekhead.it\/wp-uploads\/www.tekhead.org\/2026\/03\/article-1200-2.jpg 1200w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n<\/div>\n\n\n<h2 class=\"wp-block-heading\">Interesting Times<\/h2>\n\n\n\n<p>What I do know is that the XZ story deserves to be told widely, especially within our industry. Absolutely not as a cautionary tale about open source, because closed source has its own risks and horror stories that just happen behind closed doors.<\/p>\n\n\n\n<p>To me, it&#8217;s a reminder that the humans behind the code matter as much as the code itself. Fund the hard working maintainers, buy them a ko-fi, support the people doing unglamorous work, and maybe, occasionally, investigate when something takes half a second longer than it should.<\/p>\n\n\n\n<p>As my favourite author of all time, Sir Terry Pratchett reminded us, &#8220;may you live in interesting times&#8221;. As the world tries to keep up with supply chain security, I can confirm interesting times are in the current sprint&#8230;<\/p>\n\n\n\n<p><\/p>\n\n\n\n<p>PS &#8211;  Amusingly as I was writing this, another article on a similar topic hit the headlines, about <a href=\"https:\/\/www.theregister.com\/2026\/02\/28\/open_source_opinion\/\" type=\"link\" id=\"https:\/\/www.theregister.com\/2026\/02\/28\/open_source_opinion\/\" target=\"_blank\" rel=\"noreferrer noopener\">FOSS repos<\/a>. Worth a few minutes of your time too I reckon.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The XZ Utils backdoor is one of the most fascinating near-misses in internet security history. A story about trust, burnout, open source, and the 500 milliseconds that stopped a catastrophic supply chain attack. <a href=\"http:\/\/tekhead.it\/blog\/2026\/03\/half-a-second-saved-the-internet\/\"><span class=\"read-more\">Read more <span class=\"meta-nav\">&raquo;<\/span><\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":2546,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_yoast_wpseo_focuskw":"open source security","_yoast_wpseo_title":"Half a Second Saved the Internet | Open Source Security","_yoast_wpseo_metadesc":"The XZ Utils backdoor nearly compromised millions of servers. A story about open source trust, developer burnout, social engineering, and whether AI will make supply chain security better or worse.","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"\"Half a Second Saved the Internet\"\n\nI watched a fantastic video this weekend from Veritasium about security & it inspired this post.\n\nAlso, nod to my all time favourite author (& Nanobanana who helped me generate the image when other providers tools wouldnt!)\n\n\n#interestingtimes #linux #security #ai","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[45,377],"tags":[1102,542,1125,548,1127,390,1126,1128],"class_list":["post-2540","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-linux","category-security","tag-ai","tag-linux","tag-open-source","tag-security","tag-social-engineering","tag-ssh","tag-supply-chain-security","tag-xz-utils"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"http:\/\/tekhead.it\/wp-uploads\/www.tekhead.org\/2026\/03\/article-1200-2.jpg","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p2l3lU-EY","amp_enabled":true,"_links":{"self":[{"href":"http:\/\/tekhead.it\/blog\/wp-json\/wp\/v2\/posts\/2540","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/tekhead.it\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/tekhead.it\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/tekhead.it\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/tekhead.it\/blog\/wp-json\/wp\/v2\/comments?post=2540"}],"version-history":[{"count":3,"href":"http:\/\/tekhead.it\/blog\/wp-json\/wp\/v2\/posts\/2540\/revisions"}],"predecessor-version":[{"id":2549,"href":"http:\/\/tekhead.it\/blog\/wp-json\/wp\/v2\/posts\/2540\/revisions\/2549"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/tekhead.it\/blog\/wp-json\/wp\/v2\/media\/2546"}],"wp:attachment":[{"href":"http:\/\/tekhead.it\/blog\/wp-json\/wp\/v2\/media?parent=2540"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/tekhead.it\/blog\/wp-json\/wp\/v2\/categories?post=2540"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/tekhead.it\/blog\/wp-json\/wp\/v2\/tags?post=2540"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}