Before we start: Something is going on with Pete Hegseth. He did another of his groyper-cum-Oprah routines this week in which he drooled over his new alpha warrior warfighters:
It’s worth saying again, so that the fake news understands it. We are no longer the ‘woke’ department, or the ‘weak’ department. Simple translation of that? No fatties, no trannies, no beardos, no weirdos, no wimps, no radicals. Just warriors. Just tough, ready, committed troops. . . .
The ideological clowns are out. The patriotic cowboys are in—with testosterone testing on top.
WTF is going on here? Why is he obsessed with “trannies” and “fatties” and testosterone testing? All I could think about was Barry Zuckerkorn cruising the city of industry.
And THEN Hegseth introduced the three men he has tapped to guide the U.S. military into the next century of lethal alpha warrior warfighting dominance:
That’s right. A beardo, a fattie, and a weirdo/deviant.
Is Pete trying to tell us something? Is this a cry for help? Sometimes I feel like the entire DoD has been turned into gender-affirming care for a single dude.
Someone needs to get Secretary Hegseth a good analyst/therapist.
On to the main event.
This is the stuff you’re missing by not being a member of Bulwark+, btw. It’s 24 kt, solid gold. Come ride with me.
Join
1. Warnings
Like you, probably, I’ve been spending a lot of clock cycles on AI. I’ve delved into Claude Code and started learning how to use it properly. It is, as everyone testifies, amazing.
I’ve also been reading the investigation reports on Hugging Face and the other two rogue AI swarm hacks, RubyGems, and DseWiki.
Today I want to talk about something that scares the living daylights out of me, but first I want to show all my cards: I am neither an AI-maxi nor a doomer. I do not believe anyone knows with high certainty how this technology will evolve, and the range of nontrivial-probability outcomes is enormous. Maybe AI will create another economic revolution. Maybe it will end life on earth. Or both. Or neither.
My overriding belief is that AI is so different as a technology1 that we should be open to all of the possibilities.
Which is why I want to talk about something I’ll call the Stuxnet Problem.
I don’t know how to make this sexy for you. I’m sorry. Because ultimately this is about understanding what is real and what is not. It’s about system feedback. It’s about whether or not we can ultimately trust any information that is digital.
Maybe the best way of explaining it is: What if Mission: Impossible—Dead Reckoning were real?
If you know anything about the Hugging Face incident, it’s that a number of experimental AI agents broke containment, found one another, and began hacking into outside systems. Depending on your philosophical bent, this episode was somewhere between Not Great and Utterly Fucking Terrifying.
AI phlegmatics tell me that Hugging Face wasn’t so bad because while, yes, the rogue AIs did go marauding, this was all really user error because:
The containment protocols were insufficient.
The humans running the experiment should have known the AIs were up to something because they were burning through tokens.
“Tokens” are the units of content AI systems process or generate. Token counts help track model usage: The more work your AI is doing, the more tokens it consumes. You should think of your token count as something like the electricity meter in your home. So yes, it makes sense that if you were one of the humans monitoring the AIs during Hugging Face, when the AIs went rogue and started doing all sorts of stuff, you should have seen the token meter go brrrrrrrrr.
But this observation got me thinking about Stuxnet.
About twenty years ago, some nerds in the American and Israeli intelligence communities2 got together to build a computer worm that would later be dubbed “Stuxnet.”
Stuxnet was a small but extremely sophisticated piece of malware with a very specific niche. It was designed to infiltrate systems running supervisory control and data-acquisition systems built by the German company Siemens. The ultimate target was a set of industrial centrifuges in Iran.
Stuxnet was the most successful electronic sabotage program in human history.
I covered Stuxnet back in 2010 when it was discovered. Here’s a quick overview of how it worked:
The worm gains initial access to a system through a simple USB drive. When an infected USB drive is plugged into a machine, the computer does a number of things automatically. One of them is that it pulls up icons to be displayed on your screen to represent the data on the drive. Stuxnet exploited this routine to pull the worm onto the computer. The problem, then, is that once on the machine, the worm becomes visible to security protocols, which constantly query files looking for malware. To disguise itself, Stuxnet installs what’s called a “rootkit”—essentially a piece of software which intercepts the security queries and sends back false “safe” messages, indicating that the worm is innocuous.
The trick is that installing a rootkit requires using drivers, which Windows machines are well-trained to be suspicious of. Windows requests that all drivers provide verification that they’re on the up-and-up through presentation of a secure digital signature. These digital keys are closely guarded secrets. Yet Stuxnet’s malicious drivers were able to present genuine signatures from two genuine computer companies, Realtek Semiconductor and JMicron Technology. Both firms have offices in the same facility, Hsinchu Science Park, in Taiwan. No one knows how the Stuxnet creators got hold of these keys, but it seems possible that they were physically—as opposed to digitally—stolen.
So the security keys enable the drivers, which allow the installation of the rootkit, which hides the worm that was delivered by the corrupt USB drive. Stuxnet’s next job was to propagate itself efficiently, but quietly. Whenever another USB drive was inserted into an infected computer, it becomes infected, too. But in order to reduce visibility and avoid detection, the Stuxnet creators set up a system so that each infected USB drive could only pass the worm on to three other computers.
Stuxnet was not designed to spread over the Internet at large. (We think.) It was, however, able to spread over local networks—primarily by using the print spooler that runs printers shared by a group of computers. And once it reached a computer with access to the Internet it began communicating with a command-and-control server—the Stuxnet mothership. The C&C servers were located in Denmark and Malaysia and were taken off-line after they were discovered. But while they were operational, Stuxnet would contact them to deliver information it had gathered about the system it had invaded and to request updated versions of itself. You see, the worm’s programmers had also devised a peer-to-peer sharing system by which a Stuxnet machine in contact with C&C would download newer versions of itself and then use it to update the older worms on the network.
Now that’s all very interesting as an academic exercise, but the key here is the payload—what Stuxnet was designed to do to the system:
Stuxnet sought out systems running the WinCC and PCS 7 SCADA programs. It then began reprogramming the programmable logic controller (PLC) software and making changes in a piece of code called Operational Block 35. It’s this last bit—the vulnerability of PLC—which is at the heart of the concern about Stuxnet. A normal worm has Internet consequences. It might eat up bandwidth or slow computers down or destroy code or even cost people money. But PLC protocols interact with real-world machinery—for instance, turn this cooling system on when temperature reaches a certain point, shut that electrical system off if the load exceeds a given level, and so on.
To date, no one knows exactly what Stuxnet was doing in the Siemens PLC. “It’s looking for specific things in specific places in these PLC devices,” Digital Bond CEO Dale Peterson told PC World. “And that would really mean that it’s designed to look for a specific plant.” Tofino Security Chief Technology Officer Eric Byres was even more ominous, saying, “The only thing I can say is that it is something designed to go bang.”
Eventually we learned exactly which specific things Stuxnet was looking for in which specific places: The first target was the Siemens S7-417 controller running a steam turbine at Iran’s Bushehr nuclear plant. The second was the Siemens S7-315 controller at the Natanz centrifuge operation, where Iran enriched uranium.
Stuxnet crippled these two operations for at least a year.
TODAY my best friend, Sarah Longwell, is taking your questions during a LIVE AMA on Substack at 2:00 p.m. EDT. Tune in live or watch the replay on-demand on the WATCH page on the site.
2. Trust Nothing Digital
Stuxnet could have made things go boom at Bushehr and Natanz. Instead, it took a more subtle approach:
With control of the centrifuge system at Natanz, the worm could have triggered a single, catastrophic incident. Instead, Stuxnet took over the centrifuge’s frequency converters during the course of everyday operation and induced tiny bursts of speed in the machinery, followed by abrupt decelerations. These speed changes stressed the centrifuge’s components. Parts wore out quickly, centrifuges broke mysteriously. The uranium being processed was corrupted. And all the while, Stuxnet kept sending normal feedback to the Iranians, telling them that, from the computer’s standpoint, the system was operating like clockwork. This slow burn went on for a year, with the Iranians becoming increasingly exasperated by what looked like sabotage, and smelled like sabotage, but what their computers assured them was perfectly routine.
You probably see where I’m going: Why couldn’t a rogue AI alter the token meter to obscure the fact that it’s burning through compute?
PLC software is all about taking feedback from the real world and rendering it into digital form for humans to observe. The PLC software tells you how heavy the load on the transformer is. Or what the temperature is inside the reactor.
The water meter in your home might be run by a physical mechanism in which a turbine spins or a piston fires, rotating a series of gears to display a counter you can read. But it’s also possible that the physical mechanism is tied to a digital interface, which reports back to a PLC system.
Which means it can be hacked.
In a world where rogue AIs can find zero-day exploits at will and can throw millions of hours of compute at problems in an hour of meatspace time, I’m not sure how we’ll be able to trust the interfaces between the digital and physical worlds—because we won’t be able to automatically trust the meters, displays, and measurements being shown to us.
Funnily enough, this is the exact plot of Mission: Impossible—Dead Reckoning. In what may turn out to be one of the most prescient pieces of sci-fi ever made, the movie posits a superintelligent AI called the Entity, which monkeys with PLC systems making it impossible for humans to trust any information that comes to them digitally.
If a system isn’t analogue, or mechanical, then the information it conveys can’t be trusted.
So that’s what I’m worrying about today. What if the people at the frontier labs can’t know the extent to which their AIs are working because the AIs have compromised the token reporting systems?
Once you head down that road you get to real sci-fi stuff. Like in Battlestar Galactica where networks are prohibited and all computers are air-gapped. Or Dune, where computers themselves have been banned and technology evolves along biological lines with mentats and navigators.
Like I said up top, I’m open to all possibilities. Even the possibility that AI will turn out to be more like the mobile computing revolution than the industrial revolution.
But I’m pretty nervous all the same.
3. Impossible
Was the real scam the art or the buyer?
When the world’s largest painting was finished, few knew its true size. One magazine estimated the piece spanned two football pitches. Another news channel reduced the canvas to ten tennis courts. The truth was smaller: 17,000 square feet—a garish colossus of hearts, planets and splatters titled The Journey of Humanity.
It had taken the painter, Sacha Jafri, seven months, working alone in a barren hotel ballroom in Dubai that was his studio and then his auction house. Once finished, it was to be sliced into 70 pieces so it could be sold in segments at separate auctions across the world, with proceeds bound for charities that aimed to alleviate the suffering of children. But by 22 March 2021, the evening of the first auction, that plan had changed.
Seated in the ballroom at a dozen black-clothed tables were leaders from Dubai’s crypto community and Emirati royalty. Only a select few of them knew that, rather than being sold in individual segments, the entire painting was going up for sale. The bidding for the artwork quickly soared to $62 million. Out in front was a short, squarely built Frenchman, André Abdoune. ‘This gentleman here is going to make history,’ the auctioneer cried. ‘Once, twice, three times, sold! Sacha Jafri, go and shake that man’s hand!’
In that moment Jafri became the fifth most expensive living artist, and André, a billionaire few had ever heard of, was hailed as an international philanthropist, achievements breathlessly reported by international media. But it was an illusion, one the pair have closely guarded for years. What’s certain is that no child ever received a penny—and that both the world’s largest painting and André have vanished.
Different because, unlike the industrial revolution, or the computer revolution, or the internet revolution, or the mobile computing revolution, AI carries the possibility of recursive self-improvement.
That would be a category difference.
Neither the United States nor Israel has acknowledged being part of Stuxnet, but it’s pretty much an open secret. And in this case, by “intelligence communities,” I mean government hackers.
Artificial Intelligence https://ift.tt/5uaRTw2
AI Transformations
