Hello again, friend! I originally wrote this behind-the-scenes look at the “stage 2” hack in September 2016, shortly after the season finale, “eps2.9_pyth0n-pt2.p7z”, aired. Over five years later, reality once again mirrors fiction: IoT security firm Armis published research showing that they were able to remotely hack a UPS device and cause it to overheat.
My original write-up was on a company site that has since been taken down, so I decided to dust off this blog and re-publish here. And if you’ve enjoyed reading this, please consider donating to one of the many charity organizations that are helping support the people of Ukraine.
Show writer and technical producer Kor Adana came to me with a challenging request: to help craft the “Stage 2” hack that finally comes to light in the concluding episode of season two. The overarching plan was already set in the script: Tyrell and Elliot needed to compromise the power control system in an Evil Corp storage facility, overcharge the batteries in the UPS devices to release hydrogen gas, and finally — while the fans and backup power are down — repeatedly re–engage the UPS autotransformer to create a spark. Boom.
My initial reaction was, “How the hell am I going to make this work — and keep the short glimpses of on–screen activity as realistic as possible?” Many of the prior scenes I had worked on were grounded in familiar attack vectors — for example, Trenton hacking Mobley’s phone with a Stagefright exploit embedded in a malicious web site. But this was on a whole other level!
Kor and James Plouffe, another technical advisor on the show, initially drew inspiration from an incident that occurred at a Sacramento data center in 2001. In that case, overcharged and unventilated UPS batteries culminated in a hydrogen explosion, causing significant damage to a 50,000 sq. foot building. As I got to thinking about scenarios that could allow a hacker to deliberately cause a similar incident, I recalled the recent hack of the power grid in Ukraine, which shut down dozens of substations and left hundreds of thousands of residents in the dark.
My first consideration was that any computer–controlled enterprise hardware, inclusive of something like data center power supplies and control systems, have several layers of “capabilities”:
- End-user control interfaces for standard administration and configuration
- Manufacturer control interfaces for maintenance, upgrades, and features that may be hidden from end-users
- An underlying operating system installed on the firmware — essentially, software embedded within the device — atop which all user and administration functionality resides
- The raw functionality and physical limits of the underlying hardware itself
I started brainstorming about how someone could cause a UPS system to overcharge its batteries, or to repeatedly switch the autotransformer on and off. As you might expect, administrators can customize performance and operational settings and thresholds for these devices — but not beyond safe limits. If someone wanted to override these limits, or operate the hardware in a way that the original software didn’t permit, they’d have to modify the underlying firmware. This would require a few steps.
You’d first need a clean firmware image for the targeted device. Getting this is easier than you might expect: many vendors provide complete (i.e. non–differential) firmware updates for their devices on their support web pages — open for anyone to download.
Next, you’d have to reverse engineer the firmware — essentially identifying the portions of executable code that relate to the functions you want to subvert, then modifying them accordingly. Depending on the obscurity or complexity of the underlying code, this can be an incredibly challenging task — but people with these skills are definitely out there. I’ve had the privilege of working with several brilliant reverse engineers during my career, and never failed to be amazed by what they can do.
Finally, you’d have to figure out how to actually load tampered firmware onto the device. While most modern control systems perform integrity checks and signature validation on firmware updates, that’s sadly not the case for all hardware; and even if present, attackers may identify bypasses.
With this approach in mind, Kor and I discussed altering some of the dialogue in the script to change how Tyrell and Elliot describe the hack. We wanted to make it clear that Tyrell was modifying firmware that overrided software–enforced safety limits — not just running some evil script or fictitious “super–malware” that automatically led to such a disastrous kinetic effect.
At the same time, I got to work on mocking up screens and simulating stages of the attack in Kali Linux. My first step was to get a real enterprise UPS management system’s firmware and documentation — as mentioned before, this was as easy as exploring the publicly–available support web pages of a few major vendors.
I subsequently used a tool called “binwalk” to analyze the firmware image and identify its constituent parts. When Elliot first looks at the screen where Tyrell is working, two binwalk commands have already completed. The first used the options: “–y ‘base64’ –y ‘filesystem’ apc_hw05_aos_640.bin”. This searches the firmware for regions containing either Base64–encoded data or file system signatures. Resources supporting the firmware often reside in these sections.
The second run of binwalk used the single option “–A”. This searches the firmware for opcodes for common CPU architectures, such as Intel x86 or MIPS instructions, thereby pinpointing where the firmware contains executable code that would be the target for reverse engineering and tampering. The command identified two sections containing what appears to be Intel x86 code.
Later in the episode, Elliot confronts Tyrell while he is on–keyboard continuing to work on the hack. I wanted to show Tyrell in the process of analyzing and modifying one of the regions of executable code identified by “binwalk”. Since he’s in Kali Linux, I decided to have him use “radare2” — an open source binary reverse engineering toolkit. (Sorry IDA Pro fans!)
The screen shows that Tyrell ran a few commands. The initial command line loaded the firmware into “radare2” at an offset near one of the executable sections identified by “binwalk”:
radare2 –s 0x32C24 –w apc_hw05_aos_640.bin
On–screen, the viewer sees an excerpt of output from Tyrell’s first command within radare2, “pd 16”, which dumped 16 bytes of disassembly beginning from that offset. The next command, “s 0x00032c34”, set the cursor to the specified address, followed by “pd 1” which showed that it contained the instruction “je” — “jump if equal”. Tyrell had just finished typing “wx 75” which overwrote one byte to change the instruction to “jne” — “jump if not equal”, effectively reversing the logic that dictates the flow of a block of code.
After taking control, Elliot quit “radare2” and ran the Linux “shred” command to securely delete the tampered firmware.
All of these screenshots and mock–ups were built using these exact commands and tools, operating on a real firmware image, in a Kali VM. Did I actually create a “weaponized” firmware or find any vulnerabilities? Of course not! In truth, I picked an arbitrary code segment and flipped a single jump instruction — I have no idea what that piece of code actually does, and it’s doubtful it would have caused any harm beyond creating a corrupt firmware that wouldn’t load. A practical attack might be several orders of magnitude more complex, and Tyrell would still need an additional exploit to bypass any firmware integrity checks. But the overarching process is as close to real as any TV show or movie has attempted.
End–to–end, Kor and I spent well over a week on the pre–production design for this scene, plus additional time to review and tweak things before the final network cut. All for a few seconds of on–screen time — but at an absolutely critical juncture of the story.