Check out more on the tech advisors to the Mr. Robot show.

{Warning to readers: Contains major spoilers for the final episode of Mr. Robot season 2.}

Show writer and technical producer Kor Adana came to me with a challenging request: to help craft the “Stage 2” hack that finally comes to light in the concluding episode of season two. The overarching plan was already set in the script: Tyrell and Elliot needed to compromise the power control system in an Evil Corp storage facility, overcharge the batteries in the UPS devices to release hydrogen gas, and finally – while the fans and backup power are down – repeatedly re–engage the UPS autotransformer to create a spark. Boom.

My initial reaction was, “How the hell am I going to make this work – and keep the short glimpses of on–screen activity as realistic as possible?” Many of the prior scenes I had worked on were grounded in familiar attack vectors – for example, Trenton hacking Mobley’s phone with a Stagefright exploit embedded in a malicious web site. But this was on a whole other level!

Kor and James Plouffe, another technical advisor on the show, initially drew inspiration from an incident that occurred at a Sacramento data center in 2001. In that case, overcharged and unventilated UPS batteries culminated in a hydrogen explosion, causing significant damage to a 50,000 sq. foot building. As I got to thinking about scenarios that could allow a hacker to deliberately cause a similar incident, I recalled the recent hack of the power grid in Ukraine, which shut down dozens of substations and left hundreds of thousands of residents in the dark.

My first consideration was that any computer–controlled enterprise hardware, inclusive of something like data center power supplies and control systems, have several layers of “capabilities”:

• End-user control interfaces for standard administration and configuration
• Manufacturer control interfaces for maintenance, upgrades, and features that may be hidden from end-users
• An underlying operating system installed on the firmware – essentially, software embedded within the device – atop which all user and administration functionality resides
• The raw functionality and physical limits of the underlying hardware itself

I started brainstorming about how someone could cause a UPS system to overcharge its batteries, or to repeatedly switch the autotransformer on and off. As you might expect, administrators can customize performance and operational settings and thresholds for these devices – but not beyond safe limits. If someone wanted to override these limits, or operate the hardware in a way that the original software didn’t permit, they’d have to modify the underlying firmware. This would require a few steps.

You’d first need a clean firmware image for the targeted device. Getting this is easier than you might expect: many vendors provide complete (i.e. non–differential) firmware updates for their devices on their support web pages – open for anyone to download.

Next, you’d have to reverse engineer the firmware – essentially identifying the portions of executable code that relate to the functions you want to subvert, then modifying them accordingly. Depending on the obscurity or complexity of the underlying code, this can be an incredibly challenging task – but people with these skills are definitely out there. I’ve had the privilege of working with several brilliant reverse engineers during my career, and never failed to be amazed by what they can do.

Finally, you’d have to figure out how to actually load tampered firmware onto the device. While most modern control systems perform integrity checks and signature validation on firmware updates, that’s sadly not the case for all hardware; and even if present, attackers may identify bypasses.

With this approach in mind, Kor and I discussed altering some of the dialogue in the script to change how Tyrell and Elliot describe the hack. We wanted to make it clear that Tyrell was modifying firmware that overrided software–enforced safety limits – not just running some evil script or fictitious “super–malware” that automatically led to such a disastrous kinetic effect.

At the same time, I got to work on mocking up screens and simulating stages of the attack in Kali Linux. My first step was to get a real enterprise UPS management system’s firmware and documentation – as mentioned before, this was as easy as exploring the publicly–available support web pages of a few major vendors.

I subsequently used a tool called “binwalk” to analyze the firmware image and identify its constituent parts. When Elliot first looks at the screen where Tyrell is working, two binwalk commands have already completed. The first used the options: “–y ‘base64’ –y ‘filesystem’ apc_hw05_aos_640.bin”. This searches the firmware for regions containing either Base64–encoded data or file system signatures. Resources supporting the firmware often reside in these sections.

The second run of binwalk used the single option “–A”. This searches the firmware for opcodes for common CPU architectures, such as Intel x86 or MIPS instructions, thereby pinpointing where the firmware contains executable code that would be the target for reverse engineering and tampering. The command identified two sections containing what appears to be Intel x86 code.

Screen Shot: Original mock–up of “binwalk” analysis of firmware image

Later in the episode, Elliot confronts Tyrell while he is on–keyboard continuing to work on the hack. I wanted to show Tyrell in the process of analyzing and modifying one of the regions of executable code identified by “binwalk”. Since he’s in Kali Linux, I decided to have him use “radare2” – an open source binary reverse engineering toolkit. (Sorry IDA Pro fans!)

The screen shows that Tyrell ran a few commands. The initial command line loaded the firmware into “radare2” at an offset near one of the executable sections identified by “binwalk”:

radare2 –s 0x32C24 –w apc_hw05_aos_640.bin

On–screen, the viewer sees an excerpt of output from Tyrell’s first command within radare2, “pd 16”, which dumped 16 bytes of disassembly beginning from that offset. The next command, “s 0x00032c34”, set the cursor to the specified address, followed by “pd 1” which showed that it contained the instruction “je” – “jump if equal”. Tyrell had just finished typing “wx 75” which overwrote one byte to change the instruction to “jne” – “jump if not equal”, effectively reversing the logic that dictates the flow of a block of code.

Screen Shot: Original mock–up of reverse engineering the firmware in “radare2”

After taking control, Elliot quit “radare2” and ran the Linux “shred” command to securely delete the tampered firmware.

All of these screenshots and mock–ups were built using these exact commands and tools, operating on a real firmware image, in a Kali VM. Did I actually create a “weaponized” firmware or find any vulnerabilities? Of course not! In truth, I picked an arbitrary code segment and flipped a single jump instruction — I have no idea what that piece of code actually does, and it’s doubtful it would have caused any harm beyond creating a corrupt firmware that wouldn’t load. A practical attack might be several orders of magnitude more complex, and Tyrell would still need an additional exploit to bypass any firmware integrity checks. But the overarching process is as close to real as any TV show or movie has attempted.

End–to–end, Kor and I spent well over a week on the pre–production design for this scene (mostly late evening work for me, given that I still have my awesome day job at Tanium!), plus additional time to review and tweak things before the final network cut. All for a few seconds of on–screen time — but at an absolutely critical juncture of the story.

It’s been really fun to see how tech–savvy viewers picked up on subtle on–screen details and hunted down easter eggs throughout this season. I also hope that the hacks can inspire and educate those that are new to the security field to learn more, experiment, and explore what they’ve seen in the show: be it understanding how tools like Mimikatz work, the attack vectors used to deliver Android exploits like Stagefright, or the steps that go into reverse engineering unknown binaries.

Like what you see? Click here and sign up to receive the latest Tanium news and learn about our upcoming events.

About the author: In his role as Tanium’s Chief Security Architect, Ryan Kazanciyan brings more than 14 years of experience in incident response, forensic analysis, and penetration testing. Ryan oversees the design and roadmap for Tanium’s Threat Response offerings, and leads the Tanium Endpoint Detection and Response (EDR) team. Prior to joining Tanium, Ryan oversaw investigation and remediation efforts at Mandiant, partnering with dozens of Fortune 500 organizations affected by targeted attacks. Ryan has trained hundreds of incident responders as an instructor for Black Hat and the FBI’s cyber squad. He is a contributing author for “Incident Response and Computer Forensics 3rd Edition” (McGraw-Hill, 2014). Ryan also works as a technical consultant for the television series “Mr. Robot”, where he collaborates with the writers and production team to design the hacks depicted in the show.