Back to the crash: Time Travel Analysis for Fuzzing Crash Analysis

Introduction

Fuzz testing is a powerful way to find vulnerabilities by throwing unexpected inputs at software. But when something crashes? That’s where things get tricky.

Using CVE-2023-4041, a Secure Boot vulnerability, as our example, we’ll demonstrate how time travel analysis helps trace the root cause of a crash, assess its impact, and understand the flaw with speed and accuracy.

CVE-2023-4041: buffer overflow in Silicon Labs’ Gecko Secure Boot

CVE-2023-4041 is a vulnerability present on the EFx32x modules, from Silicon Labs vendor, using the Gecko Bootloader before version 4.3.1. It was discovered and published by Quarkslab, and described in different conferences. They identified a flaw in the updating process which enabled them to get control of the execution flow. So, an unsigned firmware could be updated: the Secure Boot was bypassed.

The exploitation was carried out on the BGM220-EK4314A board (see image below), a low-cost (~20€) development kit equipped with an on-board debugger utilizing a Serial Wire Debug (SWD) interface. It's noteworthy that all Gecko Bootloader code and documentation are open source, with Silicon Labs providing a dedicated IDE and example code for rapid development.

This board supports Bluetooth connectivity, allowing seamless over-the-air (OTA) updates for the bootloader.

The update process can be secured with encryption and digital signatures, a mechanism referred to as Secure Boot by Silicon Labs. However, despite its name, it's important to note that Secure Boot only verifies firmware authenticity during updates, not at startup.

In their blog post, Quarkslab details the steps they took to identify and exploit the CVE-2023-4041 vulnerability.

• Firmware Acquisition and Emulation: They retrieved the firmware and emulated it using Unicorn.
• Target Fuzzing: Utilizing AFL++ for fuzzing campaigns, they complemented these efforts by defining the GBL file structure using protobuf format. This enhanced the effectiveness of their fuzzing by generating more meaningful test inputs.
• Vulnerability Identification: In-depth analysis of a fuzzing crash revealed the flaw in the bootloader's parser.
• Exploitation for Unsigned Bootloader Installation: The flaw was successfully exploited to install an unsigned bootloader on the target device, demonstrating the Secure Boot could be bypassed.

Their blogpost provides a great overview of the entire vulnerability discovery process as well as exploitation. But, what was the difficulty in understanding a crash and tracing its origin?

Leveraging Time Travel Analysis in esReverse

Time Travel Analysis is a powerful technique that allows analysts to record and explore a program's complete execution history. Instead of repeatedly running the code, analysts capture a persistent dataset representing every step. This “execution trace” can be revisited at will, allowing analysts to rewind through time, pinpoint crucial moments, and investigate vulnerabilities, bugs, or unexpected behavior.

The ability to rewind through the execution history eliminates the need for repeated debugging sessions. We explored this in-depth in our comparison blogpost.

This analysis was conducted using esReverse platform. With its all-in-one environment, we created a project and integrated all the necessary tools for our analysis. This included the fuzzing tool AFL++, time travel debugging tool on an Arm binary, and static analysis tools like Ghidra and IDA.

Overview of IDA integrated into esReverse

To power time travel debugging, esReverse combines key emulation technologies. It uses QEMU, Unicorn, or Qiling to run the binary and capture a full execution trace. Once recorded, this trace can be explored in depth, either through a user-friendly graphical interface or directly via Python scripts for full control.

Our target: the GBL (Gecko BootLoader) file format used for bootloader updates.

The Gecko BootLoader File

The update process uses a proprietary format called Gecko BootLoader (GBL), which employs a Tag Length Value (TLV) structure. In this format, each piece of information is represented as a triplet:

• A tag identifier
• The length of the associated data
• The data value itself

This structure allows for flexible and straightforward parsing of the update file.

The GBL file contains more than just the code. It also includes essential metadata about the update, such as:

• Code Update Location – Specifies where the new code should be written within the target device.
• Code Signature – Optional digital signatures can be included to verify the authenticity and integrity of the code.
• Transmission Method – Supports both cleartext and encrypted transmission.
• Update Target – Updates can apply not only to the bootloader but also to smaller components, such as applications.

Here's a snippet of a GBL file, along with an explanation of its structure.

This intricate structure necessitates a parser to analyze and interpret the GBL file. It is within this parser that the vulnerability lies.

Time travel analysis: trace generation

We used a firmware emulation with Unicorn and AFL++ for fuzzing. Notably, one crash caused Unicorn to halt, indicating the execution of an invalid instruction.

A snippet of this GBL file is given below.

It's a perfect case to analyze with time travel analysis: understanding what happens during the crash and its origin.

As described at another blog post of ours, generating a time travel analysis trace from a Python Unicorn script is straightforward:

• Initially, import the Reven Unicorn Python library, then mirror the unicorn.mem_map and unicorn.mem_write methods with reven_unicorn.mem_map and reven_unicorn.mem_write, respectively.
• Finally, indicate when to start tracing, and that’s all! All registers and memory accesses/updates will be recorded automatically.

Since less than 40 000 instructions are emulated, the time travel trace generation does not affect Unicorn's emulation performance in our case: the emulation and trace generation are immediate. The generated TTA files collectively weigh less than 2 MB.

We are now ready to display this trace and understand the crash's origin!

Time travel analysis: vulnerability identification

The time travel debugger's user interface was used to dive into the generated trace:

• First, we gained a clear understanding of the events happening during program execution: there is a stack overflow leading to overwriting the saved PC.
• Second, we could pinpoint the precise location where the crash occurs, and find which bytes of the GBL file overwrite the PC value to get an effective flaw exploitation.

Opening the generated trace with the user interface brings us to the main interface as seen in the image below.

Moving forward, we will refer to the number displayed for each instruction in the “1. Disassembly” view as the 'transition'. This name is chosen because of the “2. CPU” view shows us the register values both before and after an instruction's execution.

When we start from the middle of the trace, we notice the same instruction repeating continuously, strongly suggesting that incorrect code is being executed.

We can quickly trace back to the last correctly executed instructions at transition 7468.

• It is located at address 0xA38 and belongs to the parser_parse function.
• It's a POP instruction restoring the PC value, along with other registers.
• Note that the SP registers switch from 0x20000CDC to 0x20000D00. We will use this information later on. It's clear that an incorrect PC value is restored. The key question now is the cause of this alteration: are we dealing with stack pointer corruption or a stack overflow?

The time travel debugger user interface provides a backtrace feature, which lists the sequence of function calls leading to the current execution point, helping to trace the flow of execution and diagnose issues. We used it to verify whether returning to the previous functions work correctly.

• It allowed us to observe that execution jumps from the advanceParser function to the parser_parse function during the transition 3890.
• The next transition, 3891, is a PUSH instruction and is particularly insightful:
  • It stores multiple values on the stack. It includes the PC register (which values 0x1549), using the LR register. It is saved at address 0x20000CF0.
  • The SP register shifts from 0x20000D00 to 0x20000CDC, perfectly reflecting the behavior of transition 7468. This suggests that the SP register remained intact and was not corrupted.

Therefore, it must be a stack overflow, which has overwritten the return PC value stored at 0x20000CF0. A quick bird’s-eye view of all the reading and writing operations affecting a specific memory location are possible. Ticking “Show access history of selection” in the “Memory History Access” immediately shows all the memory accesses.

• In transition 3891, there is a write operation: it occurs when the PUSH instruction saves the PC value (which is the return address for the function).
• However, after that, there are 4 transitions (6326, 6339, 6352, 6365) that erase the saved PC value byte by byte, ultimately resulting in the incorrect value 0x1FA93.
• We notice that for each transition, the same instruction is called at the same PC address 0x25DC: STRB R2, [R7, R3]. Also, we are inside the gbl_getData function.
• We can conclude that we are in a copy loop.

This conclusion is confirmed when opening the ELF code file in IDA, for example:

• We are indeed in a copy loop in the the gbl_getData function.
• As seen previously, the destination pointer of the write instruction is controlled by the R3 and R7 registers.
• After a brief inspection, it appears these registers are not altered: R3 serves as the loop counter, while R7 holds a base destination pointer derived from one of the gbl_getData function’s parameters.

We compiled the binary from the source code, and took this opportunity to link the call to the gbl_getData function, which is at the address 0xAE0, to the source code for a clearer view. We can use addr2line tool, for instance.

addr2line -fiC -e bootloader-storage-internal-single-512k.axf -a 0xae0                        
0x00000ae0
parser_parseApplicationInfo
gecko_sdk/platform/bootloader/parser/gbl/btl_gbl_parser.c:1304
parser_parse
gecko_sdk/platform/bootloader/parser/gbl/btl_gbl_parser.c:627

The following source code excerpt corresponds to the previous findings.

// gecko_sdk/platform/bootloader/parser/gbl/btl_gbl_parser.c:L626
case GblParserStateApplication:
 retval = parser_parseApplicationInfo(parserContext,
                                      &input,
                                      imageProperties);
 if (retval != BOOTLOADER_ERROR_PARSER_PARSED) {
   return retval;
 }
 break;

// gecko_sdk/platform/bootloader/parser/gbl/btl_gbl_parser.c:1295
static int32_t parser_parseApplicationInfo(ParserContext_t   *parserContext,
                                          GblInputBuffer_t  *input,
                                          ImageProperties_t *imageProperties)
{
 volatile int32_t retval;
 uint8_t tagBuffer[GBL_PARSER_BUFFER_SIZE];


 while (parserContext->offsetInTag < parserContext->lengthOfTag) {
   // Get data
   retval = gbl_getData(parserContext,
                        input,
                        tagBuffer,
                        parserContext->lengthOfTag,
                        true,
                        true);

gecko_sdk/platform/bootloader/parser/gbl/btl_gbl_parser.h:60
/// GBL parser buffer size
#define GBL_PARSER_BUFFER_SIZE              64UL

We conclude the following:

• We call the parser_parseApplicationInfo function, meaning that we are parsing the GBL Application Info Tag structure. You can also notice we do not see this function parser_parseApplicationInfo in the assembly code, simply meaning it was inlined.
• The structure content is retrieved with the gbl_getData function, and is stored in the local array tagBuffer which has 64-byte size.
• The length field in the GBL Application Info Tag, named parserContext->lengthOfTag in the source code, determines the amount to copy
• With the time travel analysis GUI, we can observe the calling parameters of gbl_getData:
  • The register R2 holds the address of the destination pointer 0x20000C98, located in the stack.
  • The register R3 holds the length value to be copied, which is 0x68 (104 bytes). This value exceeds the capacity of the destination buffer!

• However, the gbl_getData function responsible for this copy didn't check if the length field within the structure was smaller than or equal to 64 bytes. This unchecked copying ultimately led to the stack overflow vulnerability.

So, an attacker can exploit this by carefully crafting malicious GBL to overwrite data on the stack, especially the saved Program Counter (PC). This gives him the ability to hijack control of the program's execution flow and potentially take complete control of the system. Our findings lead us to the same conclusion presented in Quarkslab's article, even if there are a few differences between our analyzed crash and the one described in their article:

• Our overflow leads to corruption of several registers, including the PC register, while in the other article only R4 and R5 are modified because the corrupted structure length is set to a smaller value.
• In both cases, the interest of time travel analysis remains the same, the only difference being that we chose a more straightforward case to explain. The analysis of the other case could be the subject of another blog post.

The final piece of the puzzle is identifying the specific bytes within the GBL that corrupt the saved Program Counter (PC) value. It will enable crafting malicious GBL updates to jump to any location in the bootloader’s code.

• We know the destination pointer starts at 0x20000C98 and the PC value is saved at 0x20000CF0.
• By displaying this memory area within the time travel analysis GUI and comparing it to the GBL file content, we can identify the specific data that overwrote the PC value.
• The bytes of interest are highlighted in red in the image below.

The following provides a concise summary of our investigative work.

TL;DR

This work leverages Quarkslab blog post titled breaking secure boot on the Silicon Labs Gecko Platform which details how the CVE-2023-4041 vulnerability was identified and exploited through a fuzzing campaign. The blogpost provides an overview of the entire vulnerability discovery process as well as exploitation. However, a key question remains: how challenging was it to interpret a crash and trace its origin?

The purpose of this post is to see how time travel analysis could accelerate the identification and the understanding of the flaw. For this, we generated a time travel analysis trace from a fuzzing crash and analyzed. We could confirm that the tool was efficient to mitigate both the time and complexity for identifying the flaw.

• Recording the trace on Unicorn and resulting to meaningful a trace of less than 40 000 instructions and less than 2 Mbytes was not a great effort.
• The trace analysis then led to a rapid identification of the vulnerability's source, thanks to features like memory access history.
• Finally, it also enabled precise identification of the bytes responsible for hijacking control flow execution.

Initially reserved to complex analyses involving full systems emulation, this blogpost shows that the recent developments open more opportunities to leverage Time Travel Analysis for boosting security investigations and improve their depth and length in the IoT or connected object space.