Benchmarking Side-Channel solutions... Why? How?

Basically, because we think our solution is good and we want to prove it!

Why?

At eShard, we are convinced that computational performance is a key point for side-channel attacks. We are confident that our side-channel solutions are among the bests and we want to demonstrate this to the side-channel community. And that, for both our open source tools with the scared library as well as our commercial products with our esDynamic platform.

Obviously, for open source tools such as lascar, jlsca, Dardevil, etc., we can download the sources and test by ourself. It is exactly what we did before releasing scared for the very first time, you can still check the benchmark results in this blog post.

But there are also a lot of commercial and closed source products, like Rambus DPA Workstation, Riscure Inspector, Secure IC Analyzr and some others. Sadly, we cannot evaluate their performances (without buying them!) and worst, there is no public records related to their performances. And we can add to that a variety of custom tools developed by laboratories or chip makers that are jealously kept secret.

To summarize, we cannot easily state if 'yes or no' our tools are good compared to those you are using. However, you can! And we are going to help you with that. Indeed, we now are providing the community with an easy way to test and benchmark eShard side-channel solutions. By doing that, we hope to contribute to a global improvement of the side-channel tools performances.

How?

A side-channel attack is performed on a dataset. Instead of providing several datasets, potentially heavy, with no guarantee to match your use cases, we have decided to totally abstract the data. This is why we recently released a new format / reader in our open source library estraces: the random reader. This random reader allows using scared to perform side-channel attacks without a real dataset, generally stored on a hard-drive.

The strategy is the following: as an analyst, you already know the performance of your side-channel tools on you favorite datasets (of course). The random reader allows you performing an attack on a fake dataset for which you can choose the amount and length of traces, the numerical data type as well as the range for your sample values. The memory footprint of this reader is very low, as the random data are generated on the fly.

Therefore, you can use this reader to replay your reference attack with eShard's scared library on the same hardware and then compare the timings with your own attack implementation.

Let's install the scared library:

# With pip
pip install scared
# or with conda
conda install scared

... and then customize the example scripts below, based on this random reader.

Random reader

As explained, the random reader is the fake dataset format to run scared side-channel analyses. This format was designed to be very versatile and flexible, and sufficiently efficient to not slow-down the analyses that use it.

As shown in the following code, this flexibility allows you to easily define a dataset that fits exactly your reference dataset by choosing:

• the number of traces
• the trace length, i.e. the number of samples per trace
• the range / interval of the generated data
• the numerical type (dtype) of the generated data
• an optional seed for repeatability
• a headers dictionary like for the other trace formats

Also, you can add some metadatas as **kwargs, by passing their length and type.

The following code defines a random reader, which could correspond to a dataset collected with an oscilloscope, targeting an AES, with one Million traces of ten thousand samples each. These parameters correspond to a 10GB dataset.

If the following code raises an ImportError exception, verify your version of the estraces library with: pip list | grep estraces.

The version number must be equal to or greater than 1.7.1.

from estraces.formats import read_ths_from_random

ths = read_ths_from_random(nb_traces=1_000_000, trace_length=10_000, 
                           interval=(-128, 128), dtype='int8',
                           plaintext=(16, 'uint8'),
                           ciphertext=(16, 'uint8'),
                           key=(16, 'uint8'))

Obviously the whole 10GB are not stored in memory. The data are generated when requested by the API. As its name suggests, the generated data are not coherent, but random. The goal is just to provide data relevant in terms of size, number and type.

The random reader was designed to be as fast as possible. It turned out that using true randoms is too slow. It was then decided to generate only two random vectors and an outer product to generate a matrix (see the code). With this method it takes 22 seconds to read the 10GB dataset instead of 101 seconds using the true random approach:

%%time

split_ths = ths.split(2500)
for part in split_ths:
    part.samples[:]

CPU times: user 2min 22s, sys: 26.8 s, total: 2min 49s
Wall time: 21.2 s

This corresponds to a read rate of 450 MB/s, which is consistent with modern storage technologies.

Benchmarks

It's now time to see if scared is efficient or not!

Take your favorite attack on your favorite dataset and reproduce it with scared. The following code defines:

• a CPA attack with the Hamming weight model,
• targeting the SubBytes output of an AES,
• on a dataset of 1 Million traces of 10,000 samples each,
• a convergence step of 100,000, i.e. the attack will compute 10 intermediate results.

import scared
from estraces.formats import read_ths_from_random
import time

ths = read_ths_from_random(nb_traces=1_000_000, trace_length=10_000, 
                           plaintext=(16, 'uint8'), 
                           ciphertext=(16, 'uint8'),
                           key=(16, 'uint8'))
attack = scared.CPAAttack(selection_function=scared.aes.selection_functions.encrypt.FirstSubBytes(),
                          model=scared.HammingWeight(),
                          discriminant=scared.maxabs,
                          convergence_step=100_000)
t0 = time.time()
attack.run(scared.Container(ths))
print(f'Execution time: {round(time.time() - t0, 2)} seconds.')

You will need to customize all the parameters to match your reference attack. For instance, if you don't want to perform a CPA, other types of attacks are available in scared: scared.DPAAttack, scared.MIAAttack or scared.ANOVAAttack for partitioned attack.

You may be also interested in benchmarking second-order attacks. The following code defines a second-order with:

• the centered product as combination function,
• traces of 141 samples, which leads to recombined traces of 10,011 samples,
• all the other parameters are the same as for the previous attack.

ths = read_ths_from_random(nb_traces=1_000_000, trace_length=141, 
                           plaintext=(16, 'uint8'),
                           ciphertext=(16, 'uint8'),
                           key=(16, 'uint8'))
attack = scared.CPAAttack(selection_function=scared.aes.selection_functions.encrypt.FirstSubBytes(),
                          model=scared.HammingWeight(),
                          discriminant=scared.maxabs,
                          convergence_step=100_000)
preprocess = scared.preprocesses.high_order.CenteredProduct()
t0 = time.time()
attack.run(scared.Container(ths, preprocesses=[preprocess]))
print(f'Execution time: {round(time.time() - t0, 2)} seconds.')

Again, each parameter can be customized to better fit your use case.

Conclusion

We have seen how to use the random reader to replay your reference attacks with scared. It allows you to compare, on the same machine, the computational efficiency of your actual side-channel analysis tool versus the scared library. Install it on your machine with pip install scared and try it yourself!

By doing that, we hope to contribute leveling-up and standardizing the performances of side-channel tools.

In the coming weeks, we will publish some benchmarks using the scared library on different hardware.

Meanwhile, you can contact us to know more about scared and our solutions for Side-Channel.