This article is part of series “System Under Test”. It provides an overview of the test suites that are used by LLVM project to maintain a quality of its libraries and tools on a high level.
What is LLVM about?
http://llvm.org says that
The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.
It is not that much I can add here besides one link:
The Architecture of Open Source Applications: LLVM by Chris Lattner.
which sheds light on compilers in general and LLVM particularly.
LLVM is rather an umbrella project than a single project. It consists of compiler, debugger, linker, assemblers for several CPUs and of the most important - its Core: back-end and middle-end.
In this article I refer to LLVM as a back-end + middle-end, rather than the whole umbrella (that would be too much for one post).
LLVM is a huge project. Therefore it has quite a few groups of tests: unit tests, regression tests, performance tracking and fuzzing tests. Since the project is not trivial the tools used for testing are mostly written from scratch and are part of LLVM project. Though, I wish I could use some of them without having LLVM as their dependency.
Amount of unit tests is pretty small comparing to regression tests. One reason behind that decision is that LLVM internals constantly change all the time. Supporting tests under such conditions is very time consuming. However there are still parts that do not change very often, that is they are good target for unit testing. These tests are located in ‘unittests’ directory.
They can be run using
Showing this beautiful output:
As you can see there are about 1,5k tests, and that leads to a pretty short execution time: ~30 seconds using 4 threads.
Unit Tests are written using Google Test framework. Here is an example of a simple test:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Pretty trivial. Let’s move forward and look at another, more interesting group of tests.
The aim of this test suite is to verify the output of different tools, hence the internals can change separately from tests, making support less time-consuming.
This test suite located in
It is the largest group of tests used in LLVM. It is 10 times bigger than Unit Tests: ~15k vs ~1,5k.
It takes about 4 minutes to run on my machine using 4 threads.
The output is pretty similar to one above:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
This test can be split into three parts:
Run command (the top line):
Expectations (the bottom line):
The rest (LLVM IR in the middle) is the body.
All tests in this suite have one or more ‘run’ command.
lit uses set of rules to substitute the string into real runnable command. Substitutions are either built-in (such as
%s) or configurable (such as
%s with the full path to a file under test, e.g.:
Configurable substitutions however are taken from
lit.cfg file, which is basically a Python script.
For example, this config says that
%lli is to be replaced with
Having these parameters in place
lit will run the test using this command:
Which will interpret
frem.ll using LLVM Interpreter (
lli) and pass the output to the
FileCheck in turn takes two arguments: filename with expectations and input that needs to be examined.
Summary of this example:
The test interprets the body (LLVM IR) from
~/llvm/test/ExecutionEngine/frem.ll using LLVM Interpreter (
/usr/local/bin/lli) and checks if the output of interpretation contains string
Double value: 2.0.
FileCheck have lots of useful options. Consider looking at documentation to learn more.
Performance is one of the most important goals of any software. LLVM is not an exception.
Here LLVM also uses custom tool - LNT. This tool was initially written to be used inside LLVM, but its design allows it to be usable for performance testing of any other software.
Performance tests suite is not a part of LLVM source tree. It has to be fetched separately. It is a set of programs that are compiled and executed to track performance changes.
LNT the test suite can be used within CMake as described in LLVM
At the moment of writing this article external test suite contains 485 test cases. It takes ~5 minutes to run them using
Another powerful technique used to increase quality of LLVM is Fuzz Testing.
Here is an example of a fuzz test:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
LibFuzzer generates huge amount of different inputs using Genetic programming and calls
LLVMFuzzerTestOneInput within each input.
This test then tries to parse the input as an assembly. The parser should not crash.
At the moment there are two targets for fuzz testing within LLVM source tree:
llvm-mc-fuzzer. They are located in
LLVM uses a few test suites for different needs. There are ~1,5k Unit Tests, ~15k Regression Tests. It takes ~4-5 minutes to run both tests in Debug mode on 2 y/o MacBook Pro using 4 threads.
LLVM uses Fuzzing Tests to prevent system from abnormal exit when erroneous input received.
LLVM has out-of-source-tree test-suite for performance tracking.
LLVM mostly uses custom tools for testing.