Frama-C news and ideas

To content | To menu | To search | Frama-C Home

Tag - sulfur

Entries feed - Comments feed

Thursday, February 15 2018

Analysis scripts: helping automate case studies, part 2

In the previous post, we used analysis-scripts to prepare our analysis of Recommender. In this post, we will run EVA on the code and see how the iterative refinement of the analysis can be done. We assume the reader has performed all of the steps in the previous post, and is currently in the directory containing the GNUmakefile created previously.

Running the first EVA analysis

Running make main.eva will run the EVA plug-in in the console, and then output some metrics information. This is useful for scripts and experienced users. For now, running make main.eva.gui is more useful: it will run the analysis and then open the results in the GUI.

A *.{parse,eva}.gui target in analysis-scripts only run parsing/EVA if necessary; if it has already been run, and no source files or analysis parameters were modified, it will directly open the GUI.

In the GUI, in the Properties panel (you have to click Refresh after loading the GUI) if we inspect only Unknown/Invalid properties, we will notice 148 of them, several related to finiteness of floating-point values, as well as many related to variable initialization.

Note that the results of each analysis are stored in the main.eva directory, and also copied to a timestamped directory in the form main_YYYY-MM-DD_hh-mm-ss.eva. Thus main.eva always contains the latest analysis, while the timestamped copies are useful to retrieve past results and compare them, e.g. using meld's folder comparison view.

Iterating on EVA analyses

Let us use the Loop Analysis plug-in to obtain more precise results. Since analysis-scripts already created a Frama-C save file (extension .sav by default), we just need to load it and add -loop:

frama-c -load main.eva/framac.sav -loop > main.slevel

Open the main.slevel file with a text editor. It contains the output of the Loop Analysis plug-in: some informative text, and then a line that says:

[loop] Add this to your command line:

Followed by several command-line arguments related to the use of slevel. Let us erase everything up to and including the Add this ... line, and then include the file contents in the EVAFLAGS variable of our GNUmakefile:

EVAFLAGS    += $(shell cat main.slevel | tr '\\' ' ')

Note that we use the tr command-line utility to remove the backslashes from the output of Loop Analysis.

If we re-run make main.eva.gui, the dependencies managed by Make will re-run the analysis because its command-line changed, but not the parsing, since it was not impacted.

This time, we obtain 114 unknown/invalid properties.

Appeal to brute-force

Since this is a test case that runs very quickly, and because Loop Analysis' heuristics are not perfect, we can allow ourselves to try a somewhat aggressive -slevel. Let us remove the heuristics given by Loop Analysis, and instead use a global slevel of 500:

EVAFLAGS    += -slevel 500

The reason why we remove Loop Analysis' results is that, in some cases, it forces merging of states to avoid slowing down the analysis, and those settings take precedence over the global slevel.

The analysis will be much longer this time, but should still terminate in a few seconds. Given such a high slevel, we are lucky that it terminates at all. We will obtain 86 unknown/invalid properties this time.

Extra hypotheses for more precision

There are several options in EVA that allow a more precise analysis of less-than-ideal code that can be found in the wild. Most of them require knowing some details about the semantics of C, but others can be used more leniently, to help get a better idea of the code under analysis, before some serious analysis time is spent. One of them is -no-val-malloc-returns-null. This option is only useful when the code contains dynamic memory allocation, and what it does is to consider that memory allocation never fails.

By default, the memory allocation built-ins used by EVA suppose that memory allocation can fail, as stated in the standard (i.e. malloc can return NULL, for instance when there is not enough available memory). However, many code bases fail to test for such cases; which is admittedly a relatively rare situation, especially on a system that allows memory overcommitment. The -no-val-malloc-returns-null thus adds a new hypothesis to the underlying analysis ("... and considering that memory allocation never fails..."), in exchange for a more precise result.

In Recommender, we notice that there are some spots where malloc is tested for NULL (e.g. in src/learned_factors.c:47), but others where no such test is performed (e.g. in src/learned_factors.c:67). Thus adding this option should (hopefully) result in fewer alarms. Our EVAFLAGS line becomes:

EVAFLAGS    += -slevel 500 -no-val-malloc-returns-null

Indeed, running the analysis again results in only 56 unproved properties. However, it also reveals another issue.

Tracking red alarms

Opening in the GUI the result of the last iteration of the analysis, if we scroll through the main function, we see that the analysis has stopped (became red) before returning from the call to recommend_items. Jumping to its definition, we see a loop where j should go from 0 to 2 (the value contained in (estim_param->tset)->items_number)), but inspecting the value of j in the while condition shows {0; 1}. This means that the third iteration never took place, its trace cut short by some evaluation that resulted in a definitely invalid property.

There are several ways to proceed here:

  • One can follow the "red path" in the GUI, jumping inside reachable functions in the GUI until the limit of the red path is found;

  • Alternatively, in this case we have a warning (in the Messages tab) that says that all target addresses were invalid, which often indicates an issue with the evaluation of a state;

  • Finally, one can prioritize alarms in the Properties tab, by checking only Invalid properties at first, before including Unknown ones.

In this case, all three would end up around the same source line (src/binary_heap.c:153), though this is not always true.

Spoiler alert: the Chlorine 17 release of Frama-C will include an extra panel, Red alarms, which will provide another way to get to properties that warrant extra attention when starting an analysis.

Note that, whenever the C code contains calls to assert(), Frama-C generates a call to __FC_assert whose precondition is requires \false. This leads to an Invalid alarm in the Properties panel, however this does not indicate that the source code necessarily has undefined behavior: it may be due to an imprecision in the assertion condition, which if possibly true, leads to the call being made, which will itself definitely indicate an error. But if the call may not happen, then the assertion should be treated like an Unknown alarm.

Spoiler alert: in Frama-C Chlorine (17), the behavior of assert will be changed to more closely match the usual approach: an Invalid alarm will only be generated when the assertion is definitely violated.

Back to the Invalid alarm: if we inspect the source code in question, we will see that the only valid index for the buffer bheap->buffer is 0, due to the ACSL assertion inserted by EVA:

/*@ assert Value: index_bound: bheap->filled_elements < 1; */
bheap->buffer[bheap->filled_elements] = value;

However, filled_elements is 1 in this call, which seems like an error. But why is it there? While it is impossible to know exactly the intent of the author of the code, by inspecting the definition of the buffer, we can obtain some clues. In the Information panel, if we inspect bheap, we see that:

Variable bheap has type `binary_heap_t *'.
It is a formal parameter.
It is referenced and its address is not taken.

And then if we click binary_heap_t itself, we obtain its type information:

Type information for `binary_heap_t':
(sizeof 16, defined at src/binary_heap.h:53)
 struct binary_heap {
    size_t max_size ;
    size_t filled_elements ;
    int (*value_comparer)(void const *, void const *) ;
    void *buffer[1] ;

Finally, clicking the source line will take us to the definition in the binary_heap.h header, which contains some useful comments:

 * binary_heap:  The heap is tree based data structure
 * Members:
 *   max_size        The number of elements in the binary heap
 *   filled_elements The number of inserted elements so far
 *   buffer          The buffer is a dynamically allocated array
 *                   containing the heap's elements
typedef struct binary_heap
  size_t          max_size;
  size_t          filled_elements;
  bh_value_cmp    value_comparer;
  void*           buffer[1];
} binary_heap_t;

From the comment, it seems that the code is implementing the pattern of a flexible array member, where the last element of the binary_heap_t structure should be an incomplete array type. However, its current declaration corresponds to that of a fixed-size array, which according to the C standard cannot be accessed beyond its static bounds. We can fix this by using the proper flexible array member notation defined in the C99 standard.

Syntactic fixes

By replacing void* buffer[1]; with void* buffer[]; in src/binary_heap.h, we define a proper flexible array member. As a consequence, the code can access all elements of the dynamically-allocated array, as long as enough memory has been allocated for them.

You may notice that, after modifying the .h file and re-running make main.eva.gui, Make will not re-parse nor re-run the analysis, it will simply open the GUI again. This is because the .h file dependencies are not tracked by the Makefile rules: they were never included in the list of .c sources of main.parse (because such headers are directly included via preprocessor directives), so Make has no way to know about these implicit dependencies. You can remedy that by forcing Make to unconditionally make all targets, adding -B to the command line arguments.

After doing so, everything is recomputed, and we obtain … almost the same alarm.

This time, the alarm happens on the third iteration of the loop, instead of the second one: progress! … But not much. There is still a problem when accessing the last element of the array. To find out why, we can use the GUI again. However, this time the Invalid alarm is no longer present; the problem is more subtle. Still, the warning about all target addresses were invalid is still there. It points to this statement:

    /*@ assert
        Value: mem_access: \valid(&bheap->buffer[bheap->filled_elements]);
    bheap->buffer[bheap->filled_elements] = value;

By inspecting bheap->filled_elements, we see its possible values were {1; 2} before the statement, but {1} afterwards. This indicates that index 2 is not valid for this memory access. But why? We can check the location being accessed, to see its validity.

If we inspect the values for bheap, we see, in the Values tab:

bheap -> {{ (binary_heap_t *)&__malloc_init_binary_heap_l42 }}

Then, if we left-click on __malloc_init_binary_heap_l42, the Information panel will display some extra lines about this value in particular, with a clickable link pointing to its syntactic information. Clicking on that link will display the type of the dynamically allocated block:

Variable __malloc_init_binary_heap_l42 has type `char [23]'.

For dynamically allocated bases, EVA does not always have access to the original type intended for the allocated base, so it uses heuristics to restore that information. When such heuristics fail, it resorts to the "catch-all" char [].

"23" is a somewhat unusual size (not a multiple of the word size). Let us investigate where that number came from.

Using Studia

Right-clicking on __malloc_init_binary_heap_l42, in the Information tab, will display a context menu, the same one as if we had right-clicked some lvalue on the Cil code. We will use the Studia plug-in, in particular its Writes feature, to identify all points in the source code where this memory location was written prior to the statement indicated by the warning.

Studia will compute and highlight the statements in the code, and also add an extra column to the filetree display (upper left corner in the GUI), indicating all functions directly writing to the memory location (indicated by a 'tick' symbol in the Studia column) and the functions indirectly writing to it, that is, the callers of the former (indicated by an arrow symbol in the Studia column) as well as their own callers, recursively.

The functions writing to buffer are: balance_heap, init_binary_heap and insert_binary_heap. The first and the last write to fields in the buffer structure, but only init_binary_heap actually allocated the memory.

Well, I guess you could have directly inferred that from the variable name, which serves this exact purpose, but then I wouldn't have had an excuse to shamelessly plug Studia, one of the new features in Frama-C 16 Sulfur, would I?

Inside init_binary_heap lies our answer: the computation of the size of the bheap includes a -1, which is due to the fact that the non-flexible array member notation already had a byte allocated for the array, but our proper C99-compatible version no longer includes this byte. So the malloc'ed memory was 1 byte shorter, which led to the invalid memory access. So all we have to do is to fix the computation here (and take note of the fact that modifying other people's code without properly understanding it can lead to bugs).

At last … unable to progress

This time, we re-run the analysis (no need to use -B this time, since the .c file is tracked in the dependencies of our GNUmakefile), expecting to triumph over the syntactic distractions. However, we are greeted with a somewhat unpleasant surprise:

src/binary_heap.c:118:[value] warning: detected recursive call
    (balance_children <- balance_children :: src/binary_heap.c:135 <-
                         pop_binary_heap :: src/recommender.c:95 <-
                         recommend_items :: test/test.c:108 <-
    Use -val-ignore-recursive-calls to ignore (beware this will make the analysis
[value] user error: Degeneration occurred:
    results are not correct for lines of code that can be reached from the degeneration point.

We should have seen it coming: binary heaps are fond of recursion. Our coverage did improve from the previous analysis (from about 75% to about 85%), but we now hit a harder obstacle. To deal with this, we'll have to stub the balance_children function, possibly over-approximating its behavior, or rewrite an equivalent, iterative version of the function. In either case, such transformations are out of the scope of this tutorial.

Conclusion and bonus features

In this tutorial, we showed how to use analysis-scripts, presenting most of its features in a quick usage scenario. There are still a few tricks up its sleeve which were not mentioned:

  • flamegraph.txt: this file is computed by option -val-flamegraph, and it produces a Flamegraph of the analysis performed by EVA. This graph is useful when profiling larger case studies, to quickly visualize where the analysis is spending most of its time. This can indicate functions where less slevel or ACSL stubs might help.

  • metrics.log: its output is displayed after running a .eva target. The coverage estimation is a quick indicator of progression in the analysis during the refining stage, i.e. sometimes when an analysis is made more precise, the number of alarms may actually increase, and the reason may be that the coverage has improved, so that more code is actually being analyzed, which might explain why now there are more alarms.

  • alarms.csv: a CSV report with the same contents of the Properties panel in the GUI, obtained via the Report plug-in. Very useful for scripts.

  • stats.txt: similar to the one present in the .parse directory; however, since parsing is usually very fast when compared to an EVA analysis, this version of the file is the one that is actually useful most of the time. In particular, if you want to compare the execution time of a new parameterization of the analysis, you just need to look at the user_time line. Very useful when you realize you forgot to type time make instead of make.

We hope this tutorial was useful to you and that the next Frama-C release will make things even easier! Don't forget to check some usage examples in open-source-case-studies, and please consider proposing your own case studies via Github issues.

Thursday, January 25 2018

Analysis scripts: helping automate case studies, part 1

(kindly reviewed by T. Antignac, D. Bühler and F. Kirchner)

Among Frama-C 16 Sulfur's new features, one of them is dedicated to help setting up and iterating case studies. In this series of posts, we will explain how to use such scripts to save time when starting a new case study with EVA.

This series supposes some familiarity with EVA, as in, its basic usage is not detailed. Reading A simple EVA tutorial or the tutorial in the EVA user manual should be sufficient.

In this first part, we will explore the basics of the analysis-scripts framework and its use on an open source project. We will see how to get the preprocessor flags needed to prepare an analysis template. This template is in fact a Makefile which will be used to set up the analysis. It will enable us to define the sources to parse by editing its targets.

In the next post, we will see that, once the set-up is done, the analysis steps are similar to the ones presented in previous EVA tutorials: run a simple analysis, then refine, improve the precision, iterate. Once an analysis has been run, we will see how to track red alarms to improve the coverage. This will lead to some syntactic fixes that will allow the analysis to go further.

Analysis scripts

The release 16 (Sulfur) of Frama-C includes a small set of files, called analysis scripts, whose purpose is to help starting new analyses with Frama-C. It is currently focused on analyses with EVA, but other plug-ins such as WP and E-ACSL may incorporate it as well.

The new files are available in the analysis-scripts of Frama-C's share directory. Use frama-c -print-share-path to obtain its exact location. This directory currently contains a README, two small Bash scripts and a Makefile ( which contains the bulk of it.

Note: the mentions fcscripts, the old name of this set of files (it was already being used by the open-source-case-studies Git repository, under that name, but now it is part of the Frama-C release). The upcoming Frama-C release will update that.

analysis-scripts relies on some GNU-specific, Make 4.0+ features. This version of Make (or a more recent one) is already available on most Linux distributions and on Homebrew for MacOS, but in the worst case, downloading and compiling GNU Make from source should be an option.

Basically, what these files provide is a set of semi-generated Makefile targets and rules that automate most of the steps typically used by Frama-C/EVA developers to set up a new case study. The scripts themselves are extensively used in the open-source-case-studies repository, which serves as use example.

Using analysis scripts

To illustrate the use of analysis-scripts, we picked one project from ffaraz's awesome-cpp repository: GHamrouni's Recommender, a library from the Machine Learning section.

We start by cloning the repository1:

git clone

1 Note: at the time of this posting, the latest commit is 649fbfc. The results presented later in this tutorial may differ in the future. You can select this commit via git checkout 649fbfc in case this happens.

Then, inside the newly-created Recommender directory, we look for its build instructions. Many C open-source libraries and tools are based on autoconf/configure/make, which may require running some commands before all headers are available (e.g., ./configure often produces a config.h file from a template). Frama-C does not require compiling the sources, so in most cases you can stop before running make. However, since Frama-C does require preprocessor flags, you can use existing Makefiles to cheaply obtain that information.

Obtaining preprocessor flags from a compile_commands.json

In order to find out which preprocessor flags are needed for a given program, you can use tools such as CMake or Build EAR to produce a JSON compilation database, commonly called compile_commands.json. This is a JSON file which contains the list of commands (including all of their arguments) to be given to the compiler. It contains many options which are irrelevant to Frama-C (such as warning and optimization flags, typically -W and -O), but it also contains preprocessor flags, mostly -I and -D, which we are interested in.

The compile_commands.json file can be produced as follows:

  1. If the project is based on CMake, add -DCMAKE_EXPORT_COMPILE_COMMANDS=ON to the cmake command-line, e.g. instead of cmake <source dir>, use cmake <source dir> -DCMAKE_EXPORT_COMPILE_COMMANDS=ON.

  2. If the project is based on Makefile, use Build EAR. After Build EAR is installed, you just have to prefix make with bear: typing bear make is usually enough to obtain the JSON database.

Note: Frama-C 17 (Chlorine) includes option -json-compilation-database, which allows using compile_commands.json directly, rendering the next step unnecessary.

Once you have the file, simply grep it for -I and -D flags, e.g:

grep '\-I\|\-D' compile_commands.json

Those flags should be added to the preprocessor flags in Frama-C's Makefile (described in the next section), CPPFLAGS.

Note that these recommendations may not work in complex setups. Manual inspection of the commands used during compilation might be necessary to obtain all necessary flags.

Preparing an analysis template

We will create a Makefile for Frama-C, to manage dependencies and help re-run analyses. In order to avoid having to type make -f <name> each time, we will name it GNUmakefile, for the following reasons:

  1. GNU Make gives preference to GNUmakefile over Makefile if both exist, so the default file used when typing make will be ours, even if the project already has its own Makefile;

  2. This avoids having to rename/overwrite existing makefiles (or, worse, having Frama-C's Makefile erased when re-running ./configure);

  3. The analysis-scripts Makefile already relies on some features specific to GNU Make, so there is no compatibility downside here.

If you want to name your Makefile otherwise, just remember to always add -f <your makefile name> to the make commands presented in this tutorial.

Our GNUmakefile will be created with content based on the template available on Frama-C's Github repository.

In this tutorial, we consider that Frama-C is installed and in the PATH to keep the template concise.

include $(shell frama-c-config -print-share-path)/analysis-scripts/

# Global parameters
FCFLAGS     +=


# Default targets
all: main.eva

# Input files
main.parse: <TO BE COMPLETED>

The essential element to complete this template is the list of files to be parsed. Other arguments, such as flags for the C preprocessor (CPPFLAGS), for the Frama-C kernel (FCFLAGS) and for the EVA plug-in (EVAFLAGS) are also to be filled by the user, when necessary.

Finally, note that the target name (main) is completely arbitrary. You can have multiple targets if your code contains multiple main functions. The important part is the use of the suffixes .parse and .eva, which are hard-coded in the analysis-scripts' Makefile to generate targets with the appropriate dependencies and rules.

The .parse suffix is used by analysis-scripts to set the list of source files to be parsed. It is associated to an .eva target which runs the EVA analysis. This target is generated by analysis-scripts itself; we just need to tell the Makefile that it should be run when we run make all, or simply make, since all is the first rule in our Makefile.

Setting sources and testing parsing

The list of source files to be given to Frama-C can be obtained from the compile_commands.json file. However, it is often the case that the software under analysis contains several binaries, each requiring a different set of sources. The JSON compilation database does not map the sources used to produce each binary, so it is not always possible to entirely automate the process. You may have to manually adjust the sets of files to be given to Frama-C. For a whole-program analysis with EVA, in particular, you might want to ensure that there is exactly one file defining a main function.

In the case of Recommender, since it is a library, the src directory does not contain a main function. However, the test directory contains two files, each of them defining a main function. In this tutorial, we will use the test.c file as the main entry point of the analysis. We could also have used the -lib-entry option on one or more functions in the library. More advanced users of Frama-C may prefer this option though we will keep it simple and use the main function in test.c as unique entry point in this tutorial.

Therefore, the list of sources to be filled in the main.parse target is the following:

main.parse: src/*.c test/test.c

We can test that Frama-C is able to parse the sources:

make main.parse

If you are strictly following this tutorial, you should have the following error:

[kernel] Parsing test/test.c (with preprocessing)
test/test.c:1:25: fatal error: recommender.h: No such file or directory
 #include "recommender.h"
compilation terminated.

This is because we never included the actual -I lines that we found in the compile_commands.json file. Note that they include flag -I../../src/, which is relative to one of the subdirectories in tools/*. Since our Makefile (and thus Frama-C) will run relative to the base directory in Recommender, the actual include directive needs to be -Isrc, which we add to CPPFLAGS:


Running make main.parse now should succeed. Run it again. You will notice that nothing is done: thanks to the dependencies managed by analysis-scripts, unless you modify one of the source files, or the flags given to Frama-C (CPPFLAGS or FCFLAGS), Frama-C will not waste time reparsing the results: they have already been saved inside the main.parse directory.

Contents of the .parse directory

For each .parse target, analysis-scripts will create a corresponding directory with several files:

  • command: the command-line arguments used by Frama-C;

  • framac.ast: the pretty-printed normalized AST produced by Frama-C;

  • stats.txt: performance information (user time and memory consumption);

  • warnings.log: warnings emitted during parsing;

  • framac.sav: a Frama-C save file (a binary) with the result of parsing;

  • parse.log: the entire log of the parsing (includes the warnings in warnings.log).

All of these files are used internally by the analysis scripts or some other tools (notably for regression testing and profiling purposes), however only the last 2 files are occasionally relevant for the end user:

  • framac.sav can be used with frama-c -load, for instance when trying different plug-ins;

  • parse.log contains a copy of all messages emitted during parsing.

If we want to load the parsed files in the Frama-C GUI, we can use either frama-c -load framac.sav, or more conveniently, make main.parse.gui. The advantage of the latter is that it will generate the .parse directory if it has not been done already.

This concludes the first part of this tutorial. In the next post, we will run EVA using our Makefile, then iterate to improve the analysis.