## Definition of FLT_EPSILON

By pascal on Thursday, May 9 2013, 11:12 - Permalink

## Correct and wrong definitions for the constant FLT_EPSILON

If I google “FLT_EPSILON”, the topmost result is this page with this definition:

FLT_EPSILON the minimum positive number such that 1.0 + FLT_EPSILON != 1.0.

No, no, no, no, no.

I don't know where this definition originates from, but it is obviously from some sort of standard C library, and it is wrong, wrong, wrong, wrong, wrong. The definition of the C99 standard is:

the difference between 1 and the least value greater than 1 that is representable in the given floating point type, b^(1−p)

The GNU C library gets it right:

FLT_EPSILON: This is the difference between 1 and the smallest floating point number of type float that is greater than 1.

## The difference

On any usual architecture, with the correct definition, FLT_EPSILON is `0x0.000002p0`

, the difference between `0x1.000000p0`

and the smallest float above it, `0x1.000002p0`

.

The notation `0x1.000002p0`

is a convenient hexadecimal input format, introduced in C99, for floating-point numbers. The last digit is a `2`

where one might have expected a `1`

because single-precision floats have 23 explicit bits of mantissa, and 23 is not a multiple of 4. So the `2`

in `0x1.000002p0`

represents the last bit that can be set in a single-precision floating-point number in the interval [1…2).

If one adds FLT_EPSILON to `1.0f`

, one does obtain `0x1.000002p0`

. But is it the smallest `float`

with this property?

#include <stdio.h> void pr_candidate(float f) { printf("candidate: %.6a\tcandidate+1.0f: %.6a\n", f, 1.0f + f); } int main(){ pr_candidate(0x0.000002p0); pr_candidate(0x0.000001fffffep0); pr_candidate(0x0.0000018p0); pr_candidate(0x0.000001000002p0); pr_candidate(0x0.000001p0); }

This program, compiled and executed, produces:

candidate: 0x1.000000p-23 candidate+1.0f: 0x1.000002p+0 candidate: 0x1.fffffep-24 candidate+1.0f: 0x1.000002p+0 candidate: 0x1.800000p-24 candidate+1.0f: 0x1.000002p+0 candidate: 0x1.000002p-24 candidate+1.0f: 0x1.000002p+0 candidate: 0x1.000000p-24 candidate+1.0f: 0x1.000000p+0

No, `0x0.000002p0`

is not the smallest number that, added to `1.0f`

, causes the result to be above `1.0f`

. This honor goes to `0x0.000001000002p0`

, the smallest float above half FLT_EPSILON.

Exactly half FLT_EPSILON, the number `0x0.000001p0`

or `0x1.0p-24`

as you might prefer to call it, causes the result of the addition to be exactly midway between `1.0f`

and its successor. The rule says that the “even” one has to be picked in this case. The “even” one is `1.0f`

.

## Conclusion

Fortunately, in the file that initiated this rant, the value for FLT_EPSILON is correct:

#define FLT_EPSILON 1.19209290E-07F // decimal constant

This is the decimal representation of `0x0.000002p0`

. Code compiled against this header will work. It is only the comment that's wrong.

## Comments

Pascal, once you finish with the fun rants and the low-level hacks, where are you going with this?

I took some numerical analysis in college and found it really difficult. Also, the arguments made in those classes were totally disconnected from the behavior of IEEE floats.

These days when I talk to numerical people they say FP issues are not that big of a deal because they structure their computations to avoid them. This involves domain knowledge, algorithm choice, and also low-level coding issues. Basically a rather large body of knowledge and experience seems to be necessary to write good FP code. I suppose it goes without saying that many real FP codes are created by people lacking this knowledge.

Have you read _Safer C_? If not, you must read it. There's a hilarious story early in the book where Hatton discovered that a weather simulation was dropping all terms except the highest-order one on every second simulation step. When fixed, it made very little difference.

I'm just saying random things. But really I want to know what's going to come out of your work here. Is it perhaps possible to capture and reason about those concerns that the knowledgable numerical people claim to be in control of?

Hello, John.

The principal reason I often find myself writing light-hearted remarks about floating-point is that floating-point is not my day job. In other words, I am not necessarily going anywhere with this. I am probably just having fun.

I took some numerical analysis classes, too. That was very much like math and not the kind of math I liked. My recent affair with floating-point must come from the progressive realization that on the other hand, at the lowest level, floating-point is very much computer stuff. If we just remove the layers of abstraction, think about it in binary, and pay attention to details, everything I have written on the subject is fairly obvious, this post that we are commenting on in particular. IEEE 754 is the ultimate bit-twiddling adventure.

Some of my friends are physicists, and apart from that, a few weeks ago I had the chance to have a short chat with someone who had worked at CERN. I would say they have the right level of understanding of the pitfalls of the format.

My encounter with someone from CERN meant that my attention was grabbed when, while browsing for reading material the homepage of the floating-point group at my alma mater, I found a file named cern.pdf.

Here is the introduction I wrote when I submitted the slides to reddit/programming (a good place for programming-related links that I discovered when one of the posts from this blog was listed there): http://www.reddit.com/r/programming/comments/1dusl3/computing_with_floating_point_its_not_dark_magic/c9u1byy

And the slides were these: http://lipforge.ens-lyon.fr/www/crlibm/documents/cern.pdf

I think these give a good sense of direction. It's not my direction… I am not on this ship. I am on the shore making snarky comments.

I am putting “Safer C” on my reading list.

John —

There are good reasons why numerical analysis courses tend to be disconnected from low-level IEEE-754 numerics. First, most numerical analysis courses historically have been taught by people whose experience predates IEEE-754; in the wild-west days of numerics, generic backwards error analysis was about the best you could hope to do if you wanted your analysis to be useful for more than one specific architecture, and it's also sufficient for the stability analysis of most of the classical applied math algorithms, so there’s very little incentive to get into the nitty-gritty details in all but the most specialized courses. There are also very few good textbooks (I like the Handbook of Floating-point Arithmetic, but it’s probably only suitable for a course dedicated to the subject); and relatively few experts in academia; much of the knowledge is passed down within industry.

I think that “FP issues not being a big deal” really says more about the success of IEEE-754 than the cleverness of modern numerical programmers. It is imperfect (and imperfectly adhered to), but the situation is *vastly* better than it was before standardization. There was a time when vendor-supplied math libraries suffered catastrophic cancellation and only delivered a couple bits for some arguments, when subtraction was performed without a guard digit, when hexadecimal floating-point and other wacky formats were widespread, when rounding couldn’t be trusted, and when floating-point arithmetic couldn’t be assumed to be closed. Those sorts of serious issues resulted in nearly uncontrollable numerical programming environments, forced programmers to consider every eventuality, and resulted in lots of terrible bugs. FP arithmetic today is imperfect, but for most users (even unsay ones) most of the time, it behaves pretty much as expected.