Calculator Precision

A tolerance of ± 0.005 is used when determining how close a value is to an integer. I.e. 3.001! = 6 but 3.008! = NaN. The same tolerance is also used for comparisons. I.e. 3.034>=3.037 = 1  and 3.034==3.037 = 1 since they are within 0.005 of each other thus they are considered equal to each other.

You can bypass this behaviour for comparisons by multiplying the input by a large value such as 1E9 to ensure deviations are pushed outside tolerances. Make sure the result is within the range of a double which is (2 - 2^-52) * 2¹⁰²³.

The tolerance is currently set at compile time and cannot be adjusted.