Project

General

Profile

Actions

Bug #5273

closed

Float#round returns the wrong floats for higher precision

Added by marcandre (Marc-Andre Lafortune) over 12 years ago. Updated about 6 years ago.

Status:
Closed
Target version:
-
ruby -v:
r33186
[ruby-core:39267]

Description

Float#round returns the wrong floats for higher precision.

Rounding can be off:
2.5e-22.round(22) # => 2.0e-22

It can also return values that have extra decimals:
2.5e-31.round(31) # => 3.0000000000000003e-31
2.5e-36.round(36) # => 2.9999999999999998e-36

Both errors can occur at the same time too:
2.5e-39.round(39) # => 2.0000000000000002e-39

I believe these errors occur only for ndigits >= 22. For ndigits > 22, 10**(-ndigits) is not an exact Float (as 5 ** 23 won't fit in the 53 bits of a double's mantissa). For ndigits < 22, there should be enough bits left to avoid rounding errors.

For ndigits >= 22, the algorithm to use should be similar to the one used to convert doubles to string.

Hopefully, this is the last bug for #round with a given precision.


Related issues 1 (0 open1 closed)

Related to Ruby master - Bug #14635: Float#round(n) returns a wrong result when n is bigClosedActions
Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0