Bug #22246: Rational#to_f incorrectly rounds when the numerator is a bignum that cannot be represented exactly by a double, and the denominator is a fixnum

• Status: Open
• Assignee: -
• Target version: -
• ruby -v: 4.0.6
• Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN

[AndrewDile (Andrew Dile)]

Rational(65803600513127829623, 10**12).to_f

The bignum numerator cannot be exactly represented by a double, but the code in rb_big_fdiv_double rounds it to a double (dx) before dividing by the fixnum denominator of 10^12, resulting in 65803600.51312783, instead of the correctly rounded value, 65803600.513127826. The isinf(dx) test does not cover this case. Moreover, not all fixnums (in 64-bit systems) can be represented exactly as doubles.

[AndrewDile (Andrew Dile)]

• Subject changed from Rational#to.f incorrectly rounds when the numerator is a bignum that cannot be represented exactly by a double, and the denominator is a fixnum to Rational#to_f incorrectly rounds when the numerator is a bignum that cannot be represented exactly by a double, and the denominator is a fixnum

[nobu (Nobuyoshi Nakada)]

$ ruby -e 'x = 65803600513127829623; [65803600.51312783, 65803600.513127826].each {|n| p ((n*10**12).to_i-x).abs}'
3977
4215

Isn't the current result closer?

[AndrewDile (Andrew Dile)]

In your code example, the multiplication of n by 10**12 uses floating point arithmetic, which introduces a rounding error. Instead, it needs to convert n to an exact rational number before doing the multiplication.

x = 65803600513127829623; [65803600.51312783, 65803600.513127826].each {|n| p ((n.to_r*10**12).to_i-x).abs}
3981
3469

[byroot (Jean Boussier)]

• Related to Feature #21308: Replacing the Float#to_s (dtoa.c) implementation with a modern algorithm added

[rgburger (Bob Burger)]

This bug is not related to Feature #21308 except for the same author. The implementation of Rational#to_f does not perform any base conversions.

The bug is caused by the assumption that a floating-point conversion of a bignum numerator followed by a floating-point division by a fixnum denominator produces the correctly rounded result. The relevant code is in rb_big_fdiv_double:

    dx = big2dbl(x);
    if (FIXNUM_P(y)) {
        dy = (double)FIX2LONG(y);
    [...]
    v = rb_flo_div_flo(DBL2NUM(dx), DBL2NUM(dy));

In order for rb_flo_div_flo to produce the correct result, dx and dy must exactly represent x and y.

[byroot (Jean Boussier)]

This bug is not related to Feature #21308 except for the same author.

Yes, I confused it with another ticket, that's why I deleted my comment, sorry.

[byroot (Jean Boussier)]

• Related to deleted (Feature #21308: Replacing the Float#to_s (dtoa.c) implementation with a modern algorithm)

[rgburger (Bob Burger)]

Consider the related code in fix_fdiv_double, which demonstrates a check for the denominator y being a fixnum that's exactly represented by a double.

static double
fix_fdiv_double(VALUE x, VALUE y)
{
    if (FIXNUM_P(y)) {
        long iy = FIX2LONG(y);
#if SIZEOF_LONG * CHAR_BIT > DBL_MANT_DIG
        if ((iy < 0 ? -iy : iy) >= (1L << DBL_MANT_DIG)) {
            return rb_big_fdiv_double(rb_int2big(FIX2LONG(x)), rb_int2big(iy));
        }
#endif
        return double_div_double(FIX2LONG(x), iy);
    }

N.B. A simple round-trip check would be simpler and handle all the cases without needing to know anything about mantissa length:
if (iy != (long)(double)iy)

[rgburger (Bob Burger)]

Here's an example of Rational#to_f giving an incorrect result when the numerator is a bignum and the denominator is a fixnum that can't be exactly represented as a double:
Rational(10**24, 10**17 + 9).to_f should return 10000000.0, but it returns 9999999.999999998.

10**34 = 1000000000000000000000000, but (10**24).to_f.to_i = 999999999999999983222784.

10**17 + 9 = 100000000000000009, but (10**17 + 9).to_f.to_i = 100000000000000016.

Since rb_big_fdiv_double converts both numerator and denominator to doubles, it computes 999999999999999983222784 / 100000000000000016, which gives an incorrect answer.

[rgburger (Bob Burger)]

I found two more cases where rb_big_fdiv_double produces an incorrectly rounded result. I added them to pull request #18599.

assert_equal(3.333333333333333e39, (10**40).fdiv(3), "Bug #22246")
assert_equal(3.333333333333333e39, (10**40).fdiv(3.0), "Bug #22246")