Bug #8514: Rounding off error

• Status: Rejected
• Assignee: -
• Target version: -
• ruby -v: ruby 2.0.0p195 (2013-05-14) [i386-mingw32]
• Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN

[manojtc (Jonam Rakesardnahc)]

The expression (10/3.0) returns "3.3333333333333335". It should actually be "3.3333333333333333".

The expression (11/3.0) returns "3.6666666666666665". It should actually be "3.6666666666666667".

(13/3) is evaluated correctly though.

[drbrain (Eric Hodel)]

• Status changed from Open to Rejected

No, Ruby correctly rounds the imprecise value of the IEEE floating point number:

$ ruby -e 'puts "%0.30f" % (10/3.0)'
3.333333333333333481363069950021
$ ruby -e 'puts "%0.30f" % (11/3.0)'
3.666666666666666518636930049979

Please learn floating point numbers:

What Every Computer Scientist Should Know About Floating-Point Arithmetic
http://download.oracle.com/docs/cd/E19957-01/806-3568/ncg_goldberg.html
http://wiki.github.com/rdp/ruby_tutorials_core/ruby-talk-faq#floats_imprecise
http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems

[manojtc (Jonam Rakesardnahc)]

Sorry.. moment of stupidity.