Bug #8514
closedRounding off error
Description
The expression (10/3.0) returns "3.3333333333333335". It should actually be "3.3333333333333333".
The expression (11/3.0) returns "3.6666666666666665". It should actually be "3.6666666666666667".
(13/3) is evaluated correctly though.
Updated by drbrain (Eric Hodel) almost 11 years ago
- Status changed from Open to Rejected
No, Ruby correctly rounds the imprecise value of the IEEE floating point number:
$ ruby -e 'puts "%0.30f" % (10/3.0)'
3.333333333333333481363069950021
$ ruby -e 'puts "%0.30f" % (11/3.0)'
3.666666666666666518636930049979
Please learn floating point numbers:
What Every Computer Scientist Should Know About Floating-Point Arithmetic
http://download.oracle.com/docs/cd/E19957-01/806-3568/ncg_goldberg.html
http://wiki.github.com/rdp/ruby_tutorials_core/ruby-talk-faq#floats_imprecise
http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems
Updated by manojtc (Jonam Rakesardnahc) almost 11 years ago
Sorry.. moment of stupidity.