Bug #8514

Rounding off error

Added by Jonam Rakesardnahc 10 months ago. Updated 10 months ago.

[ruby-core:55439]
Status:Rejected
Priority:Normal
Assignee:-
Category:core
Target version:-
ruby -v:ruby 2.0.0p195 (2013-05-14) [i386-mingw32] Backport:1.9.3: UNKNOWN, 2.0.0: UNKNOWN

Description

The expression (10/3.0) returns "3.3333333333333335". It should actually be "3.3333333333333333".

The expression (11/3.0) returns "3.6666666666666665". It should actually be "3.6666666666666667".

(13/3) is evaluated correctly though.

History

#1 Updated by Eric Hodel 10 months ago

  • Status changed from Open to Rejected

No, Ruby correctly rounds the imprecise value of the IEEE floating point number:

$ ruby -e 'puts "%0.30f" % (10/3.0)'
3.333333333333333481363069950021
$ ruby -e 'puts "%0.30f" % (11/3.0)'
3.666666666666666518636930049979

Please learn floating point numbers:

What Every Computer Scientist Should Know About Floating-Point Arithmetic
http://download.oracle.com/docs/cd/E19957-01/806-3568/ncg_goldberg.html
http://wiki.github.com/rdp/ruby_tutorials_core/ruby-talk-faq#floats_imprecise
http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems

#2 Updated by Jonam Rakesardnahc 10 months ago

Sorry.. moment of stupidity.

Also available in: Atom PDF