Project

General

Profile

Bug #8514

Rounding off error

Added by Jonam Rakesardnahc almost 3 years ago. Updated almost 3 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
ruby -v:
ruby 2.0.0p195 (2013-05-14) [i386-mingw32]
[ruby-core:55439]

Description

The expression (10/3.0) returns "3.3333333333333335". It should actually be "3.3333333333333333".

The expression (11/3.0) returns "3.6666666666666665". It should actually be "3.6666666666666667".

(13/3) is evaluated correctly though.

History

#1 [ruby-core:55446] Updated by Eric Hodel almost 3 years ago

  • Status changed from Open to Rejected

No, Ruby correctly rounds the imprecise value of the IEEE floating point number:

$ ruby -e 'puts "%0.30f" % (10/3.0)'
3.333333333333333481363069950021
$ ruby -e 'puts "%0.30f" % (11/3.0)'
3.666666666666666518636930049979

Please learn floating point numbers:

What Every Computer Scientist Should Know About Floating-Point Arithmetic
http://download.oracle.com/docs/cd/E19957-01/806-3568/ncg_goldberg.html
http://wiki.github.com/rdp/ruby_tutorials_core/ruby-talk-faq#floats_imprecise
http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems

#2 [ruby-core:55459] Updated by Jonam Rakesardnahc almost 3 years ago

Sorry.. moment of stupidity.

Also available in: Atom PDF