Project

General

Profile

Actions

Bug #8514

closed

Rounding off error

Added by manojtc (Jonam Rakesardnahc) almost 11 years ago. Updated almost 11 years ago.

Status:
Rejected
Assignee:
-
Target version:
-
ruby -v:
ruby 2.0.0p195 (2013-05-14) [i386-mingw32]
[ruby-core:55439]

Description

The expression (10/3.0) returns "3.3333333333333335". It should actually be "3.3333333333333333".

The expression (11/3.0) returns "3.6666666666666665". It should actually be "3.6666666666666667".

(13/3) is evaluated correctly though.

Updated by drbrain (Eric Hodel) almost 11 years ago

  • Status changed from Open to Rejected

No, Ruby correctly rounds the imprecise value of the IEEE floating point number:

$ ruby -e 'puts "%0.30f" % (10/3.0)'
3.333333333333333481363069950021
$ ruby -e 'puts "%0.30f" % (11/3.0)'
3.666666666666666518636930049979

Please learn floating point numbers:

What Every Computer Scientist Should Know About Floating-Point Arithmetic
http://download.oracle.com/docs/cd/E19957-01/806-3568/ncg_goldberg.html
http://wiki.github.com/rdp/ruby_tutorials_core/ruby-talk-faq#floats_imprecise
http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems

Updated by manojtc (Jonam Rakesardnahc) almost 11 years ago

Sorry.. moment of stupidity.

Actions

Also available in: Atom PDF

Like0
Like0Like0