Project

General

Profile

Actions

Bug #10429

closed

Ruby 2.x exponents produce wildly varying precision for integers vs decimals

Added by dresselm (Matt Dressel) over 9 years ago. Updated over 4 years ago.

Status:
Closed
Assignee:
-
Target version:
-
ruby -v:
2.0.0-p576, 2.1.3
[ruby-core:65911]

Description

2.0.0

> 1.1.to_d ** -1.9999999
 => #<BigDecimal:7fdcc1fcdfa0,'0.8264462888 68609939E0',18(45)> 
> 1.1.to_d ** -2
 => #<BigDecimal:7fdccd3e8d00,'0.8264462809 9173553719 0082644628 0991735537 1900826446 2809917355 3719008264 46E0',72(72)> 
> 1.1.to_d ** -2.0
 => #<BigDecimal:7fdccd3e8d00,'0.8264462809 9173553719 0082644628 0991735537 1900826446 2809917355 3719008264 46E0',72(72)> 
> 1.1.to_d ** -2.0000001
 => #<BigDecimal:7fdcc1f702d8,'0.8264462731 14861211E0',18(45)>

2.1.3

> 1.1.to_d ** -1.9999999
=> #<BigDecimal:7ff690a7d6c0,'0.8264462888 68609938E0',18(36)>
> 1.1.to_d ** -2
=> #<BigDecimal:7ff690ace0c0,'0.8264462809 9173553719 0082644628 0991735537 1900826446 2809917355 371E0',63(63)>
> 1.1.to_d ** -2.0
=> #<BigDecimal:7ff690ace0c0,'0.8264462809 9173553719 0082644628 0991735537 1900826446 2809917355 371E0',63(63)>
> 1.1.to_d ** -2.0000001
=> #<BigDecimal:7ff674c0b0a8,'0.8264462731 1486121E0',18(36)>

Additionally, when the integer value increases, the precision increases:

2.0.0

> 1.1.to_d ** 2
 => #<BigDecimal:7fb0969fb7f0,'0.121E1',18(72)> 
> 1.1.to_d ** 3
 => #<BigDecimal:7fb096928468,'0.1331E1',18(90)> 
> 1.1.to_d ** 4
 => #<BigDecimal:7fb096a05890,'0.14641E1',18(117)> 
> 1.1.to_d ** 5
 => #<BigDecimal:7fb096a0fbb0,'0.161051E1',18(135)> 
> 1.1.to_d ** 6
 => #<BigDecimal:7fb096a1e110,'0.1771561E1',18(153)> 

Updated by dresselm (Matt Dressel) over 9 years ago

What do I need to do to get someone to look at this?

Updated by shyouhei (Shyouhei Urabe) over 9 years ago

Matt Dressel wrote:

What do I need to do to get someone to look at this?

You would better show us your expectation. You are just saying something is strange. If you think it's wrong instead of just being strange, tell us how you think it should behave.

Updated by dresselm (Matt Dressel) over 9 years ago

Thank you. I will add some additional thoughts.

Actions #4

Updated by jeremyevans0 (Jeremy Evans) over 4 years ago

  • Status changed from Open to Closed
Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0