Bug #10633: OpenSSL RangeError on update

• Status: Closed
• Priority: Normal
• Assignee: -
• Target version: 2.2.0
• ruby -v: 2.1.3p242 (2014-09-19 revision 47630) [x86_64-linux]
• Backport: 2.0.0: DONE, 2.1: DONE, 2.2: DONE

[khoffmann (Kenneth Hoffmann)]

OpenSSL Cipher throws a RangeError on the update function when given a very large string. I've verified that it happens with a file of 2 or 3GB in size. 1 GB works fine. This has been tested with 2.1.2, 2.1.3, and 2.2rc-1.

require 'openssl'
cipher = OpenSSL::Cipher::Cipher.new('aes-256-cbc')
cipher.decrypt
cipher.iv = "fooooooooooooooooooooooooooooooooo"
cipher.key = "barrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr"
f = File.open("<path to a 2 GB file generated with dd>", "r")
contents = f.read;
cipher.update(contents)

Running this will return:

RangeError: integer 2147483648 too big to convert to `int'

File generated with dd:

dd if=/dev/zero of=2g.img bs=1 count=0 seek=2G

[tdg5 (Danny Guinther)]

I get the same result on

ruby 1.9.2p330 (2014-08-07 revision 47094) [x86_64-linux]
ruby 1.9.3p545 (2014-02-24 revision 45159) [x86_64-linux]
ruby 2.1.1p76 (2014-02-24 revision 45161) [x86_64-linux]

Maybe an Ubuntu OpenSSL issue? Either that or the issue has been around for a while.

Affected Ubuntu OpenSSL Versions:

1.0.1-4ubuntu5.20
1.0.1f-1ubuntu2.7

[nobu (Nobuyoshi Nakada)]

• Description updated (diff)

It's caused by OpenSSL's API design.

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed
• % Done changed from 0 to 100

Applied in changeset r48923.

---

ossl_cipher.c: workaround of OpenSSL API

• ext/openssl/ossl_cipher.c (ossl_cipher_update_long): update huge
  data gradually not to exceed INT_MAX. workaround of OpenSSL API
  limitation. [ruby-core:67043] [Bug #10633]

[tdg5 (Danny Guinther)]

I don't have much experience with building Ruby by hand, so part of my issue may be there, or perhaps the example provided by Kenneth should never work, but should blow up in a different way than what he ran into, but I pulled down and built ruby-trunk and I still fail out of the example that Kenneth provided. It doesn't fail in the same way, which makes me think my build is good, but the error is pretty nondescript:

ossl_large.rb:8:in `update': OpenSSL::Cipher::CipherError

That's it. Line 8 is consistent with Kenneth's line 8:

cipher.update(contents)

Should the provided script work now or is this indicative of something else going on? I tried connecting to the script process w/ strace and there wasn't much informative there. I also played around with gdb, but I don't have enough experience with gdb to actually dig into which of the "ossl_raise(eCipherError, NULL)" calls the script was hitting.

Thoughts?

[nagachika (Tomoyuki Chikanaga)]

• Backport changed from 2.0.0: UNKNOWN, 2.1: UNKNOWN to 2.0.0: REQUIRED, 2.1: REQUIRED, 2.2: REQUIRED

[naruse (Yui NARUSE)]

• Backport changed from 2.0.0: REQUIRED, 2.1: REQUIRED, 2.2: REQUIRED to 2.0.0: REQUIRED, 2.1: REQUIRED, 2.2: DONE

ruby_2_2 r48951 merged revision(s) 48923.

[usa (Usaku NAKAMURA)]

• Backport changed from 2.0.0: REQUIRED, 2.1: REQUIRED, 2.2: DONE to 2.0.0: DONE, 2.1: REQUIRED, 2.2: DONE

Backported into ruby_2_0_0 at r49249.

[nagachika (Tomoyuki Chikanaga)]

• Backport changed from 2.0.0: DONE, 2.1: REQUIRED, 2.2: DONE to 2.0.0: DONE, 2.1: DONE, 2.2: DONE

Backported into ruby_2_1 at r49383.