Project

General

Profile

Bug #10633

OpenSSL RangeError on update

Added by khoffmann (Kenneth Hoffmann) over 5 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
openssl
Target version:
ruby -v:
2.1.3p242 (2014-09-19 revision 47630) [x86_64-linux]
[ruby-core:67043]

Description

OpenSSL Cipher throws a RangeError on the update function when given a very large string. I've verified that it happens with a file of 2 or 3GB in size. 1 GB works fine. This has been tested with 2.1.2, 2.1.3, and 2.2rc-1.

require 'openssl'
cipher = OpenSSL::Cipher::Cipher.new('aes-256-cbc')
cipher.decrypt
cipher.iv = "fooooooooooooooooooooooooooooooooo"
cipher.key = "barrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr"
f = File.open("<path to a 2 GB file generated with dd>", "r")
contents = f.read;
cipher.update(contents)

Running this will return:

RangeError: integer 2147483648 too big to convert to `int'

File generated with dd:

dd if=/dev/zero of=2g.img bs=1 count=0 seek=2G

Updated by tdg5 (Danny Guinther) over 5 years ago

I get the same result on

ruby 1.9.2p330 (2014-08-07 revision 47094) [x86_64-linux]
ruby 1.9.3p545 (2014-02-24 revision 45159) [x86_64-linux]
ruby 2.1.1p76 (2014-02-24 revision 45161) [x86_64-linux]

Maybe an Ubuntu OpenSSL issue? Either that or the issue has been around for a while.

Affected Ubuntu OpenSSL Versions:

1.0.1-4ubuntu5.20
1.0.1f-1ubuntu2.7

Updated by nobu (Nobuyoshi Nakada) over 5 years ago

  • Description updated (diff)

It's caused by OpenSSL's API design.

#3

Updated by nobu (Nobuyoshi Nakada) over 5 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

Applied in changeset r48923.


ossl_cipher.c: workaround of OpenSSL API

  • ext/openssl/ossl_cipher.c (ossl_cipher_update_long): update huge data gradually not to exceed INT_MAX. workaround of OpenSSL API limitation. [ruby-core:67043] [Bug #10633]

Updated by tdg5 (Danny Guinther) over 5 years ago

I don't have much experience with building Ruby by hand, so part of my issue may be there, or perhaps the example provided by Kenneth should never work, but should blow up in a different way than what he ran into, but I pulled down and built ruby-trunk and I still fail out of the example that Kenneth provided. It doesn't fail in the same way, which makes me think my build is good, but the error is pretty nondescript:

ossl_large.rb:8:in `update': OpenSSL::Cipher::CipherError

That's it. Line 8 is consistent with Kenneth's line 8:

cipher.update(contents)

Should the provided script work now or is this indicative of something else going on? I tried connecting to the script process w/ strace and there wasn't much informative there. I also played around with gdb, but I don't have enough experience with gdb to actually dig into which of the "ossl_raise(eCipherError, NULL)" calls the script was hitting.

Thoughts?

Updated by nagachika (Tomoyuki Chikanaga) over 5 years ago

  • Backport changed from 2.0.0: UNKNOWN, 2.1: UNKNOWN to 2.0.0: REQUIRED, 2.1: REQUIRED, 2.2: REQUIRED

Updated by naruse (Yui NARUSE) over 5 years ago

  • Backport changed from 2.0.0: REQUIRED, 2.1: REQUIRED, 2.2: REQUIRED to 2.0.0: REQUIRED, 2.1: REQUIRED, 2.2: DONE

ruby_2_2 r48951 merged revision(s) 48923.

Updated by usa (Usaku NAKAMURA) over 5 years ago

  • Backport changed from 2.0.0: REQUIRED, 2.1: REQUIRED, 2.2: DONE to 2.0.0: DONE, 2.1: REQUIRED, 2.2: DONE

Backported into ruby_2_0_0 at r49249.

Updated by nagachika (Tomoyuki Chikanaga) over 5 years ago

  • Backport changed from 2.0.0: DONE, 2.1: REQUIRED, 2.2: DONE to 2.0.0: DONE, 2.1: DONE, 2.2: DONE

Backported into ruby_2_1 at r49383.

Also available in: Atom PDF