Bug #12428: OpenSSL::PKey::RSA.public_encrypt crashes ruby when initiated without key size.

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]
• Backport: 2.1: WONTFIX, 2.2: DONE, 2.3: DONE

[cjeon (CHEOLHO JEON)]

Hi, I think I found a bug. OpenSSL::PKey::RSA initiated without keysize crashes when public_encrypt is called.
Below script will crash ruby (it is attached as opensslcrash.rb)

require('openssl')
p = OpenSSL::PKey::RSA.new
p.public_encrypt('hi')

I'm attaching error message(ruby opensslcrash.rb &> error_message.txt) and crash reports under /Users/CJeon/Library/Logs/DiagnosticReports.

Let me know if you guys need further information. Thanks!

[cjeon (CHEOLHO JEON)]

• File opensslcrash.rb opensslcrash.rb added

• Status changed from Open to Closed

Applied in changeset r55175.

---

openssl: avoid NULL dereference in {DH,DSA,RSA}_size()

• ext/openssl/ossl_pkey_dh.c (ossl_dh_compute_key): Check that the DH
  has 'p' (the prime) before calling DH_size(). We can create a DH with
  no parameter but DH_size() does not check and dereferences NULL.
  [ruby-core:75720] [Bug #12428]

• ext/openssl/ossl_pkey_dsa.c (ossl_dsa_sign): Ditto. DSA_size() does
  not check dsa->q.

• ext/openssl/ossl_pkey_rsa.c (ossl_rsa_public_encrypt,
  ossl_rsa_public_decrypt, ossl_rsa_private_encrypt,
  ossl_rsa_private_decrypt): Ditto. RSA_size() does not check rsa->n.

[usa (Usaku NAKAMURA)]

• Backport changed from 2.1: UNKNOWN, 2.2: UNKNOWN, 2.3: UNKNOWN to 2.1: WONTFIX, 2.2: REQUIRED, 2.3: REQUIRED

[nagachika (Tomoyuki Chikanaga)]

• Backport changed from 2.1: WONTFIX, 2.2: REQUIRED, 2.3: REQUIRED to 2.1: WONTFIX, 2.2: REQUIRED, 2.3: DONE

ruby_2_3 r55904 merged revision(s) 55175.

[usa (Usaku NAKAMURA)]

• Backport changed from 2.1: WONTFIX, 2.2: REQUIRED, 2.3: DONE to 2.1: WONTFIX, 2.2: DONE, 2.3: DONE

ruby_2_2 r55935 merged revision(s) 55175.

[nobu (Nobuyoshi Nakada)]

• Has duplicate Bug #12865: openssl: segmentation fault when nil is passed as key added