Bug #4944

crash in FIPS mode after unchecked EVP_DigestInit_ex failure

Added by Jared Jennings almost 4 years ago. Updated over 3 years ago.

[ruby-core:37670]
Status:Closed
Priority:High
Assignee:Martin Bosslet
ruby -v:ruby 1.9.3dev (2011-06-28 trunk 32273) [i686-linux] Backport:

Description

=begin
I've got a host configured to be compliant with (()) (FIPS 140-2). On this host, the OpenSSL library refuses to do an MD5 checksum, because the MD5 algorithm is not FIPS Approved. Any attempt to do an MD5 checksum using Ruby's openssl module (OpenSSL::Digest::MD5) presently results in the interpreter quitting with either a SIGSEGV or SIGABRT. This exists both in Ruby 1.8.7 as packaged in Red Hat Enterprise Linux 6.1, and in the nightly snapshot whose (({ruby -v})) you see below.

Here is a script which causes such a crash under FIPS mode:

require 'openssl'
md5 = OpenSSL::Digest::MD5.new
md5 << 'hi'
puts md5.hexdigest

The problem progresses like this: At source:/ext/openssl/ossl_digest.c#L36, GetDigestPtr fetches the MD5 algorithm using EVP_get_digestbyname or EVP_get_digestbyobj; this goes fine. At line 71, line 125 or line 162, we attempt to initialize the digest with EVP_DigestInit_ex. This returns 0 instead of 1, to indicate failure. The return value is presently ignored. (Even the example usage in my man page for EVP_DigestInit_ex doesn't check the return value!) Later on, either a SIGSEGV happens when a null function pointer is called, or some part of OpenSSL says on stderr,

digest.c(149): OpenSSL internal error, assertion failed: Digest init previous FIPS forbidden algorithm error ignored

Then it calls abort(), resulting in a SIGABRT. I haven't teased out exactly what leads to each outcome: both seem bad to me.

If the EVP_DigestInit_ex failure is tested for, the openssl module can throw an exception instead of causing an interpreter crash. The attached patch applies against the snapshot and does this.

Earlier discussion of this issue in the Puppet redmine is at ((URL:http://projects.puppetlabs.com/issues/8120)) (see note 2 particularly); a patch against 1.8.7, which is the same except for whitespace, is in the ruby-talk message ((URL:http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/384989)).
=end

snapshot-digestfail.patch Magnifier - Check for EVP_DigestInit_ex failure and raise exception (1.02 KB) Jared Jennings, 06/30/2011 12:19 AM


Related issues

Related to Ruby trunk - Bug #5076: Mac OS X Lion Support Closed 03/14/2011

Associated revisions

Revision 32606
Added by emboss over 3 years ago

  • ext/openssl/ossl_digest.c: Check return value of EVP_DigestInit_ex.
  • ext/openssl/ossl_hmac.c: Check return value of HMAC_Init_ex. Thanks, Jared Jennings, for the patch. [ Ruby 1.9 - Bug #4944 ]

Revision 32606
Added by emboss over 3 years ago

  • ext/openssl/ossl_digest.c: Check return value of EVP_DigestInit_ex.
  • ext/openssl/ossl_hmac.c: Check return value of HMAC_Init_ex. Thanks, Jared Jennings, for the patch. [ Ruby 1.9 - Bug #4944 ]

History

#1 Updated by Martin Bosslet over 3 years ago

  • Assignee set to Martin Bosslet
  • Target version set to 1.9.3
  • Priority changed from Normal to High

Thanks Jared,
I'll have a look. The same issue may arise with Ciphers as well iirc. I'm going to check whether we check return values there, otherwise this would probably have to be updated as well.

#2 Updated by Martin Bosslet over 3 years ago

  • Status changed from Open to Assigned

#3 Updated by Anonymous over 3 years ago

  • % Done changed from 0 to 100
  • Status changed from Assigned to Closed

This issue was solved with changeset r32606.
Jared, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.


  • ext/openssl/ossl_digest.c: Check return value of EVP_DigestInit_ex.
  • ext/openssl/ossl_hmac.c: Check return value of HMAC_Init_ex. Thanks, Jared Jennings, for the patch. [ Ruby 1.9 - Bug #4944 ]

#4 Updated by Martin Bosslet over 3 years ago

  • Status changed from Closed to Feedback

I currently don't have a FIPS build of OpenSSL to check this with.

Jared, could you please verify that it throws an exception now instead of segfaulting?

I applied the patch in r32606 on trunk and backported it to 1.9.3 in r32607.

Thanks,
Martin

#5 Updated by Eric Hodel over 3 years ago

r32607 breaks compilation on OS X 10.7:

ossl_hmac.c: In function ‘ossl_hmac_initialize’:
ossl_hmac.c:73: warning: ‘HMAC_Init’ is deprecated (declared at /usr/include/openssl/hmac.h:96)
ossl_hmac.c:73: error: void value not ignored as it ought to be

The relevant lines from openssl/hmac.h are:

void HMAC_Init(HMAC_CTX ctx, const void *key, int len,
const EVP_MD *md) DEPRECATED_IN_MAC_OS_X_VERSION_10_7_AND_LATER; /
deprecated */
void HMAC_Init_ex(HMAC_CTX *ctx, const void *key, int len,
const EVP_MD *md, ENGINE *impl) DEPRECATED_IN_MAC_OS_X_VERSION_10_7_AND_LATER;

#6 Updated by Eric Hodel over 3 years ago

Full log:

/Users/drbrain/Work/svn/ruby/branches/ruby_1_9_3/ext/openssl
gcc -I. -I../../.ext/include/x86_64-darwin11.0.0 -I../.././include -I../.././ext/openssl -DRUBY_EXTCONF_H=\"extconf.h\" -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -fno-common -O3 -ggdb -Wextra -Wno-unused-parameter -Wno-parentheses -Wno-long-long -Wno-missing-field-initializers -Wpointer-arith -Wwrite-strings -Wdeclaration-after-statement -Wshorten-64-to-32 -Wimplicit-function-declaration -pipe -o ossl_hmac.o -c ossl_hmac.c
ossl_hmac.c: In function ‘ossl_hmac_free’:
ossl_hmac.c:44: warning: ‘HMAC_CTX_cleanup’ is deprecated (declared at /usr/include/openssl/hmac.h:91)
ossl_hmac.c: In function ‘ossl_hmac_alloc’:
ossl_hmac.c:55: warning: ‘HMAC_CTX_init’ is deprecated (declared at /usr/include/openssl/hmac.h:90)
ossl_hmac.c: In function ‘ossl_hmac_initialize’:
ossl_hmac.c:73: warning: ‘HMAC_Init’ is deprecated (declared at /usr/include/openssl/hmac.h:96)
ossl_hmac.c:73: error: void value not ignored as it ought to be
ossl_hmac.c: In function ‘ossl_hmac_update’:
ossl_hmac.c:108: warning: ‘HMAC_Update’ is deprecated (declared at /usr/include/openssl/hmac.h:99)
ossl_hmac.c: In function ‘hmac_final’:
ossl_hmac.c:119: warning: ‘CRYPTO_malloc’ is deprecated (declared at /usr/include/openssl/crypto.h:478)
ossl_hmac.c:119: warning: ‘EVP_MD_size’ is deprecated (declared at /usr/include/openssl/evp.h:483)
ossl_hmac.c:120: warning: ‘HMAC_CTX_cleanup’ is deprecated (declared at /usr/include/openssl/hmac.h:91)
ossl_hmac.c:121: warning: ‘EVP_MD_size’ is deprecated (declared at /usr/include/openssl/evp.h:483)
ossl_hmac.c:124: warning: ‘HMAC_Final’ is deprecated (declared at /usr/include/openssl/hmac.h:100)
ossl_hmac.c:125: warning: ‘HMAC_CTX_cleanup’ is deprecated (declared at /usr/include/openssl/hmac.h:91)
ossl_hmac.c: In function ‘ossl_hmac_hexdigest’:
ossl_hmac.c:165: warning: ‘CRYPTO_free’ is deprecated (declared at /usr/include/openssl/crypto.h:480)
ossl_hmac.c:168: warning: ‘CRYPTO_free’ is deprecated (declared at /usr/include/openssl/crypto.h:480)
ossl_hmac.c: In function ‘ossl_hmac_reset’:
ossl_hmac.c:185: warning: ‘HMAC_Init’ is deprecated (declared at /usr/include/openssl/hmac.h:96)
ossl_hmac.c:185: error: void value not ignored as it ought to be
ossl_hmac.c: In function ‘ossl_hmac_s_digest’:
ossl_hmac.c:205: warning: ‘HMAC’ is deprecated (declared at /usr/include/openssl/hmac.h:103)
ossl_hmac.c: In function ‘ossl_hmac_s_hexdigest’:
ossl_hmac.c:227: warning: ‘HMAC’ is deprecated (declared at /usr/include/openssl/hmac.h:103)

#7 Updated by Martin Bosslet over 3 years ago

The issue with OS X 10.7 is solved, I had to revert the checks for an int return value of HMAC_Init_ex as OpenSSL versions prior to 1.0.0 did not have that feature yet. As concerns the deprecation of OpenSSL in OS X >= 10.7, that's a different issue. Seems to cause problems elsewhere, too: http://nodejs.debuggable.com/2011-03-05.txt (I like "Go away code we don't like. You can't use it." :)

Still, Jared, could you please confirm that the patch for EVP_Digest_init_ex works for you now?

#8 Updated by Jared Jennings over 3 years ago

Output from r32273, before the fix, as a sanity check:

rhel6w-32 ~ $ export PATH=/bin:/sbin:/usr/bin:/usr/sbin
rhel6w-32 ~ $ export PATH=$HOME/ruby-snapshot-prefix/bin:$PATH
rhel6w-32 ~ $ export LD_LIBRARY_PATH=$HOME/ruby-snapshot-prefix/lib
rhel6w-32 ~ $ which ruby
~/ruby-snapshot-prefix/bin/ruby
rhel6w-32 ~ $ cat fips-md5.rb
require 'openssl'
md5 = OpenSSL::Digest::MD5.new
md5 << 'hi'
puts md5.hexdigest
rhel6w-32 ~ $ ruby fips-md5.rb
fips-md5.rb:3: [BUG] Segmentation fault
ruby 1.9.3dev (2011-06-28 trunk 32273) [i686-linux]
[...]

Output from r32821, after the fix:

rhel6w-32 ~ $ export PATH=$HOME/ruby-snapshot-32821-prefix/bin:$PATH
rhel6w-32 ~ $ export LD_LIBRARY_PATH=$HOME/ruby-snapshot-32821-prefix/lib
rhel6w-32 ~ $ which ruby
~/ruby-snapshot-32821-prefix/bin/ruby
rhel6w-32 ~ $ ruby fips-md5.rb
/home/jenninjl/ruby-snapshot-32821-prefix/lib/ruby/1.9.1/openssl/digest.rb:36:in initialize': Digest initialization failed.: unknown cipher (OpenSSL::Digest::DigestError)
from /home/jenninjl/ruby-snapshot-32821-prefix/lib/ruby/1.9.1/openssl/digest.rb:36:in
block (3 levels) in class:Digest'
from fips-md5.rb:2:in new'
from fips-md5.rb:2:in
'

Looks good to me.

#9 Updated by Martin Bosslet over 3 years ago

  • Status changed from Feedback to Closed

Thanks, Jared! I'll close it then.

Also available in: Atom PDF