Bug #6137


openssl: hardcoded MD5 use leads to SSL server failure in FIPS mode

Added by jared.jennings.ctr (Jared Jennings) almost 10 years ago. Updated over 7 years ago.

Target version:
ruby -v:
ruby 1.8.7 (2011-06-30 patchlevel 352) [i386-linux]


I've got a host configured to be compliant with (()) (FIPS 140-2). On this host, the OpenSSL library refuses to do an MD5 checksum, because the MD5 algorithm is not FIPS Approved.

When I try to run Puppet's master subcommand, it sets up a secure HTTP server using WEBrick, which in turn uses the openssl module. But in the OpenSSL::SSL::SSLServer class, at source:ext/openssl/lib/openssl/ssl.rb@33695#L149, the MD5 digest is used to make a session ID from a context. On my host this fails as follows:

/usr/lib/ruby/1.8/openssl/digest.rb:55:in `initialize': Digest initialization failed.: unknown cipher (OpenSSL::Digest::DigestError)
    from /usr/lib/ruby/1.8/openssl/digest.rb:55:in `initialize'
    from /usr/lib/ruby/1.8/openssl/digest.rb:30:in `digest'
    from /usr/lib/ruby/1.8/openssl/digest.rb:30:in `digest'
    from /usr/lib/ruby/1.8/openssl/digest.rb:46:in `hexdigest'
    from /usr/lib/ruby/1.8/openssl/digest.rb:46:in `hexdigest'
    from /usr/lib/ruby/1.8/openssl/ssl-internal.rb:143:in `initialize'
    from /usr/lib/ruby/1.8/webrick/ssl.rb:94:in `new'
    from /usr/lib/ruby/1.8/webrick/ssl.rb:94:in `listen'
    from /usr/lib/ruby/1.8/webrick/ssl.rb:93:in `collect!'
    from /usr/lib/ruby/1.8/webrick/ssl.rb:93:in `listen'
    from /usr/lib/ruby/1.8/webrick/server.rb:63:in `initialize'
    from /usr/lib/ruby/1.8/webrick/httpserver.rb:24:in `initialize'
    from /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:33:in `new'
    from /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:33:in `listen'

I'm not sure exactly how, but ext/openssl/lib/openssl/ssl.rb from the source tree appears to be installed as /usr/lib/ruby/1.8/openssl/ssl-internal.rb on the system.

I replaced the instantiation of OpenSSL::Digest::MD5 with OpenSSL::Digest::SHA256 on my own system. The puppet master command worked, and no other bad things happened. Accordingly I suggest this change for Ruby in general. - Reasons to make the change:

  • Anyone trying to use OpenSSL::SSL::SSLServer who is in the U.S. government, a company contracting with the U.S. government, or possibly a bank, will appreciate if it works. (That's who cares about FIPS 140-2.)
  • I haven't seen any migration issues.
  • According to my reading of the code, any cryptographic hash will do.

Possible reasons not to make the change:

  • SHA256 takes more time than MD5. I haven't checked how often the hash is called. Embedded servers that use OpenSSL::SSL::SSLServer may slow down.
  • SHA256 hash values are longer than those of MD5. I don't think the hash values are stored in any variables with fixed size, but I haven't exhaustively confirmed it.



Also available in: Atom PDF