Bug #9659

crash in FIPS mode after unchecked algo->init_func failure

Added by jared.jennings.ctr (Jared Jennings) over 6 years ago. Updated almost 3 years ago.

Target version:
ruby -v:
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]


This is just like #4944, but in the digest extension instead of the openssl extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)

The digest extension, in the rb_digest_base_alloc, rb_digest_base_reset, and rb_digest_base_finish functions, is ignoring the return code of algo->init_func. If OpenSSL is present at build time, algo->init_func works out to be the MD5_Init function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

  • source:ext/digest/digest.c@43668#L551
  • source:ext/digest/digest.c@43668#L589
  • source:ext/digest/digest.c@43668#L627


002-builtin-indicate-digest-failure.patch (10.4 KB) 002-builtin-indicate-digest-failure.patch jared.jennings.ctr (Jared Jennings), 03/27/2014 11:45 PM
001-detect-digest-failure.patch (2.12 KB) 001-detect-digest-failure.patch jared.jennings.ctr (Jared Jennings), 03/27/2014 11:45 PM
003-digest-openssl-md5-use-evp-api.patch (1.8 KB) 003-digest-openssl-md5-use-evp-api.patch jared.jennings.ctr (Jared Jennings), 03/27/2014 11:45 PM

Also available in: Atom PDF