Bug #12504: test failure of openssl with ubuntu 16.04

• Status: Closed
• Assignee: rhenium (Kazuki Yamaguchi)
• Target version: -
• Backport: 2.1: DONTNEED, 2.2: DONTNEED, 2.3: DONTNEED

[hsbt (Hiroshi SHIBATA)]

CI is broken at OpenSSL Ubuntu 16.04(xenial). It shows following log.

[ 1071/16956] OpenSSL::TestEOF1#test_eof_0fips.c(139): OpenSSL internal error, assertion failed: FATAL FIPS SELFTEST FAILURE

see full of log. http://rubyci.s3.amazonaws.com/ubuntu/ruby-trunk/log/20160618T063004Z.fail.html.gz

version of openssl on above instance is here:

Package: openssl
Version: 1.0.2g-1ubuntu4.1
(snip)

[rhenium (Kazuki Yamaguchi)]

• Status changed from Open to Assigned
• Assignee set to rhenium (Kazuki Yamaguchi)

I reproduced it, and can avoid it by applying this.

diff --git a/test/openssl/test_pkey_ec.rb b/test/openssl/test_pkey_ec.rb
index e05b70a..e15d875 100644
--- a/test/openssl/test_pkey_ec.rb
+++ b/test/openssl/test_pkey_ec.rb
@@ -14,12 +14,12 @@ def setup
     OpenSSL::PKey::EC.builtin_curves.each do |curve, comment|
       group = OpenSSL::PKey::EC::Group.new(curve)
 
-      key = OpenSSL::PKey::EC.new(group)
-      key.generate_key!
-
       # Oakley curves and X25519 are not suitable for signing
       next if ["Oakley", "X25519"].any? { |n| curve.start_with?(n) }
 
+      key = OpenSSL::PKey::EC.new(group)
+      key.generate_key!
+
       @groups << group
       @keys << key
     end

The following code sometimes causes the crash (also with Ruby 2.3.0), I didn't dig any deeper but I suspect there is a bug in Ubuntu's patched OpenSSL.

require "openssl"

begin
  OpenSSL::PKey::EC.new("Oakley-EC2N-4").generate_key
rescue
  p $!
end
OpenSSL::PKey::RSA.new(512)

#<OpenSSL::PKey::ECError: EC_KEY_generate_key: pairwise test failed>
fips.c(139): OpenSSL internal error, assertion failed: FATAL FIPS SELFTEST FAILURE
Aborted

• Status changed from Assigned to Closed

Applied in changeset r55444.

---

openssl: avoid test crash on Ubuntu 16.04

• test/openssl/test_pkey_ec.rb (setup): Don't call EC#generate_key! for
  Oakley-* curves. This causes an odd error on Ubuntu 16.04 with openssl
  1.0.2g-1ubuntu4.1.

  begin
  OpenSSL::PKey::EC.new("Oakley-EC2N-4").generate_key
  rescue
  p $!
  end
  OpenSSL::PKey::RSA.new(512)

  This sometimes causes:

  #<OpenSSL::PKey::ECError: EC_KEY_generate_key: pairwise test failed>
  fips.c(139): OpenSSL internal error, assertion failed: FATAL FIPS SELFTEST FAILURE

  [ruby-dev:49670] [Bug #12504]

[hsbt (Hiroshi SHIBATA)]

Thank you for your patch.

I confirmed to fix it with r55444.

http://rubyci.s3.amazonaws.com/ubuntu/ruby-trunk/log/20160618T093003Z.fail.html.gz

[rhenium (Kazuki Yamaguchi)]

I filed an issue at https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1593953

[usa (Usaku NAKAMURA)]

• Backport changed from 2.1: UNKNOWN, 2.2: UNKNOWN, 2.3: UNKNOWN to 2.1: DONTNEED, 2.2: DONTNEED, 2.3: DONTNEED