Bug #19136: OpenSSL::PKey::EC.check_key is useless when linked against OpenSSL 3

• Status: Third Party's Issue
• Assignee: -
• Target version: -
• ruby -v: 3.1.2, 2.7.2
• Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN

[bannable (Joe Truba)]

When calling #check_key against an OpenSSL::PKey::EC instance representing an invalid point for the group, the method always returns true.

I believe this is because OpenSSL 3 deprecated EC_KEY_check_key, and the underlying call is swapped out for EVP_PKEY_public_check in ruby/openssl 3+.

However, the EVP_PKEY_public_check does not serve the same purpose as EC_KEY_check_key.

EVP_PKEY_public_check validates only the resulting public component, and does not validate the private component.

Reproducer

# check.rb
ver = ARGV[0]
gem 'openssl', ver
require 'openssl'

# ECDSA secp384r1 encoded key where the point is not on the curve
pem = <<~INVALID_KEY
-----BEGIN EC PRIVATE KEY-----
MIGkAgEBBDDA1Tm0m7YhkfeVpFuarAJYVlHp2tQj+1fOBiLa10t9E8TiQO/hVfxB
vGaVEQwOheWgBwYFK4EEACKhZANiAASyGqmryZGqdpsq5gEDIfNvgC3AwSJxiBCL
XKHBTFRp+tCezLDOK/6V8KK/vVGBJlGFW6/I7ahyXprxS7xs7hPA9iz5YiuqXlu+
lbrIpZOz7b73hyQQCkvbBO/Avg+hPAk=
-----END EC PRIVATE KEY-----
INVALID_KEY

begin
  result = OpenSSL::PKey::EC.new(pem).check_key
rescue => e
  result = e.message
end

puts format('%25s: %s','RUBY_VERSION', RUBY_VERSION)
puts format('%25s: %s','OPENSSL_LIBRARY_VERSION', OpenSSL::OPENSSL_LIBRARY_VERSION)
puts format('%25s: %s','OPENSSL_VERSION', OpenSSL::VERSION)
puts format('%25s: %s','result', result)

OpenSSL 1.1.1

$ rvm 2.7 do ruby check.rb 2.1.2
             RUBY_VERSION: 2.7.2
  OPENSSL_LIBRARY_VERSION: OpenSSL 1.1.1f  31 Mar 2020
          OPENSSL_VERSION: 2.1.2
                   result: EC_KEY_check_key: invalid private key
$ rvm 2.7 do ruby check.rb 3.0.1
             RUBY_VERSION: 2.7.2
  OPENSSL_LIBRARY_VERSION: OpenSSL 1.1.1f  31 Mar 2020
          OPENSSL_VERSION: 3.0.1
                   result: EVP_PKEY_public_check: invalid private key

$ rvm 3.1.2 do ruby check.rb
             RUBY_VERSION: 3.1.2
  OPENSSL_LIBRARY_VERSION: OpenSSL 1.1.1f  31 Mar 2020
          OPENSSL_VERSION: 3.0.1
                   result: EVP_PKEY_public_check: invalid private key

OpenSSL 3.0.2

$ ruby check.rb
             RUBY_VERSION: 3.1.2
  OPENSSL_LIBRARY_VERSION: OpenSSL 3.0.2 15 Mar 2022
          OPENSSL_VERSION: 3.0.1
                   result: true
$ ruby check.rb 3.0.0
             RUBY_VERSION: 3.1.2
  OPENSSL_LIBRARY_VERSION: OpenSSL 3.0.2 15 Mar 2022
          OPENSSL_VERSION: 3.0.0
                   result: true

[bannable (Joe Truba)]

I just realized that ruby/openssl takes issue reports on the GitHub. Sorry about that!

[hsbt (Hiroshi SHIBATA)]

• Status changed from Open to Third Party's Issue

This issue has been filed at https://github.com/ruby/openssl/issues/563