Project

General

Profile

Bug #12324

Support OpenSSL 1.1.0 (and drop support for 0.9.6/0.9.7)

Added by rhenium (Kazuki Yamaguchi) over 1 year ago. Updated 11 months ago.

Status:
Closed
Priority:
Normal
Target version:
-
[ruby-core:75225]

Description

The last beta for OpenSSL1.1.0 (1.1.0-pre5) was released a week ago. According to OpenSSL's website[1], OpenSSL 1.1.0 final will be released on 2016-05-12.
OpenSSL 1.1.0 introduces many many compatibility breaking changes[2][3] but they are almost done and I think we can start work on it.

Also, I think we can drop support for ancient (-0.9.7) versions of OpenSSL. This can save many compatibility macros (about 700 lines). Considering the last official release is over 9 years ago (2007-02-23) and even 0.9.8/1.0.0 series are no longer supported[1], there should be no reason to continue supporting them. In fact builds with OpenSSL 0.9.7 are already broken since r40461 (2013-04-25, Ruby 2.1.0) but it looks like nobody noticed it. (I personally think all unsupported version should be removed but 0.9.8 is unfortunately still widely used...)

[1] https://www.openssl.org/policies/releasestrat.html
[2] https://www.openssl.org/news/openssl-1.1.0-notes.html
[3] https://www.openssl.org/news/changelog.html


I found some incompatibilities in OpenSSL 1.1.0 which affect Ruby-world API:

  • SSL_CTX_set_tmp_ecdh_callback() is removed

    OpenSSL::SSL::SSLContext#tmp_ecdh_callback (exists in Ruby 2.3, [Feature #11356]) breaks. This is used to enable ECDH and specify the curve to be used.
    OpenSSL 1.0.2 and later provide SSL_CTX_set1_curves_list(). This sets the "supported (named) curves" and OpenSSL selects the most appropriate curve from the list. The automatic selection is enabled by SSL_CTX_set_ecdh_auto() on 1.0.2. It is always enabled on 1.1.0.

    Summarizing... ECDH is enabled:

    • OpenSSL 1.1.0: always.
    • OpenSSL 1.0.2, LibreSSL 2.3: if SSL_CTX_set_ecdh_auto() is called.
    • OpenSSL -1.0.2, LibreSSL 2.3: if SSL_CTX_set_tmp_ecdh{_callback,}() is called.

    The curve to be used is:
    - OpenSSL 1.0.2/1.1.0: automatically selected by OpenSSL, if enabled. The "supported curves" can be changed by SSL_CTX_set1_curves{_list,}().
    - LibreSSL 2.3: automatically selected if enabled. The list can't be changed.
    - OpenSSL -1.0.2, LibreSSL 2.3: set by SSL_CTX_set_tmp_ecdh{_callback,}().

    In my patch, I made this deprecated (rb_warn() on SSLContext#setup) it and added SSLContext#set_ecdh_curves:
    - OpenSSL 1.0.2/1.1.0: wraps SSL_CTX_set1_curves_list()
    - OpenSSL -1.0.1, LibreSSL 2.3: wraps SSL_CTX_set_tmp_ecdh() (so only 1 curve can be set)

    # assume the client supports P-224 and P-521
    ctx = OpenSSL::SSL::SSLContext.new
    ctx.set_ecdh_curves "P-384:P-224:P-521"
    svr = OpenSSL::SSL::SSLServer.new(sock, ctx)
    svr.accept # will use P-224
    
  • Accessors for OpenSSL::PKey::{DH,DSA,RSA}'s paramters/keys

    Since most structures of OpenSSL are made opaque, we can no longer set directly these values. OpenSSL 1.1.0 provides setter functions (e.g. DH_set0_key()) but they are not equivalent; they require setting all relevant values at the same time. For example, such code (4 usage in tests) is no longer possible:

    dh = OpenSSL::PKey::DH.new(File.read("dhparams.pem"))
    dh.priv_key = OpenSSL::BN.new("000..")
    

    In my patch I deprecated them and added XX#set_XXX methods (such as DH#set_key, which sets pub_key and priv_key).

    dh = OpenSSL::PKey::DH.new(File.read("dhparams.pem"))
    dh.set_key(OpenSSL::BN.new("000.."), # pub_key
               OpenSSL::BN.new("000..")) # priv_key
    

    This is not beautiful but it looks like there is no other way for this...

  • Security level

    Please see the manpage SSL_CTX_set_security_level(3):

    https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_security_level.html

    OpenSSL 1.1.0 introduces "security level", which disables insecure settings, such as insecure ciphers, depending on the level. The default level is 1 and this means:

    The security level corresponds to a minimum of 80 bits of security. Any parameters offering below 80 bits of security are excluded. As a result RSA, DSA and DH keys shorter than 1024 bits and ECC keys shorter than 160 bits are prohibited. All export ciphersuites are prohibited since they all offer less than 80 bits of security. SSL version 2 is prohibited. Any ciphersuite using MD5 for the MAC is also prohibited.
    (from 1.1.0-pre5 doc)

    New methods would be essential. Actually this breaks many tests which use aNULL cipher suites.

    I added SSLContext#security_level and SSLContext#security_level= in my patch.


Here is my try. Because of the amount of OpenSSL changes, almost all .c files had to be modified. Roughly splitted into 38 commits:

https://github.com/ruby/ruby/compare/trunk...rhenium:feature/openssl-110-v1
(the attached gzip'ed is the same)

This includes:

  • Add check for SSL_CTX_clear_options(), which doesn't exist in OpenSSL 0.9.8l and older.
  • Drop support for 0.9.7.
  • Remove OPENSSL_NO_HMAC support as it have never worked.
  • Deprecate (rb_warn()ing) SSLContext#tmp_ecdh_callback and add SSL::SSLContext#set_ecdh_curves method.
  • Deprecate parameters/keys setter for PKey::DH, PKey::RSA and PKey::DSA, and add PKey::XX#set_XXX methods.
  • Add SSL::SSLContext#security_level and #security_level=.
  • Test fixes for open-uri and rubygems (regenerate the test certificates; they are incorrectly created and OpenSSL 1.1.0 starts to complain about it).

This compiles and tests pass with the following versions (on my Linux/gcc environment):

  • 0.9.8zh
  • 1.0.0t
  • 1.0.1s
  • 1.0.2g
  • OpenSSL master
  • LibreSSL 2.3.3

Note that this doesn't compile with 1.1.0-pre5; OpenSSL GH-975 is required (already in master and the 1.1.0 final will include this).
https://github.com/openssl/openssl/pull/975

The patch is far from perfect - I'm not sure about the new methods and also probably there are bugs.

Any comments are welcome, I'll update the patch.

Thanks,

patches-v1.tar.gz (81.4 KB) patches-v1.tar.gz rhenium (Kazuki Yamaguchi), 04/27/2016 05:09 PM
patches-v2.tar.gz (79.9 KB) patches-v2.tar.gz rhenium (Kazuki Yamaguchi), 05/05/2016 09:56 AM
interdiff-v1-v2.patch (13.4 KB) interdiff-v1-v2.patch rhenium (Kazuki Yamaguchi), 05/05/2016 09:56 AM
patches-v3.tar.gz (80.1 KB) patches-v3.tar.gz rhenium (Kazuki Yamaguchi), 05/14/2016 11:20 AM
interdiff-v2-v3.patch (15.7 KB) interdiff-v2-v3.patch rhenium (Kazuki Yamaguchi), 05/14/2016 11:20 AM

Associated revisions

Revision 55282
Added by rhe over 1 year ago

openssl: check existence of RAND_pseudo_bytes()

  • ext/openssl/extconf.rb: Check if RAND_pseudo_bytes() is usable. It is
    marked as deprecated in OpenSSL 1.1.0.
    [Feature #12324]

  • ext/openssl/ossl_rand.c: Disable Random.pseudo_bytes if
    RAND_pseudo_bytes() is unavailable.

  • test/openssl/test_random.rb: Don't test Random.pseudo_bytes if not
    defined.

Revision 55282
Added by rhe over 1 year ago

openssl: check existence of RAND_pseudo_bytes()

  • ext/openssl/extconf.rb: Check if RAND_pseudo_bytes() is usable. It is
    marked as deprecated in OpenSSL 1.1.0.
    [Feature #12324]

  • ext/openssl/ossl_rand.c: Disable Random.pseudo_bytes if
    RAND_pseudo_bytes() is unavailable.

  • test/openssl/test_random.rb: Don't test Random.pseudo_bytes if not
    defined.

Revision 55283
Added by rhe over 1 year ago

openssl: support OpenSSL 1.1.0's new multi-threading API

  • ext/openssl/extconf.rb: Check absence of CRYPTO_lock() to see if the OpenSSL has the new threading API. In OpenSSL <= 1.0.2, an application had to set locking callbacks to use OpenSSL in a multi-threaded environment. OpenSSL 1.1.0 now finds pthreads or Windows threads so we don't need to do something special. [Feature #12324]

Also check existence of *_up_ref(). Some structures in OpenSSL have
a reference counter. We used to increment it with CRYPTO_add() which
is a part of the old API.

  • ext/openssl/openssl_missing.h: Implement *_up_ref() if missing.

  • ext/openssl/ossl.c: Don't set locking callbacks if unneeded.

  • ext/openssl/ossl_pkey.c, ext/openssl/ossl_ssl.c,
    ext/openssl/ossl_x509cert.c, ext/openssl/ossl_x509crl.c,
    ext/openssl/ossl_x509store.c: Use *_up_ref() instead of CRYPTO_add().

Revision 55283
Added by rhe over 1 year ago

openssl: support OpenSSL 1.1.0's new multi-threading API

  • ext/openssl/extconf.rb: Check absence of CRYPTO_lock() to see if the OpenSSL has the new threading API. In OpenSSL <= 1.0.2, an application had to set locking callbacks to use OpenSSL in a multi-threaded environment. OpenSSL 1.1.0 now finds pthreads or Windows threads so we don't need to do something special. [Feature #12324]

Also check existence of *_up_ref(). Some structures in OpenSSL have
a reference counter. We used to increment it with CRYPTO_add() which
is a part of the old API.

  • ext/openssl/openssl_missing.h: Implement *_up_ref() if missing.

  • ext/openssl/ossl.c: Don't set locking callbacks if unneeded.

  • ext/openssl/ossl_pkey.c, ext/openssl/ossl_ssl.c,
    ext/openssl/ossl_x509cert.c, ext/openssl/ossl_x509crl.c,
    ext/openssl/ossl_x509store.c: Use *_up_ref() instead of CRYPTO_add().

Revision 55285
Added by rhe over 1 year ago

openssl: adapt OpenSSL::PKey to OpenSSL 1.1.0 opaque structs

  • ext/openssl/openssl_missing.[ch]: Implement EVP_PKEY_get0_() and
    {RSA,DSA,EC_KEY,DH}get0
    () functions.
    OpenSSL 1.1.0 makes EVP_PKEY/RSA/DSA/DH opaque. We used to provide
    setter methods for each parameter of each PKey type, for example
    PKey::RSA#e=, but this is no longer possible because the new API
    RSA_set0_key() requires the 'n' at the same time. This commit adds
    deprecation warning to them and adds PKey::#set_ methods as direct
    wrapper for those new APIs. For example, 'rsa.e = 3' now needs to be
    rewritten as 'rsa.set_key(rsa.n, 3, rsa.d)'.
    [Feature #12324]

  • ext/openssl/ossl_pkey*.[ch]: Use the new accessor functions. Implement
    RSA#set_{key,factors,crt_params}, DSA#set_{key,pqg}, DH#set_{key,pqg}.
    Emit a warning with rb_warning() when old setter methods are used.

  • test/drb/ut_array_drbssl.rb, test/drb/ut_drb_drbssl.rb,
    test/rubygems/test_gem_remote_fetcher.rb: Don't set a priv_key for DH
    object that are used in tmp_dh_callback. Generating a new key pair
    every time should be fine - actually the private exponent is ignored
    in OpenSSL >= 1.0.2f/1.0.1r even if we explicitly set.
    https://www.openssl.org/news/secadv/20160128.txt

Revision 55285
Added by rhe over 1 year ago

openssl: adapt OpenSSL::PKey to OpenSSL 1.1.0 opaque structs

  • ext/openssl/openssl_missing.[ch]: Implement EVP_PKEY_get0_() and
    {RSA,DSA,EC_KEY,DH}get0
    () functions.
    OpenSSL 1.1.0 makes EVP_PKEY/RSA/DSA/DH opaque. We used to provide
    setter methods for each parameter of each PKey type, for example
    PKey::RSA#e=, but this is no longer possible because the new API
    RSA_set0_key() requires the 'n' at the same time. This commit adds
    deprecation warning to them and adds PKey::#set_ methods as direct
    wrapper for those new APIs. For example, 'rsa.e = 3' now needs to be
    rewritten as 'rsa.set_key(rsa.n, 3, rsa.d)'.
    [Feature #12324]

  • ext/openssl/ossl_pkey*.[ch]: Use the new accessor functions. Implement
    RSA#set_{key,factors,crt_params}, DSA#set_{key,pqg}, DH#set_{key,pqg}.
    Emit a warning with rb_warning() when old setter methods are used.

  • test/drb/ut_array_drbssl.rb, test/drb/ut_drb_drbssl.rb,
    test/rubygems/test_gem_remote_fetcher.rb: Don't set a priv_key for DH
    object that are used in tmp_dh_callback. Generating a new key pair
    every time should be fine - actually the private exponent is ignored
    in OpenSSL >= 1.0.2f/1.0.1r even if we explicitly set.
    https://www.openssl.org/news/secadv/20160128.txt

Revision 55287
Added by rhe over 1 year ago

openssl: adapt to OpenSSL 1.1.0 opaque structs

  • ext/openssl/extconf.rb: Check existence of accessor functions that
    don't exist in OpenSSL 0.9.8. OpenSSL 1.1.0 made most of its
    structures opaque and requires use of these accessor functions.
    [Feature #12324]

  • ext/openssl/openssl_missing.[ch]: Implement them if missing.

  • ext/openssl/ossl*.c: Use these accessor functions.

  • test/openssl/test_hmac.rb: Add missing test for HMAC#reset.

Revision 55287
Added by rhe over 1 year ago

openssl: adapt to OpenSSL 1.1.0 opaque structs

  • ext/openssl/extconf.rb: Check existence of accessor functions that
    don't exist in OpenSSL 0.9.8. OpenSSL 1.1.0 made most of its
    structures opaque and requires use of these accessor functions.
    [Feature #12324]

  • ext/openssl/openssl_missing.[ch]: Implement them if missing.

  • ext/openssl/ossl*.c: Use these accessor functions.

  • test/openssl/test_hmac.rb: Add missing test for HMAC#reset.

Revision 55288
Added by rhe over 1 year ago

openssl: avoid d2i_ASN1_BOOLEAN()

  • ext/openssl/ossl_asn1.c (decode_bool): Do the same thing as d2i_ASN1_BOOLEAN() does by ourselves. This function is removed in OpenSSL 1.1.0. [Feature #12324]

Revision 55288
Added by rhe over 1 year ago

openssl: avoid d2i_ASN1_BOOLEAN()

  • ext/openssl/ossl_asn1.c (decode_bool): Do the same thing as d2i_ASN1_BOOLEAN() does by ourselves. This function is removed in OpenSSL 1.1.0. [Feature #12324]

Revision 55289
Added by rhe over 1 year ago

openssl: use SSL_is_server()

  • ext/openssl/extconf.rb: Check existence of SSL_is_server(). This
    function was introduced in OpenSSL 1.0.2.
    [Feature #12324]

  • ext/openssl/openssl_missing.h: Implement SSL_is_server() if missing.

  • ext/openssl/ossl_ssl.c (ssl_info_cb): Use SSL_is_server() to see if
    the SSL is server. The state machine in OpenSSL was rewritten and
    SSL_get_state() no longer returns SSL_ST_ACCEPT.

(ossl_ssl_cipher_to_ary, ossl_sslctx_session_get_cb): Add some
consts to suppress warning.

Revision 55289
Added by rhe over 1 year ago

openssl: use SSL_is_server()

  • ext/openssl/extconf.rb: Check existence of SSL_is_server(). This
    function was introduced in OpenSSL 1.0.2.
    [Feature #12324]

  • ext/openssl/openssl_missing.h: Implement SSL_is_server() if missing.

  • ext/openssl/ossl_ssl.c (ssl_info_cb): Use SSL_is_server() to see if
    the SSL is server. The state machine in OpenSSL was rewritten and
    SSL_get_state() no longer returns SSL_ST_ACCEPT.

(ossl_ssl_cipher_to_ary, ossl_sslctx_session_get_cb): Add some
consts to suppress warning.

Revision 55294
Added by rhe over 1 year ago

openssl: fix free function of OpenSSL::Cipher

  • ext/openssl/ossl_cipher.c (ossl_cipher_free): Use EVP_CIPHER_CTX_free() to free EVP_CIPHER_CTX allocated by EVP_CIPHER_CTX_new(). [Feature #12324]

Revision 55294
Added by rhe over 1 year ago

openssl: fix free function of OpenSSL::Cipher

  • ext/openssl/ossl_cipher.c (ossl_cipher_free): Use EVP_CIPHER_CTX_free() to free EVP_CIPHER_CTX allocated by EVP_CIPHER_CTX_new(). [Feature #12324]

Revision 55304
Added by rhe over 1 year ago

openssl: avoid deprecated version-specific ssl methods if necessary

  • ext/openssl/extconf.rb: Check for SSL_CTX_set_min_proto_version()
    macro added in OpenSSL 1.1.0. Version-specific methods, such as
    TLSv1_method(), are deprecated in OpenSSL 1.1.0. We need to use
    version-flexible methods (TLS_*method() or SSLv23_*method()) and
    disable other protocol versions as necessary.
    [Feature #12324]

  • ext/openssl/ossl_ssl.c: Use SSL_CTX_set_{min,max}_proto_version() to
    fix the protocol version.

Revision 55304
Added by rhe over 1 year ago

openssl: avoid deprecated version-specific ssl methods if necessary

  • ext/openssl/extconf.rb: Check for SSL_CTX_set_min_proto_version()
    macro added in OpenSSL 1.1.0. Version-specific methods, such as
    TLSv1_method(), are deprecated in OpenSSL 1.1.0. We need to use
    version-flexible methods (TLS_*method() or SSLv23_*method()) and
    disable other protocol versions as necessary.
    [Feature #12324]

  • ext/openssl/ossl_ssl.c: Use SSL_CTX_set_{min,max}_proto_version() to
    fix the protocol version.

Revision 55309
Added by rhe over 1 year ago

openssl: add SSL::SSLContext#security_level{=,}

  • ext/openssl/extconf.rb: Check for SSL_CTX_get_security_level().
    OpenSSL 1.1.0 introduced "security level".
    [Feature #12324]

  • ext/openssl/ossl_ssl.c (ossl_sslctx_{get,set}_security_level): Add
    SSLContext#security_level and #security_level=.

  • test/openssl/test_ssl.rb (test_security_level): Add test. ...but this
    doesn't actually test it. Because #security_level= is necessary in
    order to run other tests on OpenSSL 1.1.0, go without tests for now.
    Will fix after converting SSLContext#key= and #cert= to normal methods.

Revision 55309
Added by rhe over 1 year ago

openssl: add SSL::SSLContext#security_level{=,}

  • ext/openssl/extconf.rb: Check for SSL_CTX_get_security_level().
    OpenSSL 1.1.0 introduced "security level".
    [Feature #12324]

  • ext/openssl/ossl_ssl.c (ossl_sslctx_{get,set}_security_level): Add
    SSLContext#security_level and #security_level=.

  • test/openssl/test_ssl.rb (test_security_level): Add test. ...but this
    doesn't actually test it. Because #security_level= is necessary in
    order to run other tests on OpenSSL 1.1.0, go without tests for now.
    Will fix after converting SSLContext#key= and #cert= to normal methods.

Revision 55314
Added by rhe over 1 year ago

openssl: adjust tests for OpenSSL 1.1.0

This fixes make test-all TESTS=openssl with OpenSSL master.

  • test/openssl/test_x509name.rb: Don't register OID for 'emailAddress'
    and 'serialNumber'. A recent change in OpenSSL made OBJ_create()
    reject an already existing OID. They were needed to run tests with
    OpenSSL 0.9.6 which is now unsupported.
    https://git.openssl.org/gitweb/?p=openssl.git;a=commit;h=52832e470f5fe8c222249ae5b539aeb3c74cdb25
    [Feature #12324]

  • test/openssl/test_ssl_session.rb (test_server_session): Duplicate
    SSL::Session before re-adding to the session store. OpenSSL 1.1.0
    starts rejecting SSL_SESSION once removed by SSL_CTX_remove_session().
    https://git.openssl.org/gitweb/?p=openssl.git;a=commit;h=7c2d4fee2547650102cd16d23f8125b76112ae75

  • test/openssl/test_pkey_ec.rb (setup): Remove X25519 from @keys. X25519
    is new in OpenSSL 1.1.0 but this is for key agreement and not for
    signing.

  • test/openssl/test_pair.rb, test/openssl/test_ssl.rb,
    test/openssl/utils.rb: Set security level to 0 when using aNULL cipher
    suites.

  • test/openssl/utils.rb: Use 1024 bits DSA key for client certificates.

  • test/openssl/test_engine.rb: Run each test in separate process.
    We can no longer cleanup engines explicitly as ENGINE_cleanup() was
    removed.
    https://git.openssl.org/gitweb/?p=openssl.git;a=commit;h=6d4fb1d59e61aacefa25edc4fe5acfe1ac93f743

  • ext/openssl/ossl_engine.c (ossl_engine_s_cleanup): Add a note to the
    RDoc for Engine.cleanup.

  • ext/openssl/lib/openssl/digest.rb: Don't define constants for DSS,
    DSS1 and SHA(-0) when using with OpenSSL 1.1.0. They are removed.

  • test/openssl/test_digest.rb, test/openssl/test_pkey_dsa.rb,
    test/openssl/test_pkey_dsa.rb, test/openssl/test_ssl.rb,
    test/openssl/test_x509cert.rb, test/openssl/test_x509req.rb: Don't
    test unsupported hash functions.

Revision 55314
Added by rhe over 1 year ago

openssl: adjust tests for OpenSSL 1.1.0

This fixes make test-all TESTS=openssl with OpenSSL master.

  • test/openssl/test_x509name.rb: Don't register OID for 'emailAddress'
    and 'serialNumber'. A recent change in OpenSSL made OBJ_create()
    reject an already existing OID. They were needed to run tests with
    OpenSSL 0.9.6 which is now unsupported.
    https://git.openssl.org/gitweb/?p=openssl.git;a=commit;h=52832e470f5fe8c222249ae5b539aeb3c74cdb25
    [Feature #12324]

  • test/openssl/test_ssl_session.rb (test_server_session): Duplicate
    SSL::Session before re-adding to the session store. OpenSSL 1.1.0
    starts rejecting SSL_SESSION once removed by SSL_CTX_remove_session().
    https://git.openssl.org/gitweb/?p=openssl.git;a=commit;h=7c2d4fee2547650102cd16d23f8125b76112ae75

  • test/openssl/test_pkey_ec.rb (setup): Remove X25519 from @keys. X25519
    is new in OpenSSL 1.1.0 but this is for key agreement and not for
    signing.

  • test/openssl/test_pair.rb, test/openssl/test_ssl.rb,
    test/openssl/utils.rb: Set security level to 0 when using aNULL cipher
    suites.

  • test/openssl/utils.rb: Use 1024 bits DSA key for client certificates.

  • test/openssl/test_engine.rb: Run each test in separate process.
    We can no longer cleanup engines explicitly as ENGINE_cleanup() was
    removed.
    https://git.openssl.org/gitweb/?p=openssl.git;a=commit;h=6d4fb1d59e61aacefa25edc4fe5acfe1ac93f743

  • ext/openssl/ossl_engine.c (ossl_engine_s_cleanup): Add a note to the
    RDoc for Engine.cleanup.

  • ext/openssl/lib/openssl/digest.rb: Don't define constants for DSS,
    DSS1 and SHA(-0) when using with OpenSSL 1.1.0. They are removed.

  • test/openssl/test_digest.rb, test/openssl/test_pkey_dsa.rb,
    test/openssl/test_pkey_dsa.rb, test/openssl/test_ssl.rb,
    test/openssl/test_x509cert.rb, test/openssl/test_x509req.rb: Don't
    test unsupported hash functions.

Revision 55315
Added by rhe over 1 year ago

open-uri: regenerate server certificates used in tests

  • test/open-uri/test_ssl.rb: Regenerate test certificates. The test CA certificate was incorrectly generated. A CA certificate must have the basic constraints extension with cA bit set to TRUE. OpenSSL <= 1.0.2 allowed the error when the certificate is in the trusted store but OpenSSL 1.1.0 no longer does. [Feature #12324]

Revision 55315
Added by rhe over 1 year ago

open-uri: regenerate server certificates used in tests

  • test/open-uri/test_ssl.rb: Regenerate test certificates. The test CA certificate was incorrectly generated. A CA certificate must have the basic constraints extension with cA bit set to TRUE. OpenSSL <= 1.0.2 allowed the error when the certificate is in the trusted store but OpenSSL 1.1.0 no longer does. [Feature #12324]

Revision 55317
Added by rhe over 1 year ago

rubygems: regenerate server certificates used in tests

Revision 55317
Added by rhe over 1 year ago

rubygems: regenerate server certificates used in tests

Revision 55335
Added by rhe over 1 year ago

openssl: fix build with OpenSSL 1.1.0 and no pkg-config

  • ext/openssl/extconf.rb: Check for CRYPTO_malloc() and SSL_new(). OpenSSL_add_all_digests() and SSL_library_init() are deprecated and converted to macros in OpenSSL 1.1.0. [Feature #12324]

Revision 55335
Added by rhe over 1 year ago

openssl: fix build with OpenSSL 1.1.0 and no pkg-config

  • ext/openssl/extconf.rb: Check for CRYPTO_malloc() and SSL_new(). OpenSSL_add_all_digests() and SSL_library_init() are deprecated and converted to macros in OpenSSL 1.1.0. [Feature #12324]

Revision 55344
Added by rhe over 1 year ago

openssl: use ASN1_ENUMERATED_to_BN() if needed

  • ext/openssl/ossl_asn1.c (asn1integer_to_num): Use
    ASN1_ENUMERATED_to_BN() to convert an ASN1_ENUMERATED to a BN.
    Starting from OpenSSL 1.1.0, ASN1_INTEGER_to_BN() rejects
    non-ASN1_INTEGER objects. The format of INTEGER and ENUMERATED are
    almost identical so they behaved in the same way in OpenSSL <= 1.0.2.
    [Feature #12324]

  • test/openssl/test_asn1.rb (test_decode_enumerated): Test that it
    works.

Revision 55344
Added by rhe over 1 year ago

openssl: use ASN1_ENUMERATED_to_BN() if needed

  • ext/openssl/ossl_asn1.c (asn1integer_to_num): Use
    ASN1_ENUMERATED_to_BN() to convert an ASN1_ENUMERATED to a BN.
    Starting from OpenSSL 1.1.0, ASN1_INTEGER_to_BN() rejects
    non-ASN1_INTEGER objects. The format of INTEGER and ENUMERATED are
    almost identical so they behaved in the same way in OpenSSL <= 1.0.2.
    [Feature #12324]

  • test/openssl/test_asn1.rb (test_decode_enumerated): Test that it
    works.

Revision 55387
Added by rhe over 1 year ago

openssl: avoid test failure in test_engine.rb

  • test/openssl/test_engine.rb (test_openssl_engine_builtin, test_openssl_engine_by_id_string): Skip test if 'openssl' engine is already loaded. And test the number increased by Engine.load{_by_id,}, not the total count of loaded engines. Previously, we called OpenSSL::Engine.cleanup every time running a test case, but we no longer can do it. [Feature #12324]

Revision 55387
Added by rhe over 1 year ago

openssl: avoid test failure in test_engine.rb

  • test/openssl/test_engine.rb (test_openssl_engine_builtin, test_openssl_engine_by_id_string): Skip test if 'openssl' engine is already loaded. And test the number increased by Engine.load{_by_id,}, not the total count of loaded engines. Previously, we called OpenSSL::Engine.cleanup every time running a test case, but we no longer can do it. [Feature #12324]

Revision 55408
Added by rhe over 1 year ago

openssl: fix acesssor functions for RSA and DH in openssl_missing.h

  • ext/openssl/openssl_missing.h (DH_set0_pqg, RSA_set0_key): DH_set0_pqg() allows 'q' to be NULL. Fix a typo in RSA_set0_key(). Fixes r55285. [Feature #12324]

Revision 55408
Added by rhe over 1 year ago

openssl: fix acesssor functions for RSA and DH in openssl_missing.h

  • ext/openssl/openssl_missing.h (DH_set0_pqg, RSA_set0_key): DH_set0_pqg() allows 'q' to be NULL. Fix a typo in RSA_set0_key(). Fixes r55285. [Feature #12324]

Revision 55450
Added by rhe over 1 year ago

openssl: add 'const's required in OpenSSL master

Revision 55450
Added by rhe over 1 year ago

openssl: add 'const's required in OpenSSL master

Revision 55460
Added by rhe over 1 year ago

rubygems: fix test for OpenSSL 1.1.0

Revision 55460
Added by rhe over 1 year ago

rubygems: fix test for OpenSSL 1.1.0

History

#1 [ruby-core:75351] Updated by mcr (Michael Richardson) over 1 year ago

I tried your git tree, at: commit 7085ed6411718538c0f47f8281b9089d21d4426d
I did have openssl-pre5 installed, which failed, and I installed from openssl master and rebuilt ruby again, did an rvm mount on the results, and I got:

ext-ruby-2.4-ssl :004 > OpenSSL::PKey::EC.builtin_curves.include?(['X25519','X25519'])
=> true

so that totally looks right, and I now can play with the new IRTF CFRG curve...

#2 [ruby-core:75353] Updated by mcr (Michael Richardson) over 1 year ago

Please see gist: https://gist.github.com/mcr/73e10792abd529c5f53562ed275b7c1b
(I could remove the rake/rails stuff around it if you like)

with X25519 selected as the curve, I get:

%rake highway:selfsigned
rake aborted!
OpenSSL::X509::CertificateError: EVP lib
/corp/projects/pandora/highway/lib/tasks/cert.rake:53:in sign'
/corp/projects/pandora/highway/lib/tasks/cert.rake:53:in
block (2 levels) in '
/home/mcr/.rvm/gems/ext-ruby-2.4-ssl/bin/ruby_executable_hooks:15:in eval'
/home/mcr/.rvm/gems/ext-ruby-2.4-ssl/bin/ruby_executable_hooks:15:in
'
Tasks: TOP => highway:selfsigned

With the secp112r1 curve, I get a good certificate generated:

% openssl x509 -in db/cert/vendor_secp112r1.crt -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: ecdsa-with-SHA256
...
ASN1 OID: secp112r1

I'm investigating generating the certificate with the openssl ca driver, to help isolate it as an openssl vs ruby integration issue.

#3 [ruby-core:75361] Updated by rhenium (Kazuki Yamaguchi) over 1 year ago

It's because X25519 is an algorithm for key exchange. Ed25519, the corresponding (sharing the elliptic curve) signature scheme, is not yet implemented in OpenSSL. There are tickets but it looks like nobody is working on it at the moment.

https://rt.openssl.org/Ticket/Display.html?id=4077
https://github.com/openssl/openssl/issues/487

#4 [ruby-core:75364] Updated by rhenium (Kazuki Yamaguchi) over 1 year ago

I made v2 of the patch:

https://github.com/ruby/ruby/compare/trunk...rhenium:feature/openssl-110-v2

Changes:

  • Re-ordered commits in more meaningful order (and squashed some small commits).

    I'm so sorry if you already started reviewing. I also attach the interdiff (this is small).

  • When using with an older OpenSSL, make OpenSSL::SSL::SSLContext#security_level= raise NotImplementedError if the argument is not 0.


The patches 01-05 are small cleanups/fixes.

  • [01/35] ext/openssl: check if SSL_CTX_clear_options() is available
  • [02/35] ext/openssl: fix ex_data handling for X509_STORE
  • [03/35] ext/openssl: always use our implementation of SSL_SESSION_cmp()
  • [04/35] ext/openssl: no need to check OPENSSL_FIPS in extconf.rb
  • [05/35] ext/openssl: remove 'extern "C" { }' blocks

06 drops OpenSSL -0.9.7 support.

  • [06/35] ext/openssl: drop support for OpenSSL 0.9.6/0.9.7

Then 07 removes OPENSSL_NO_HMAC support which has been broken for years.

  • [07/35] ext/openssl: drop support for OPENSSL_NO_HMAC

08-14 handles removed / deprecated functions.

  • [08/35] ext/openssl: include openssl/asn1.h instead of openssl/asn1_mac.h
  • [09/35] ext/openssl: avoid deprecated BN primes functions
  • [10/35] ext/openssl: disable OpenSSL::Random.pseudo_bytes if deprecated
  • [11/35] ext/openssl: d2i_ASN1_BOOLEAN is removed
  • [12/35] ext/openssl: avoid using deprecated protocol version specific methods
  • [13/35] ext/openssl: SSL_state() is removed
  • [14/35] ext/openssl: setting locking callbacks is no longer needed

15-24 fix compilation errors by replacing direct struct accesses with accessor functions.

  • [15/35] ext/openssl: use EVP_CIPHER_CTX_{new,free} to allocate EVP_CIPHER_CTX
  • [16/35] ext/openssl: use HMAC_CTX_{new,free,reset} to allocate HMAC_CTX
  • [17/35] ext/openssl: use EVP_MD_CTX_new() to allocate EVP_MD_CTX
  • [18/35] ext/openssl: OCSP_SINGLERESP and OCSP_CERTID are also made opaque
  • [19/35] ext/openssl: BIGNUM and BN_GENCB is made opaque
  • [20/35] ext/openssl: SSL_SESSION is made opaque
  • [21/35] ext/openssl: use *_up_ref() functions
  • [22/35] ext/openssl: X509* are made opaque
  • [25/35] ext/openssl: use SSL_CTX_get_ciphers()
  • [24/35] ext/openssl: EVP_PKEY, DH, DSA, RSA, EC_KEY are made opaque

Now it compiles. The patch 24 (↑) and 25-28 contain Ruby-land changes.

  • [25/35] ext/openssl: add SSLContext#security_level, #security_level=
  • [26/35] ext/openssl: add SSLContext#set_ecdh_curves
  • [27/35] ext/openssl: remove SHA, DSS, DSS1 if using OpenSSL 1.1.0
  • [28/35] ext/openssl: ENGINE.cleanup is no-op in OpenSSL 1.1.0

The next is TODO, I think this is a bug in OpenSSL.

  • [29/35] ext/openssl: avoid SEGV on Cipher.new("ChaCha20-Poly1305")

30-35 fixes some test cases that are affected by OpenSSL's changes.

  • [30/35] test/openssl: DSA256 is prohibited with security_level=1
  • [31/35] test/openssl: X25519 doesn't support signing
  • [32/35] test/openssl: fix test_server_session to dup the session
  • [33/35] test/openssl: don't test default session timeout
  • [34/35] test/open-uri: regenerate test certificates
  • [35/35] test/rubygems: regenerate certificates

#5 [ruby-core:75410] Updated by naruse (Yui NARUSE) over 1 year ago

  • Status changed from Open to Assigned
  • Assignee set to matz (Yukihiro Matsumoto)

I think you become a committer and commit it by yourself. (and maintain it)

#6 [ruby-core:75412] Updated by zzak (Zachary Scott) over 1 year ago

I'm happy for your help with Ruby's OpenSSL!

This is a rather large change, and would like to see it land in the (To Be Released) openssl gem:
https://github.com/ruby/openssl

If you're interested, I can give you commit to the openssl gem repo as well.

#7 [ruby-core:75414] Updated by naruse (Yui NARUSE) over 1 year ago

Zachary Scott wrote:

I'm happy for your help with Ruby's OpenSSL!

This is a rather large change, and would like to see it land in the (To Be Released) openssl gem:
https://github.com/ruby/openssl

If you're interested, I can give you commit to the openssl gem repo as well.

zzak (Zachary Scott) Could you explain him (or add explanation to README.rd of ruby/openssl) the relation and status between openssl gem and Ruby repo.

#8 [ruby-core:75415] Updated by rhenium (Kazuki Yamaguchi) over 1 year ago

Thanks! I'd be happy to maintain ext/openssl.

I have read #5481 and #9612 before, so I know that gemifying ext/openssl is in progress.

#9 [ruby-core:75428] Updated by vo.x (Vit Ondruch) over 1 year ago

even 0.9.8/1.0.0 series are no longer supported[1]

If I am not mistaken, 0.9.8 is still part of RHEL5/CentOS5 and they are still supported, although not by upstream.

#10 [ruby-core:75429] Updated by rhenium (Kazuki Yamaguchi) over 1 year ago

Vit Ondruch wrote:

even 0.9.8/1.0.0 series are no longer supported[1]

If I am not mistaken, 0.9.8 is still part of RHEL5/CentOS5 and they are still supported, although not by upstream.

Yes, that's exactly why I didn't drop them in this series, and it didn't reduce code greatly compared to dropping 0.9.6/0.9.7.

#11 [ruby-core:75506] Updated by rhenium (Kazuki Yamaguchi) over 1 year ago

Patch v3. It looks like OpenSSL 1.1.0 is delaying...

https://github.com/ruby/ruby/compare/trunk...rhenium:feature/openssl-110-v3

Changes:

  • Rebased to apply atop trunk.
  • [03/36] is new: I noticed OpenSSL::X509::Attribute#value= segfaults if the passed argument is bad.
  • Fixed my broken [v2 22/35]. It was not covered by test.

#12 [ruby-core:75508] Updated by rhenium (Kazuki Yamaguchi) over 1 year ago

matz (Yukihiro Matsumoto)

Can I have SVN access?

I'd love to help improving/maintaining Ruby and ext/openssl. Several of my patches including openssl-related and non-openssl-related, have landed with the help of nobu (Nobuyoshi Nakada) and @nurse. I was planning to send more patches, but it would be nice if I can commit without bothering other committers.

#13 [ruby-core:75511] Updated by hsbt (Hiroshi SHIBATA) over 1 year ago

I added this issue to agenda of developer meeting at next week.

https://bugs.ruby-lang.org/projects/ruby/wiki/DevelopersMeeting20160517Japan

Please wait a few days.

#14 [ruby-core:75512] Updated by rhenium (Kazuki Yamaguchi) over 1 year ago

Hiroshi SHIBATA wrote:

I added this issue to agenda of developer meeting at next week.

https://bugs.ruby-lang.org/projects/ruby/wiki/DevelopersMeeting20160517Japan

Please wait a few days.

Thanks!

#17 [ruby-core:75760] Updated by hsbt (Hiroshi SHIBATA) over 1 year ago

  • Assignee changed from matz (Yukihiro Matsumoto) to rhenium (Kazuki Yamaguchi)

I added commit-bit to Kazuki Yamaguchi-san few weeks ago.
Please do apply your proposal by yourself.

#18 Updated by Anonymous over 1 year ago

  • Status changed from Assigned to Closed

Applied in changeset r55282.


openssl: check existence of RAND_pseudo_bytes()

  • ext/openssl/extconf.rb: Check if RAND_pseudo_bytes() is usable. It is
    marked as deprecated in OpenSSL 1.1.0.
    [Feature #12324]

  • ext/openssl/ossl_rand.c: Disable Random.pseudo_bytes if
    RAND_pseudo_bytes() is unavailable.

  • test/openssl/test_random.rb: Don't test Random.pseudo_bytes if not
    defined.

#19 [ruby-core:75952] Updated by terceiro (Antonio Terceiro) over 1 year ago

  • Backport set to 2.1: UNKNOWN, 2.2: UNKNOWN, 2.3: UNKNOWN
  • Tracker changed from Feature to Bug

Hi, would it be possible to have a backport of this for Ruby 2.3? Debian 9 will be released with Ruby 2.3, and we will probably need to build against OpenSSL 1.1.0 between now and then.

#20 [ruby-core:76035] Updated by rhenium (Kazuki Yamaguchi) over 1 year ago

This is based on r55162 (openssl: drop OpenSSL 0.9.6/0.9.7 support) so backporting 1.1.0 support is not so simple.

Perhaps Debian should disable ext/openssl and bundle openssl gem (though not yet released)?

#21 [ruby-core:76184] Updated by terceiro (Antonio Terceiro) over 1 year ago

Kazuki Yamaguchi wrote:

This is based on r55162 (openssl: drop OpenSSL 0.9.6/0.9.7 support) so backporting 1.1.0 support is not so simple.

well, openssl 0.9.x is long gone from all supported Debian releases, so including that would not be a problem for Debian, but I understand why it could be an issues for Ruby 2.3 upstream. So if I take that, would I be able to also take the other patches and form a Debian-specific OpenSSL 1.1.0 support patch?

Perhaps Debian should disable ext/openssl and bundle openssl gem (though not yet released)?

That would be even better, but it's not clear to me what's the status of that standalone openssl package, e.g. is it a safe drop-in replacement from the openssl bundled with the interpreter? Do you have any pointers?

#22 [ruby-core:78693] Updated by normalperson (Eric Wong) 11 months ago

k@rhe.jp wrote:

Feature #12324: Support OpenSSL 1.1.0 (and drop support for 0.9.6/0.9.7)
https://bugs.ruby-lang.org/issues/12324

rhe: did you commit all of these?

I notice SSL_CTX_clear_options is still missing on an ancient
CentOS 5.x machine with 0.9.8e. Thanks.

#23 [ruby-core:78701] Updated by rhenium (Kazuki Yamaguchi) 11 months ago

On Sat, Dec 17, 2016 at 01:31:12AM +0000, Eric Wong wrote:

k@rhe.jp wrote:

Feature #12324: Support OpenSSL 1.1.0 (and drop support for 0.9.6/0.9.7)
https://bugs.ruby-lang.org/issues/12324

rhe: did you commit all of these?

I notice SSL_CTX_clear_options is still missing on an ancient
CentOS 5.x machine with 0.9.8e. Thanks.

I dropped that patch. I found the OpenSSL 0.9.8e package in RHEL/CentOS
5 has backported SSL_CTX_clear_options() as part of CVE-2009-3555 fix,
and no one except them would use < 0.9.8m anymore.

#24 [ruby-core:78702] Updated by normalperson (Eric Wong) 11 months ago

Kazuki Yamaguchi k@rhe.jp wrote:

On Sat, Dec 17, 2016 at 01:31:12AM +0000, Eric Wong wrote:

k@rhe.jp wrote:

Feature #12324: Support OpenSSL 1.1.0 (and drop support for 0.9.6/0.9.7)
https://bugs.ruby-lang.org/issues/12324

rhe: did you commit all of these?

I notice SSL_CTX_clear_options is still missing on an ancient
CentOS 5.x machine with 0.9.8e. Thanks.

I dropped that patch. I found the OpenSSL 0.9.8e package in RHEL/CentOS
5 has backported SSL_CTX_clear_options() as part of CVE-2009-3555 fix,
and no one except them would use < 0.9.8m anymore.

Thanks for the response. I'll work with my sysadmins and see
if they're willing to upgrade OpenSSL; but they may not :<

If they are not, would you be willing to add
SSL_CTX_clear_options? I could carry the patch myself, but
maybe there are other ancient places who don't patch :<

Fwiw, I recommend using functions with custom name to emulate
missing functions, and #define to the official name:

static unsigned long
rb_SSL_CTX_clear_options(SSL_CTX *ctx, unsigned long op)
{
return ctx->options &= ~op;
}
#define SSL_CTX_clear_options(ctx,op) rb_SSL_CTX_clear_options((ctx),(op))

It makes debugging easier since the internal symbol name won't
conflict with the official version.

Thanks.

#25 [ruby-core:78711] Updated by rhenium (Kazuki Yamaguchi) 11 months ago

On Sat, Dec 17, 2016 at 09:09:49AM +0000, Eric Wong wrote:

Kazuki Yamaguchi k@rhe.jp wrote:

On Sat, Dec 17, 2016 at 01:31:12AM +0000, Eric Wong wrote:

k@rhe.jp wrote:

Feature #12324: Support OpenSSL 1.1.0 (and drop support for 0.9.6/0.9.7)
https://bugs.ruby-lang.org/issues/12324

rhe: did you commit all of these?

I notice SSL_CTX_clear_options is still missing on an ancient
CentOS 5.x machine with 0.9.8e. Thanks.

I dropped that patch. I found the OpenSSL 0.9.8e package in RHEL/CentOS
5 has backported SSL_CTX_clear_options() as part of CVE-2009-3555 fix,
and no one except them would use < 0.9.8m anymore.

Thanks for the response. I'll work with my sysadmins and see
if they're willing to upgrade OpenSSL; but they may not :<

If they are not, would you be willing to add
SSL_CTX_clear_options? I could carry the patch myself, but
maybe there are other ancient places who don't patch :<

I'm fine with adding if it helps. I don't want to imagine such systems
with 6 years unpatched OpenSSL, though.

I've added to the GitHub repository now:

https://github.com/ruby/openssl/commit/fd2ea0c41d55f3d0b34edb1d25c673bc004f6913

Will import to trunk in a few days together with a few more minor bug fixes.

#26 [ruby-core:78731] Updated by normalperson (Eric Wong) 11 months ago

Kazuki Yamaguchi k@rhe.jp wrote:

I'm fine with adding if it helps. I don't want to imagine such systems
with 6 years unpatched OpenSSL, though.

I've added to the GitHub repository now:

Thanks! Yeah, security is an afterthought, for me :<

Also available in: Atom PDF