Project

General

Profile

Actions

Bug #7463

closed

OpenSSL::TestSSL#test_npn_* fail with "string contains null byte" on big-endian machines

Added by ngoto (Naohisa Goto) over 11 years ago. Updated over 11 years ago.

Status:
Closed
Target version:
-
ruby -v:
ruby 2.0.0dev (2012-11-28) [sparc64-solaris2.10]
Backport:
[ruby-core:50292]

Description

On Solaris10 running in sparc64 CPU, during make test-all, the following 4 tests raise "ArgumentError: string contains null byte". Patch attached fix the bug.

OpenSSL::TestSSL#test_npn_protocol_selection_ary = 0.05 s = E
OpenSSL::TestSSL#test_npn_protocol_selection_cancel = 0.05 s = E
OpenSSL::TestSSL#test_npn_protocol_selection_enum = 0.05 s = E
OpenSSL::TestSSL#test_npn_selected_protocol_too_long = 0.05 s = E

  1. Error:
    test_npn_protocol_selection_cancel(OpenSSL::TestSSL):
    ArgumentError: string contains null byte
    /HOMEDIR/.ext/common/openssl/ssl.rb:172:in initialize' /HOMEDIR/.ext/common/openssl/ssl.rb:172:in new'
    /HOMEDIR/.ext/common/openssl/ssl.rb:172:in accept' /HOMEDIR/test/openssl/utils.rb:242:in block in server_loop'
    /HOMEDIR/test/openssl/utils.rb:251:in loop' /HOMEDIR/test/openssl/utils.rb:251:in server_loop'
    /HOMEDIR/test/openssl/utils.rb:288:in `block in start_server'

  2. Error:
    test_npn_protocol_selection_ary(OpenSSL::TestSSL):
    ArgumentError: string contains null byte
    /HOMEDIR/.ext/common/openssl/ssl.rb:172:in initialize' /HOMEDIR/.ext/common/openssl/ssl.rb:172:in new'
    /HOMEDIR/.ext/common/openssl/ssl.rb:172:in accept' /HOMEDIR/test/openssl/utils.rb:242:in block in server_loop'
    /HOMEDIR/test/openssl/utils.rb:251:in loop' /HOMEDIR/test/openssl/utils.rb:251:in server_loop'
    /HOMEDIR/test/openssl/utils.rb:288:in `block in start_server'

  3. Error:
    test_npn_protocol_selection_enum(OpenSSL::TestSSL):
    ArgumentError: string contains null byte
    /HOMEDIR/.ext/common/openssl/ssl.rb:172:in initialize' /HOMEDIR/.ext/common/openssl/ssl.rb:172:in new'
    /HOMEDIR/.ext/common/openssl/ssl.rb:172:in accept' /HOMEDIR/test/openssl/utils.rb:242:in block in server_loop'
    /HOMEDIR/test/openssl/utils.rb:251:in loop' /HOMEDIR/test/openssl/utils.rb:251:in server_loop'
    /HOMEDIR/test/openssl/utils.rb:288:in `block in start_server'

  4. Error:
    test_npn_selected_protocol_too_long(OpenSSL::TestSSL):
    ArgumentError: string contains null byte
    /HOMEDIR/.ext/common/openssl/ssl.rb:172:in initialize' /HOMEDIR/.ext/common/openssl/ssl.rb:172:in new'
    /HOMEDIR/.ext/common/openssl/ssl.rb:172:in accept' /HOMEDIR/test/openssl/utils.rb:242:in block in server_loop'
    /HOMEDIR/test/openssl/utils.rb:251:in loop' /HOMEDIR/test/openssl/utils.rb:251:in server_loop'
    /HOMEDIR/test/openssl/utils.rb:288:in `block in start_server'

The cause of the bug is the following code in ssl_npn_encode_protocol_i in ext/openssl/ossl_ssl.c.

 int len = RSTRING_LENINT(cur);                                             
 if (len < 1 || len > 255)                                                  
    ossl_raise(eSSLError, "Advertised protocol must have length 1..255");   
 /* Encode the length byte */                                               
 rb_str_buf_cat(encoded, (const char *) &len, 1);                           

The above code treats "int len" as byte sequence that differs between little-endian and big-endian. In big-endian machines, when 1<=len<=255, the first byte of "len" is always 0.


Files

20121128-openssl-bigendian.patch (648 Bytes) 20121128-openssl-bigendian.patch ngoto (Naohisa Goto), 11/29/2012 12:29 PM
Actions #1

Updated by ngoto (Naohisa Goto) over 11 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

This issue was solved with changeset r37996.
Naohisa, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.


  • ext/openssl/ossl_ssl.c (ssl_npn_encode_protocol_i): fix byte order
    issue on big-endian architecture [ruby-core:50292] [Bug #7463]
Actions

Also available in: Atom PDF

Like0
Like0