Bug #7463

OpenSSL::TestSSL#test_npn_* fail with "string contains null byte" on big-endian machines

Added by Naohisa Goto over 1 year ago. Updated over 1 year ago.

[ruby-core:50292]
Status:Closed
Priority:Normal
Assignee:Martin Bosslet
Category:ext
Target version:-
ruby -v:ruby 2.0.0dev (2012-11-28) [sparc64-solaris2.10] Backport:

Description

On Solaris10 running in sparc64 CPU, during make test-all, the following 4 tests raise "ArgumentError: string contains null byte". Patch attached fix the bug.

OpenSSL::TestSSL#testnpnprotocolselectionary = 0.05 s = E
OpenSSL::TestSSL#testnpnprotocolselectioncancel = 0.05 s = E
OpenSSL::TestSSL#testnpnprotocolselectionenum = 0.05 s = E
OpenSSL::TestSSL#testnpnselectedprotocoltoo_long = 0.05 s = E

49) Error:
testnpnprotocolselectioncancel(OpenSSL::TestSSL):
ArgumentError: string contains null byte
/HOMEDIR/.ext/common/openssl/ssl.rb:172:in initialize'
/HOMEDIR/.ext/common/openssl/ssl.rb:172:in
new'
/HOMEDIR/.ext/common/openssl/ssl.rb:172:in accept'
/HOMEDIR/test/openssl/utils.rb:242:in
block in serverloop'
/HOMEDIR/test/openssl/utils.rb:251:in loop'
/HOMEDIR/test/openssl/utils.rb:251:in
server
loop'
/HOMEDIR/test/openssl/utils.rb:288:in `block in start_server'

50) Error:
testnpnprotocolselectionary(OpenSSL::TestSSL):
ArgumentError: string contains null byte
/HOMEDIR/.ext/common/openssl/ssl.rb:172:in initialize'
/HOMEDIR/.ext/common/openssl/ssl.rb:172:in
new'
/HOMEDIR/.ext/common/openssl/ssl.rb:172:in accept'
/HOMEDIR/test/openssl/utils.rb:242:in
block in serverloop'
/HOMEDIR/test/openssl/utils.rb:251:in loop'
/HOMEDIR/test/openssl/utils.rb:251:in
server
loop'
/HOMEDIR/test/openssl/utils.rb:288:in `block in start_server'

51) Error:
testnpnprotocolselectionenum(OpenSSL::TestSSL):
ArgumentError: string contains null byte
/HOMEDIR/.ext/common/openssl/ssl.rb:172:in initialize'
/HOMEDIR/.ext/common/openssl/ssl.rb:172:in
new'
/HOMEDIR/.ext/common/openssl/ssl.rb:172:in accept'
/HOMEDIR/test/openssl/utils.rb:242:in
block in serverloop'
/HOMEDIR/test/openssl/utils.rb:251:in loop'
/HOMEDIR/test/openssl/utils.rb:251:in
server
loop'
/HOMEDIR/test/openssl/utils.rb:288:in `block in start_server'

53) Error:
testnpnselectedprotocoltoolong(OpenSSL::TestSSL):
ArgumentError: string contains null byte
/HOMEDIR/.ext/common/openssl/ssl.rb:172:in initialize'
/HOMEDIR/.ext/common/openssl/ssl.rb:172:in
new'
/HOMEDIR/.ext/common/openssl/ssl.rb:172:in accept'
/HOMEDIR/test/openssl/utils.rb:242:in
block in server
loop'
/HOMEDIR/test/openssl/utils.rb:251:in loop'
/HOMEDIR/test/openssl/utils.rb:251:in
serverloop'
/HOMEDIR/test/openssl/utils.rb:288:in `block in start
server'

The cause of the bug is the following code in sslnpnencodeprotocoli in ext/openssl/ossl_ssl.c.

 int len = RSTRING_LENINT(cur);                                             
 if (len < 1 || len > 255)                                                  
    ossl_raise(eSSLError, "Advertised protocol must have length 1..255");   
 /* Encode the length byte */                                               
 rb_str_buf_cat(encoded, (const char *) &len, 1);                           

The above code treats "int len" as byte sequence that differs between little-endian and big-endian. In big-endian machines, when 1<=len<=255, the first byte of "len" is always 0.

20121128-openssl-bigendian.patch Magnifier (648 Bytes) Naohisa Goto, 11/29/2012 12:29 PM

Associated revisions

Revision 37996
Added by Naohisa Goto over 1 year ago

  • ext/openssl/osslssl.c (sslnpnencodeprotocol_i): fix byte order issue on big-endian architecture [Bug #7463]

History

#1 Updated by Naohisa Goto over 1 year ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

This issue was solved with changeset r37996.
Naohisa, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.


  • ext/openssl/osslssl.c (sslnpnencodeprotocol_i): fix byte order issue on big-endian architecture [Bug #7463]

Also available in: Atom PDF