Bug #7463
closedOpenSSL::TestSSL#test_npn_* fail with "string contains null byte" on big-endian machines
Description
On Solaris10 running in sparc64 CPU, during make test-all, the following 4 tests raise "ArgumentError: string contains null byte". Patch attached fix the bug.
OpenSSL::TestSSL#test_npn_protocol_selection_ary = 0.05 s = E
OpenSSL::TestSSL#test_npn_protocol_selection_cancel = 0.05 s = E
OpenSSL::TestSSL#test_npn_protocol_selection_enum = 0.05 s = E
OpenSSL::TestSSL#test_npn_selected_protocol_too_long = 0.05 s = E
-
Error:
test_npn_protocol_selection_cancel(OpenSSL::TestSSL):
ArgumentError: string contains null byte
/HOMEDIR/.ext/common/openssl/ssl.rb:172:ininitialize' /HOMEDIR/.ext/common/openssl/ssl.rb:172:in
new'
/HOMEDIR/.ext/common/openssl/ssl.rb:172:inaccept' /HOMEDIR/test/openssl/utils.rb:242:in
block in server_loop'
/HOMEDIR/test/openssl/utils.rb:251:inloop' /HOMEDIR/test/openssl/utils.rb:251:in
server_loop'
/HOMEDIR/test/openssl/utils.rb:288:in `block in start_server' -
Error:
test_npn_protocol_selection_ary(OpenSSL::TestSSL):
ArgumentError: string contains null byte
/HOMEDIR/.ext/common/openssl/ssl.rb:172:ininitialize' /HOMEDIR/.ext/common/openssl/ssl.rb:172:in
new'
/HOMEDIR/.ext/common/openssl/ssl.rb:172:inaccept' /HOMEDIR/test/openssl/utils.rb:242:in
block in server_loop'
/HOMEDIR/test/openssl/utils.rb:251:inloop' /HOMEDIR/test/openssl/utils.rb:251:in
server_loop'
/HOMEDIR/test/openssl/utils.rb:288:in `block in start_server' -
Error:
test_npn_protocol_selection_enum(OpenSSL::TestSSL):
ArgumentError: string contains null byte
/HOMEDIR/.ext/common/openssl/ssl.rb:172:ininitialize' /HOMEDIR/.ext/common/openssl/ssl.rb:172:in
new'
/HOMEDIR/.ext/common/openssl/ssl.rb:172:inaccept' /HOMEDIR/test/openssl/utils.rb:242:in
block in server_loop'
/HOMEDIR/test/openssl/utils.rb:251:inloop' /HOMEDIR/test/openssl/utils.rb:251:in
server_loop'
/HOMEDIR/test/openssl/utils.rb:288:in `block in start_server' -
Error:
test_npn_selected_protocol_too_long(OpenSSL::TestSSL):
ArgumentError: string contains null byte
/HOMEDIR/.ext/common/openssl/ssl.rb:172:ininitialize' /HOMEDIR/.ext/common/openssl/ssl.rb:172:in
new'
/HOMEDIR/.ext/common/openssl/ssl.rb:172:inaccept' /HOMEDIR/test/openssl/utils.rb:242:in
block in server_loop'
/HOMEDIR/test/openssl/utils.rb:251:inloop' /HOMEDIR/test/openssl/utils.rb:251:in
server_loop'
/HOMEDIR/test/openssl/utils.rb:288:in `block in start_server'
The cause of the bug is the following code in ssl_npn_encode_protocol_i in ext/openssl/ossl_ssl.c.
int len = RSTRING_LENINT(cur);
if (len < 1 || len > 255)
ossl_raise(eSSLError, "Advertised protocol must have length 1..255");
/* Encode the length byte */
rb_str_buf_cat(encoded, (const char *) &len, 1);
The above code treats "int len" as byte sequence that differs between little-endian and big-endian. In big-endian machines, when 1<=len<=255, the first byte of "len" is always 0.
Files
Updated by ngoto (Naohisa Goto) over 11 years ago
- Status changed from Open to Closed
- % Done changed from 0 to 100
This issue was solved with changeset r37996.
Naohisa, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.
- ext/openssl/ossl_ssl.c (ssl_npn_encode_protocol_i): fix byte order
issue on big-endian architecture [ruby-core:50292] [Bug #7463]