Bug #15468: Net::Protocol::BufferedIO#write raises NoMethodError when sending large multi-byte string

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
• Backport: 2.4: DONTNEED, 2.5: DONTNEED, 2.6: DONE

[eitoball (Eito Katagiri)]

ruby-2.6.0で大きなJSON文字列をNet::HTTPでPOSTすると以下のように NoMothodError が発生するようになりました。

$ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: 'あ'*100_000 }.to_json, 'Content-Type' => 'application/json')"
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
Traceback (most recent call last):
        19: from -e:1:in `<main>'
        18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
        17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
        16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
        15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
        14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
        13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
        12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
        11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
        10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
         9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
         8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
         7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
         6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
         5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
         4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
         3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
         2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
         1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)

理由は、Net::Protocol::BufferedIO#write0で、String#[]で送信されていない文字列を求める際に文字列にマルチバイト文字が含まれているとIO#write_nonblockで返ってくる送信されたバイト数が文字数を超えるため、nilになるためだと考えています。https://github.com/ruby/ruby/pull/2058 を送ってみました。

[rafaelfranca (Rafael França)]

Can we get this issue closed and the with right backport field values?

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed
• Backport changed from 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN to 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: REQUIRED

[hsbt (Hiroshi SHIBATA)]

• Backport changed from 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: REQUIRED to 2.4: DONTNEED, 2.5: DONTNEED, 2.6: REQUIRED

[naruse (Yui NARUSE)]

• Backport changed from 2.4: DONTNEED, 2.5: DONTNEED, 2.6: REQUIRED to 2.4: DONTNEED, 2.5: DONTNEED, 2.6: DONE

ruby_2_6 r66799 merged revision(s) 66582.

[ioquatix (Samuel Williams)]

Can we please fast track the release of this? It's breaking a ton of stuff.

Stepping back a little bit from this issue, I think we see a picture of Ruby where:

• Handling binary data can cause bugs like this (mismatch between binary and encoded string).
• Reading and writing strings is tricky, e.g. https://bugs.ruby-lang.org/issues/13626
• There are some odd edge cases which can cause problems, e.g. https://bugs.ruby-lang.org/issues/14975
• Reading and writing can put pressure on GC even though no additional strings should be allocated, see https://bugs.ruby-lang.org/issues/15560 for ideas to fix this.

I feel like some improvements to handling binary data could be a very useful addition to Ruby.

I welcome further discussion here and on the above issues so that we can avoid issues like this in the future. Ideally, we provide APIs such that this kind of bug is impossible.