Project

General

Profile

Bug #15468

Net::Protocol::BufferedIO#write raises NoMethodError when sending large multi-byte string

Added by eitoball (Eito Katagiri) 7 months ago. Updated 6 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
ruby -v:
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
[ruby-dev:50734]

Description

ruby-2.6.0で大きなJSON文字列をNet::HTTPでPOSTすると以下のように NoMothodError が発生するようになりました。

$ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: 'あ'*100_000 }.to_json, 'Content-Type' => 'application/json')"
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
Traceback (most recent call last):
        19: from -e:1:in `<main>'
        18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
        17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
        16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
        15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
        14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
        13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
        12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
        11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
        10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
         9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
         8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
         7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
         6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
         5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
         4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
         3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
         2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
         1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)

理由は、Net::Protocol::BufferedIO#write0で、String#[]で送信されていない文字列を求める際に文字列にマルチバイト文字が含まれているとIO#write_nonblockで返ってくる送信されたバイト数が文字数を超えるため、nilになるためだと考えています。https://github.com/ruby/ruby/pull/2058 を送ってみました。

Associated revisions

Revision 66582
Added by nobu (Nobuyoshi Nakada) 7 months ago

Fix Net::Protocol::BufferedIO#write when sending large multi-byte string

This commit should fix Net::Protocol::BufferedIO#write when sending
large multi-byte string like following example.

$ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: 'あ'*100_000 }.to_json, 'Content-Type' => 'application/json')"
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
Traceback (most recent call last):
        19: from -e:1:in `<main>'
        18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
        17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
        16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
        15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
        14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
        13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
        12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
        11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
        10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
         9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
         8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
         7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
         6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
         5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
         4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
         3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
         2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
         1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)

[Fix GH-2058]

From: Eito Katagiri eitoball@gmail.com

Revision 9548df04
Added by naruse (Yui NARUSE) 6 months ago

merge revision(s) 66582: [Backport #15468][Backport #15472]

    Fix Net::Protocol::BufferedIO#write when sending large multi-byte string

    This commit should fix Net::Protocol::BufferedIO#write when sending
    large multi-byte string like following example.

    ```
    $ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: '?'*100_000 }.to_json, 'Content-Type' => 'application/json')"
    ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
    Traceback (most recent call last):
            19: from -e:1:in `<main>'
            18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
            17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
            16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
            15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
            14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
            13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
            12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
            11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
            10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
             9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
             8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
             7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
             6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
             5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
             4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
             3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
             2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
             1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
    lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)
    ```

    [Fix GH-2058]

    From: Eito Katagiri <eitoball@gmail.com>

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_6@66799 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Revision 66799
Added by naruse (Yui NARUSE) 6 months ago

merge revision(s) 66582: [Backport #15468][Backport #15472]

Fix Net::Protocol::BufferedIO#write when sending large multi-byte string

This commit should fix Net::Protocol::BufferedIO#write when sending
large multi-byte string like following example.

```
$ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: '?'*100_000 }.to_json, 'Content-Type' => 'application/json')"
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
Traceback (most recent call last):
        19: from -e:1:in `<main>'
        18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
        17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
        16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
        15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
        14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
        13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
        12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
        11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
        10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
         9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
         8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
         7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
         6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
         5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
         4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
         3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
         2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
         1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)
```

[Fix GH-2058]

From: Eito Katagiri <eitoball@gmail.com>

History

Updated by rafaelfranca (Rafael França) 6 months ago

Can we get this issue closed and the with right backport field values?

#2

Updated by nobu (Nobuyoshi Nakada) 6 months ago

  • Backport changed from 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN to 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: REQUIRED
  • Status changed from Open to Closed
#3

Updated by hsbt (Hiroshi SHIBATA) 6 months ago

  • Backport changed from 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: REQUIRED to 2.4: DONTNEED, 2.5: DONTNEED, 2.6: REQUIRED

Updated by naruse (Yui NARUSE) 6 months ago

  • Backport changed from 2.4: DONTNEED, 2.5: DONTNEED, 2.6: REQUIRED to 2.4: DONTNEED, 2.5: DONTNEED, 2.6: DONE

ruby_2_6 r66799 merged revision(s) 66582.

Updated by ioquatix (Samuel Williams) 6 months ago

Can we please fast track the release of this? It's breaking a ton of stuff.

Stepping back a little bit from this issue, I think we see a picture of Ruby where:

I feel like some improvements to handling binary data could be a very useful addition to Ruby.

I welcome further discussion here and on the above issues so that we can avoid issues like this in the future. Ideally, we provide APIs such that this kind of bug is impossible.

Also available in: Atom PDF