Bug #16284: Net/HTTP consuming multiple times more memory compared to other libraries

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin18]
• Backport: 2.5: UNKNOWN, 2.6: UNKNOWN

[zogash (Milovan Zogovic)]

I have an issue where Net/HTTP library is causing very high memory usage when requesting for medium to large sized content.

Here is the script to reproduce the issue:

gem 'excon'

require 'uri'
require 'net/http'
require 'excon'

uri = URI('https://file-examples.com/wp-content/uploads/2017/04/file_example_MP4_1920_18MG.mp4')

def print_memory
  mem = `ps -o rss -p #{Process::pid}`.chomp.split("\n").last.strip.to_i
  puts "Memory: #{mem/1024} MB"
end

25.times do
  case ARGV[0]
  when 'EXCON' then Excon.get(uri.to_s)
  when 'NET' then Net::HTTP.get_response(uri)
  end
  print_memory
end

The script downloads 18MB size video file 25 times in a row and prints the process memory. After 25 iterations, Net/HTTP ends up almost 300MB of ram, while Excon only 60MB. Here is the complete result of RAM size the process used after each iteration:

> Excon
Memory: 41 MB
Memory: 60 MB
Memory: 95 MB
Memory: 58 MB
Memory: 95 MB
Memory: 49 MB
Memory: 55 MB
Memory: 51 MB
Memory: 71 MB
Memory: 66 MB
Memory: 58 MB
Memory: 65 MB
Memory: 89 MB
Memory: 76 MB
Memory: 83 MB
Memory: 64 MB
Memory: 75 MB
Memory: 98 MB
Memory: 64 MB
Memory: 58 MB
Memory: 63 MB
Memory: 69 MB
Memory: 75 MB
Memory: 70 MB
Memory: 54 MB

> Net::HTTP
Memory: 51 MB
Memory: 81 MB
Memory: 104 MB
Memory: 129 MB
Memory: 146 MB
Memory: 166 MB
Memory: 163 MB
Memory: 144 MB
Memory: 163 MB
Memory: 161 MB
Memory: 169 MB
Memory: 187 MB
Memory: 204 MB
Memory: 221 MB
Memory: 238 MB
Memory: 255 MB
Memory: 272 MB
Memory: 289 MB
Memory: 306 MB
Memory: 306 MB
Memory: 306 MB
Memory: 306 MB
Memory: 317 MB
Memory: 299 MB
Memory: 294 MB

I have tried with ruby 2.7.0-preview2 and the issue is present there as well.

[zogash (Milovan Zogovic)]

• Subject changed from Net/HTTP consuming multiple times more memories compared to other libraries to Net/HTTP consuming multiple times more memory compared to other libraries

[mame (Yusuke Endoh)]

Thank you for the report. Unfortunately, I cannot reproduce the issue. The result on my Linux machine is completely opposite against yours.

$ ruby -v
ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]

$ ruby t.rb EXCON
Memory: 60 MB
Memory: 81 MB
Memory: 104 MB
Memory: 83 MB
Memory: 107 MB
Memory: 124 MB
Memory: 92 MB
Memory: 109 MB
Memory: 126 MB
Memory: 148 MB
Memory: 150 MB
Memory: 154 MB
Memory: 136 MB
Memory: 136 MB
Memory: 153 MB
Memory: 170 MB
Memory: 189 MB
Memory: 191 MB
Memory: 192 MB
Memory: 192 MB
Memory: 159 MB
Memory: 159 MB
Memory: 159 MB
Memory: 175 MB
Memory: 193 MB

$ ruby t.rb NET
Memory: 49 MB
Memory: 61 MB
Memory: 62 MB
Memory: 79 MB
Memory: 96 MB
Memory: 97 MB
Memory: 99 MB
Memory: 104 MB
Memory: 80 MB
Memory: 97 MB
Memory: 97 MB
Memory: 104 MB
Memory: 63 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB

[zogash (Milovan Zogovic)]

Hey Yusuke, thanks for checking this out. I have tried this out on my friends machines and one of them has exactly the same results as I, but other has the results that are matching yours. All three of us are using MacBooks (although different models). However, running it in Docker (FROM ruby:2.6.5) it gave us consistent results where Excon is actually consuming more memory. Furthermore, I have also tried running this on Heroku dyno and here are the results:

Net/HTTP
Memory: 106 MB
Memory: 136 MB
Memory: 124 MB
Memory: 124 MB
Memory: 141 MB
Memory: 175 MB
Memory: 120 MB
Memory: 149 MB
Memory: 166 MB
Memory: 183 MB
Memory: 149 MB
Memory: 149 MB
Memory: 149 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB

EXCON
Memory: 117 MB
Memory: 125 MB
Memory: 152 MB
Memory: 139 MB
Memory: 166 MB
Memory: 182 MB
Memory: 200 MB
Memory: 147 MB
Memory: 164 MB
Memory: 181 MB
Memory: 148 MB
Memory: 164 MB
Memory: 181 MB
Memory: 201 MB
Memory: 201 MB
Memory: 207 MB
Memory: 223 MB
Memory: 239 MB
Memory: 240 MB
Memory: 240 MB
Memory: 240 MB
Memory: 240 MB
Memory: 240 MB
Memory: 240 MB
Memory: 240 MB

[mame (Yusuke Endoh)]

• Status changed from Open to Closed

Thanks. I think this is not a bug. Closing this issue.

I guess there may be room for improvement in Net::HTTP (and Excon?), though. Investigation and a patch are welcome.