Feature #14320: [PATCH] open-uri: clear string after buffering

• Status: Closed
• Assignee: -
• Target version: -

[normalperson (Eric Wong)]

open-uri: clear string after buffering

Since r58846 (in Ruby 2.5), it is safe to clear the string
yielded to Net::HTTPResponse#read_body methods. This
reduces malloc garbage (anonymous RSS) using the Linux-only
script below:

before: user system total real
0.030000 0.250000 0.280000 ( 0.280511)
RssAnon: 60240 kB

after: user system total real
0.050000 0.223333 0.273333 ( 0.273118)
RssAnon: 6676 kB

---

warning this script requires 1G free space for buffering

require 'open-uri'
require 'socket'
require 'benchmark'

s = TCPServer.new('127.0.0.1', 0)
len = 1024 * 1024 * 1024
buf = ((0..255).map(&:chr).join * 128)
nr = len / buf.size
pid = fork do
c = s.accept
c.readpartial(16384).clear
c.write("HTTP/1.1 200 OK\r\n"
"Content-Length: #{len}\r\n"
"Content-Type: application/octet-stream\r\n"
"\r\n")
buf.freeze # speeds up IO#write slightly
nr.times { c.write(buf) }
c.close
end

addr = s.addr
open("http://#{addr[3]}:#{addr[1]}/", "rb") do |fp|
bm = Benchmark.measure do
while fp.read(16384, buf)
end
end
puts bm
end
puts File.readlines("/proc/#$$/status").grep(/RssAnon/)[0]
Process.waitpid2(pid)

• lib/open-uri.rb: clear string yielded by Net::HTTPResponse#read_body

• Status changed from Open to Closed

Applied in changeset trunk|r61664.

---

open-uri: clear string after buffering

Since r58846 (in Ruby 2.5), it is safe to clear the string
yielded to Net::HTTPResponse#read_body methods. This
reduces malloc garbage (anonymous RSS) using the Linux-only
script below:

before: user system total real
0.030000 0.250000 0.280000 ( 0.280511)
RssAnon: 60240 kB

after: user system total real
0.050000 0.223333 0.273333 ( 0.273118)
RssAnon: 6676 kB

---

warning this script requires 1G free space for buffering

require 'open-uri'
require 'socket'
require 'benchmark'

s = TCPServer.new('127.0.0.1', 0)
len = 1024 * 1024 * 1024
buf = ((0..255).map(&:chr).join * 128)
nr = len / buf.size
pid = fork do
c = s.accept
c.readpartial(16384).clear
c.write("HTTP/1.1 200 OK\r\n"
"Content-Length: #{len}\r\n"
"Content-Type: application/octet-stream\r\n"
"\r\n")
buf.freeze # speeds up IO#write slightly
nr.times { c.write(buf) }
c.close
end

addr = s.addr
open("http://#{addr[3]}:#{addr[1]}/", "rb") do |fp|
bm = Benchmark.measure do
while fp.read(16384, buf)
end
end
puts bm
end
puts File.readlines("/proc/#$$/status").grep(/RssAnon/)[0]
Process.waitpid2(pid)

• lib/open-uri.rb: clear string yielded by Net::HTTPResponse#read_body
  [ruby-core:84662] [Feature #14320]

[nobu (Nobuyoshi Nakada)]

• Related to Bug #15873: FrozenError when using OpenURI.open added