Bug #19621
closed
Resolv::Hosts uses ineffective File.read, making using big hosts file 'impossible'
Description
By default on unixoid systems, Resolv will read /etc/hosts once. Privacy- and security aware people might use the file to prevent unwanted traffic, developers use it to quickly manipulate address resolution.
Resolv::Hosts uses IO.read, which seems to be inefficient when dealing with large amounts of data that should be consumed by line.
E.g. if you install the /etc/hosts additions by hblock (https://github.com/hectorm/hblock), the first call to resolve an address will likely take minutes.
Unfortunately, replacing .open ... .each with IO.foreach does not help.
Benchmarking with partial examplary /etc/hosts from above (172751 line) with this
require 'resolv'
require 'benchmark'
Benchmark.measure do
Resolv::Hosts.new.lazy_initialize
end
yields to
25.622515 8.821095 34.443610 ( 34.495448)
.
Reading in all the lines into memory first and then consuming them (File.readlines) might improve the situation, but is probably not desirable due to memory concerns.
Updated by 
Updated by