Bug #19621: Resolv::Hosts uses ineffective File.read, making using big hosts file 'impossible'

• Status: Closed
• Assignee: -
• Target version: -
• Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN

[[e-mail] (Felix Wolfsteller)]

By default on unixoid systems, Resolv will read /etc/hosts once. Privacy- and security aware people might use the file to prevent unwanted traffic, developers use it to quickly manipulate address resolution.

Resolv::Hosts uses IO.read, which seems to be inefficient when dealing with large amounts of data that should be consumed by line.

E.g. if you install the /etc/hosts additions by hblock (https://github.com/hectorm/hblock), the first call to resolve an address will likely take minutes.

Unfortunately, replacing .open ... .each with IO.foreach does not help.

Benchmarking with partial examplary /etc/hosts from above (172751 line) with this

require 'resolv'
require 'benchmark'

Benchmark.measure do
  Resolv::Hosts.new.lazy_initialize
end

yields to

25.622515   8.821095  34.443610 ( 34.495448)

.

Reading in all the lines into memory first and then consuming them (File.readlines) might improve the situation, but is probably not desirable due to memory concerns.

[[e-mail] (Felix Wolfsteller)]

• Description updated (diff)

[[e-mail] (Felix Wolfsteller)]

• Description updated (diff)

[[e-mail] (Felix Wolfsteller)]

The reading does not seem to be the problem, but the hash operations.

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed