Project

General

Profile

Actions

Feature #22604

open

Add `lstat: true` to `Find.find` to yield the `File::Stat` it already computes

Feature #22604: Add `lstat: true` to `Find.find` to yield the `File::Stat` it already computes

Added by dug (Doug E) about 20 hours ago. Updated about 20 hours ago.

Status:
Open
Assignee:
-
Target version:
-
[ruby-core:127602]

Description

Abstract

Find.find already calls File.lstat on every entry to decide whether to descend into it, but discards the result, so a block that needs the entry's size, inode or type has to call File.lstat again. I propose an lstat: true keyword for Find.find and Pathname#find that yields that File::Stat along with each path, so these walks make one lstat(2) call per entry instead of two.

Background

When the block needs the stat, every entry costs two lstat calls (attached as repro_find_lstat.rb):

require "find"
require "tmpdir"

calls = 0
File.singleton_class.prepend(Module.new { define_method(:lstat) { |path| calls += 1; super(path) } })

Dir.mktmpdir do |dir|
  3.times { |i| File.write(File.join(dir, "file#{i}"), "x" * i) }

  entries = size = 0
  Find.find(dir) do |path|
    entries += 1
    size += File.lstat(path).size
  end
  puts "#{entries} entries, #{calls} File.lstat calls"
end
$ ruby -v
ruby 4.0.9 (2026-10-23 revision 3e93b23afc) +PRISM [arm64-darwin20]
$ ruby repro_find_lstat.rb
4 entries, 8 File.lstat calls

lib/find.rb from master gives the same result. I expected 4 calls, one per entry, because the block's call repeats the one Find.find has just made.

Code that walks trees this way:

  • Homebrew sized installed packages with Pathname#find, calling lstat on each entry for its size and [dev, ino]. To avoid the second call, https://github.com/Homebrew/brew/pull/24629 replaced Pathname#find with its own copy of the traversal, which a maintainer noted in review amounts to reimplementing Ruby internals.
  • fpm computes a Debian package's Installed-Size the same way (deb.rb):
    Find.find(staging_path) do |path|
      stat = File.lstat(path)
      next if stat.directory?
      total += stat.size
    end
    
    It also selects files for md5sums with File.file?(path) && !File.symlink?(path) (deb.rb), up to two more calls per entry for what stat.file? on an lstat result already tells.
  • The example in #13896 adds up file sizes inside Find.find with FileTest.directory? and FileTest.size.

Existing ways to make one call per entry:

  • Write the traversal yourself with Dir.children and File.lstat, as Homebrew did. This duplicates Find.find, including its list of ignored errors.
  • Call Dir.glob("**/*", File::FNM_DOTMATCH, base: dir), then File.lstat on each result. It is nearly as fast (see Evaluation), but it skips dotfiles unless given File::FNM_DOTMATCH, returns the root as ".", cannot prune, and has no way to raise on errors.
  • Dir.scan (#21800, new in Ruby 4.1) yields each entry's type without an lstat, which does not help when the caller needs the size or inode.
  • FileUtils::Entry_ keeps each entry's lstat while it traverses, but it is internal.

I found no earlier proposal for this. #7596 added the existing ignore_error: keyword to Find.find.

Proposal

Find.find(*paths, ignore_error: true, lstat: false) {|path| }  # unchanged
Find.find(*paths, lstat: true) {|path, stat| }                  # stat is File.lstat(path)
Pathname#find(ignore_error: true, lstat: false)                 # same, with a Pathname
  • lstat: defaults to false. Without it, nothing changes.
  • With lstat: true, the block receives each path and File.lstat(path), or nil when that call raised an error that ignore_error: true rescues. It yields the same paths as without the keyword.
  • Without a block, it returns an Enumerator of [path, stat] pairs.
  • Find.prune works as before.
  • The keyword is named after the call that produces the stat, which also says that it does not follow symlinks.

Implementation

The change to Find.find:

-  def find(*paths, ignore_error: true) # :yield: path
-    block_given? or return enum_for(__method__, *paths, ignore_error: ignore_error)
+  def find(*paths, ignore_error: true, lstat: false) # :yield: path
+    block_given? or return enum_for(__method__, *paths, ignore_error: ignore_error, lstat: lstat)
@@
         catch(:prune) do
-          yield file.dup
+          yield file.dup unless lstat
           begin
             s = File.lstat(file)
           rescue Errno::ENOENT, Errno::EACCES, Errno::ENOTDIR, Errno::ELOOP, Errno::ENAMETOOLONG, Errno::EINVAL
             raise unless ignore_error
+            yield file.dup, nil if lstat
             next
           end
+          yield file.dup, s if lstat
           if s.directory? then

The attached find_lstat.patch also passes the keyword through Pathname#find, documents it for both methods, and adds tests to test/test_find.rb and test/pathname/test_pathname.rb. With the patch applied, both test files pass on Ruby 4.0.9.

Evaluation

With lstat: true, the script above makes 4 File.lstat calls instead of 8.

The table shows the time to walk two Homebrew packages and the whole Homebrew Cellar, adding up sizes and counting each hard-linked file once. It was measured on macOS (arm64) with Ruby 4.0.9 (bench_ticket.rb), as the median of 31 runs per package and 7 for the Cellar, in shuffled order. All four walks produce the same totals.

Walk ruby (22,080 entries) gradle@8 (27,296) Cellar (236,554)
Find.find, then File.lstat in the block 606 ms 837 ms 7,579 ms
Find.find(lstat: true) 424 ms 657 ms 4,981 ms
Dir.glob, then File.lstat 441 ms 686 ms 5,213 ms
Homebrew's own traversal 439 ms 627 ms 4,860 ms

lstat: true is 1.3–1.5x faster than calling File.lstat in the block, and within 5% of Homebrew's traversal. It saves one lstat(2) call per entry on any platform; how much time that saves depends on the cost of lstat, which I have measured only on macOS.

Discussion

  • Dir.glob comes within 5% of lstat: true, so the case is less about speed than about Find.find, the API meant for walking a tree, making callers that need the stat pay for twice as many lstat calls as a glob or a hand-written loop.
  • With lstat: true, the stat is taken before the block runs instead of after, so Find.find decides whether to descend based on the entry as it was before the block ran. With ignore_error: false, an error from lstat is raised before the entry is yielded rather than after.
  • This follows the points raised about #21800. The stat comes from an lstat made when the entry is visited, not a lazily filled object, so it is clear when it was taken. It is lstat rather than stat because that is what Find.find already uses. A keyword avoids checking the block's arity.
  • Alternatives: a separate method such as Find.find_with_lstat; or skipping entries whose lstat fails instead of yielding nil, which would yield fewer paths than Find.find does today.

Summary

Find.find already makes one lstat call per entry, and lstat: true lets the block use the result instead of making another. It is a few lines in lib/find.rb, changes nothing for existing callers, and in Homebrew's case is within 5% of the hand-written traversal it would replace.


Files

bench_ticket.rb (2.76 KB) bench_ticket.rb dug (Doug E), 11/18/2026 11:00 PM
find_lstat.patch (6.06 KB) find_lstat.patch dug (Doug E), 11/18/2026 11:00 PM
repro_find_lstat.rb (399 Bytes) repro_find_lstat.rb dug (Doug E), 11/18/2026 11:00 PM
Actions

Also available in: PDF Atom