Feature #22604
openAdd `lstat: true` to `Find.find` to yield the `File::Stat` it already computes
Description
Abstract¶
Find.find already calls File.lstat on every entry to decide whether to descend into it, but discards the result, so a block that needs the entry's size, inode or type has to call File.lstat again. I propose an lstat: true keyword for Find.find and Pathname#find that yields that File::Stat along with each path, so these walks make one lstat(2) call per entry instead of two.
Background¶
When the block needs the stat, every entry costs two lstat calls (attached as repro_find_lstat.rb):
require "find"
require "tmpdir"
calls = 0
File.singleton_class.prepend(Module.new { define_method(:lstat) { |path| calls += 1; super(path) } })
Dir.mktmpdir do |dir|
3.times { |i| File.write(File.join(dir, "file#{i}"), "x" * i) }
entries = size = 0
Find.find(dir) do |path|
entries += 1
size += File.lstat(path).size
end
puts "#{entries} entries, #{calls} File.lstat calls"
end
$ ruby -v
ruby 4.0.9 (2026-10-23 revision 3e93b23afc) +PRISM [arm64-darwin20]
$ ruby repro_find_lstat.rb
4 entries, 8 File.lstat calls
lib/find.rb from master gives the same result. I expected 4 calls, one per entry, because the block's call repeats the one Find.find has just made.
Code that walks trees this way:
- Homebrew sized installed packages with
Pathname#find, callinglstaton each entry for its size and[dev, ino]. To avoid the second call, https://github.com/Homebrew/brew/pull/24629 replacedPathname#findwith its own copy of the traversal, which a maintainer noted in review amounts to reimplementing Ruby internals. - fpm computes a Debian package's Installed-Size the same way (deb.rb):
It also selects files for
Find.find(staging_path) do |path| stat = File.lstat(path) next if stat.directory? total += stat.size endmd5sumswithFile.file?(path) && !File.symlink?(path)(deb.rb), up to two more calls per entry for whatstat.file?on anlstatresult already tells. - The example in #13896 adds up file sizes inside
Find.findwithFileTest.directory?andFileTest.size.
Existing ways to make one call per entry:
- Write the traversal yourself with
Dir.childrenandFile.lstat, as Homebrew did. This duplicatesFind.find, including its list of ignored errors. - Call
Dir.glob("**/*", File::FNM_DOTMATCH, base: dir), thenFile.lstaton each result. It is nearly as fast (see Evaluation), but it skips dotfiles unless givenFile::FNM_DOTMATCH, returns the root as".", cannot prune, and has no way to raise on errors. Dir.scan(#21800, new in Ruby 4.1) yields each entry's type without anlstat, which does not help when the caller needs the size or inode.FileUtils::Entry_keeps each entry'slstatwhile it traverses, but it is internal.
I found no earlier proposal for this. #7596 added the existing ignore_error: keyword to Find.find.
Proposal¶
Find.find(*paths, ignore_error: true, lstat: false) {|path| } # unchanged
Find.find(*paths, lstat: true) {|path, stat| } # stat is File.lstat(path)
Pathname#find(ignore_error: true, lstat: false) # same, with a Pathname
lstat:defaults tofalse. Without it, nothing changes.- With
lstat: true, the block receives each path andFile.lstat(path), ornilwhen that call raised an error thatignore_error: truerescues. It yields the same paths as without the keyword. - Without a block, it returns an Enumerator of
[path, stat]pairs. Find.pruneworks as before.- The keyword is named after the call that produces the stat, which also says that it does not follow symlinks.
Implementation¶
The change to Find.find:
- def find(*paths, ignore_error: true) # :yield: path
- block_given? or return enum_for(__method__, *paths, ignore_error: ignore_error)
+ def find(*paths, ignore_error: true, lstat: false) # :yield: path
+ block_given? or return enum_for(__method__, *paths, ignore_error: ignore_error, lstat: lstat)
@@
catch(:prune) do
- yield file.dup
+ yield file.dup unless lstat
begin
s = File.lstat(file)
rescue Errno::ENOENT, Errno::EACCES, Errno::ENOTDIR, Errno::ELOOP, Errno::ENAMETOOLONG, Errno::EINVAL
raise unless ignore_error
+ yield file.dup, nil if lstat
next
end
+ yield file.dup, s if lstat
if s.directory? then
The attached find_lstat.patch also passes the keyword through Pathname#find, documents it for both methods, and adds tests to test/test_find.rb and test/pathname/test_pathname.rb. With the patch applied, both test files pass on Ruby 4.0.9.
Evaluation¶
With lstat: true, the script above makes 4 File.lstat calls instead of 8.
The table shows the time to walk two Homebrew packages and the whole Homebrew Cellar, adding up sizes and counting each hard-linked file once. It was measured on macOS (arm64) with Ruby 4.0.9 (bench_ticket.rb), as the median of 31 runs per package and 7 for the Cellar, in shuffled order. All four walks produce the same totals.
| Walk | ruby (22,080 entries) |
gradle@8 (27,296) |
Cellar (236,554) |
|---|---|---|---|
Find.find, then File.lstat in the block |
606 ms | 837 ms | 7,579 ms |
Find.find(lstat: true) |
424 ms | 657 ms | 4,981 ms |
Dir.glob, then File.lstat |
441 ms | 686 ms | 5,213 ms |
| Homebrew's own traversal | 439 ms | 627 ms | 4,860 ms |
lstat: true is 1.3–1.5x faster than calling File.lstat in the block, and within 5% of Homebrew's traversal. It saves one lstat(2) call per entry on any platform; how much time that saves depends on the cost of lstat, which I have measured only on macOS.
Discussion¶
Dir.globcomes within 5% oflstat: true, so the case is less about speed than aboutFind.find, the API meant for walking a tree, making callers that need the stat pay for twice as manylstatcalls as a glob or a hand-written loop.- With
lstat: true, the stat is taken before the block runs instead of after, soFind.finddecides whether to descend based on the entry as it was before the block ran. Withignore_error: false, an error fromlstatis raised before the entry is yielded rather than after. - This follows the points raised about #21800. The stat comes from an
lstatmade when the entry is visited, not a lazily filled object, so it is clear when it was taken. It islstatrather thanstatbecause that is whatFind.findalready uses. A keyword avoids checking the block's arity. - Alternatives: a separate method such as
Find.find_with_lstat; or skipping entries whoselstatfails instead of yieldingnil, which would yield fewer paths thanFind.finddoes today.
Summary¶
Find.find already makes one lstat call per entry, and lstat: true lets the block use the result instead of making another. It is a few lines in lib/find.rb, changes nothing for existing callers, and in Homebrew's case is within 5% of the hand-written traversal it would replace.
Files