Project

General

Profile

Feature #14109

FileUtils: Use Dir.children instead of Dir.entries

Added by esparta (Espartaco Palma) over 1 year ago. Updated 5 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
[ruby-core:83777]

Description

Dir.children is available since Feature #11302. FileUtils uses
Dir.each on an internal method encapsulated on a private class
Entry_#entry, having no '.' neither '..' entries would make
now superfluous a chained reject filtering.

This change can improve the performance of these FileUtils
methods when the provided path covers thousands of files or
directories:

  • chmod_R
  • chown_R
  • remove_entry
  • remove_entry_secure
  • rm_r
  • remove_dir
  • copy_entry

Related: Feature #13896

Some profiling I did using 50,000 files on a given folder, using this code:

require_relative 'fileutils'

FileUtils.chmod_R 0777, 'benchmark'

Before the patch:

 28.69    15.05     15.05   200004     0.08     0.10  FileUtils::Entry_#join
 10.68    20.65      5.60   100005     0.06     0.26  FileUtils::Entry_#entries
  6.75    24.19      3.54   100004     0.04     0.11  FileUtils::Entry_#lstat
  5.68    27.17      2.98   150007     0.02     0.13  FileUtils::Entry_#path
  4.97    29.78      2.61    50002     0.05     0.44  FileUtils::Entry_#chmod
  4.97    29.78      2.61    50002     0.05     0.44  FileUtils::Entry_#chmod
  4.80    32.29      2.52    50004     0.05     2.73  FileUtils.chmod_R
  4.33    34.56      2.27        1  2268.74 52382.08  FileUtils::Entry_#preorder_traverse
  4.18    36.76      2.19   450012     0.00     0.00  String#==
  3.73    38.71      1.96   100004     0.02     0.13  FileUtils::Entry_#lstat!
  3.56    40.58      1.87   400008     0.00     0.00  BasicObject#!
  3.35    42.33      1.76    50002     0.04     0.08  FileUtils::Entry_#directory?
  3.27    44.05      1.72    50002     0.03     0.25  FileUtils::Entry_#symlink?
  2.14    45.17      1.12   150003     0.01     0.01  File.join
  2.01    46.23      1.05    50002     0.02     0.03  Class#new
  1.89    47.22      0.99    50002     0.02     0.02  FileUtils.fu_mode
  1.47    47.99      0.77        4   192.12  2755.69  Array#map
  1.40    48.72      0.73        1   733.87  2821.06  Array#reject
  0.92    49.20      0.48    50002     0.01     0.01  File.lstat
  0.85    49.65      0.44   100004     0.00     0.00  FileUtils::Entry_#dereference?
  0.82    50.07      0.43    50002     0.01     0.01  File.chmod
  0.57    50.37      0.30    50006     0.01     0.01  File.path
  0.56    50.67      0.29    50002     0.01     0.01  FileUtils::Entry_#initialize
  0.49    50.92      0.26    50002     0.01     0.01  File::Stat#directory?
  0.49    51.18      0.25    50002     0.01     0.01  File::Stat#symlink?
  0.47    51.43      0.25    50003     0.00     0.00  Array#pop
  0.47    51.67      0.24    50001     0.00     0.00  FileUtils::Entry_#rel
  0.46    51.91      0.24    50001     0.00     0.00  Kernel#untaint
  0.43    52.14      0.23    50002     0.00     0.00  Kernel#is_a?
  0.43    52.36      0.23    50001     0.00     0.00  FileUtils::Entry_#prefix
  0.04    52.38      0.02        1    18.88    18.98  Dir.entries
  0.03    52.40      0.01        1    13.28    59.69  Kernel#require_relative                                         
  0.02    52.41      0.01      320     0.04     0.09  nil#
  0.01    52.41      0.01       86     0.08     0.08  Module#module_eval
  0.01    52.42      0.01      180     0.03     0.10  FileUtils.collect_method
  0.01    52.42      0.00        4     0.67     2.51  Array#select
  0.01    52.42      0.00       44     0.06     0.15  Array#map!
  0.01    52.43      0.00        3     0.88     1.08  Module#public
  0.00    52.43      0.00        2     1.12     2.61  Kernel#require
  0.00    52.43      0.00        5     0.39 10481.37  Array#each
  0.00    52.43      0.00      262     0.01     0.01  Module#method_added
  0.00    52.43      0.00       61     0.02     0.03  Module#module_function
  0.00    52.44      0.00      226     0.00     0.00  BasicObject#singleton_method_added
  0.00    52.44      0.00      176     0.00     0.00  Hash#[]
  0.00    52.44      0.00      176     0.00     0.00  Array#include?
  0.00    52.44      0.00      168     0.00     0.00  Symbol#==
  0.00    52.44      0.00       17     0.03     0.06  FileUtils.private_module_function
  0.00    52.44      0.00       22     0.02     0.03  Module#alias_method
  0.00    52.44      0.00       86     0.00     0.00  Integer#+
  0.00    52.44      0.00        1     0.33     0.33  Array#reverse
  0.00    52.44      0.00       44     0.01     0.01  UnboundMethod#parameters
  0.00    52.44      0.00       44     0.01     0.01  Hash#[]=
  0.00    52.44      0.00        7     0.03     0.05  Module#include
  0.00    52.44      0.00       44     0.01     0.01  Module#instance_method
  0.00    52.44      0.00       44     0.00     0.00  Array#compact!
  0.00    52.44      0.00        1     0.19     0.19  Array#concat
  0.00    52.44      0.00       44     0.00     0.00  Symbol#to_s
  0.00    52.44      0.00        4     0.04     0.07  Kernel#extend
  0.00    52.44      0.00       28     0.00     0.00  String#intern
  0.00    52.44      0.00        1     0.10     0.10  Dir.open
  0.00    52.44      0.00       17     0.00     0.00  Module#private_class_method
  0.00    52.44      0.00        4     0.02     0.02  Module#extend_object                                            
  0.00    52.44      0.00        7     0.01     0.01  Module#append_features
  0.00    52.44      0.00        4     0.01     0.01  Hash#keys
  0.00    52.44      0.00        1     0.05     0.07  MonitorMixin#mon_enter
  0.00    52.44      0.00        2     0.03     0.06  FileUtils.fu_list
  0.00    52.44      0.00        7     0.01     0.01  Module#private
  0.00    52.44      0.00        2     0.03     0.03  Kernel#singleton_methods
  0.00    52.44      0.00        1     0.05     0.05  Numeric#zero?
  0.00    52.44      0.00        1     0.04    12.84  Enumerable#inject
  0.00    52.44      0.00        1     0.04     0.04  Module#private_instance_methods
  0.00    52.44      0.00        4     0.01     0.01  IO#set_encoding
  0.00    52.44      0.00        1     0.03     0.06  MonitorMixin#mon_exit
  0.00    52.44      0.00        7     0.00     0.00  Module#included
  0.00    52.44      0.00        2     0.01     0.01  Array#-
  0.00    52.44      0.00        1     0.03     0.07  Numeric#nonzero?
  0.00    52.44      0.00        1     0.02     0.02  Kernel#methods
  0.00    52.44      0.00        1     0.02     0.03  FileUtils::StreamUtils_#fu_windows?
  0.00    52.44      0.00        4     0.01     0.01  Module#extended
  0.00    52.44      0.00        1     0.02     0.02  MonitorMixin#mon_check_owner
  0.00    52.44      0.00        1     0.02     0.02  Array#&
  0.00    52.44      0.00        3     0.00     0.00  Class#inherited
  0.00    52.44      0.00        3     0.00     0.00  Thread.current
  0.00    52.44      0.00        1     0.01     0.01  Regexp#=~
  0.00    52.44      0.00        1     0.01     0.01  TracePoint#enable
  0.00    52.44      0.00        1     0.01     0.01  Gem::Specification.unresolved_deps
  0.00    52.44      0.00        1     0.01     0.01  Array#flatten
  0.00    52.44      0.00        1     0.01     0.01  Gem.find_unresolved_default_spec
  0.00    52.44      0.00        1     0.01     0.01  Kernel#respond_to?
  0.00    52.44      0.00        1     0.01     0.01  TracePoint#disable
  0.00    52.44      0.00        1     0.01     0.01  Thread::Mutex#unlock
  0.00    52.44      0.00        1     0.00     0.00  Thread::Mutex#lock
  0.00    52.44      0.00        1     0.00 52442.37  #toplevel

After the patch

 30.07    14.76     14.76   200004     0.07     0.10  FileUtils::Entry_#join
  7.87    18.62      3.86    50002     0.08     0.41  FileUtils::Entry_#entries
  7.20    22.15      3.53   100004     0.04     0.11  FileUtils::Entry_#lstat
  6.09    25.14      2.99   150007     0.02     0.12  FileUtils::Entry_#path
  5.34    27.77      2.62    50002     0.05     0.43  FileUtils::Entry_#chmod
  5.12    30.28      2.51    50004     0.05     2.59  FileUtils.chmod_R
  4.64    32.56      2.28        1  2279.40 49019.52  FileUtils::Entry_#preorder_traverse
  3.98    34.51      1.95   100004     0.02     0.13  FileUtils::Entry_#lstat!
  3.72    36.34      1.83   400008     0.00     0.00  BasicObject#!
  3.56    38.09      1.75    50002     0.03     0.08  FileUtils::Entry_#directory?
  3.54    39.82      1.74   350007     0.00     0.00  String#==
  3.47    41.53      1.70    50002     0.03     0.25  FileUtils::Entry_#symlink?
  2.29    42.65      1.13   150003     0.01     0.01  File.join
  2.05    43.66      1.01    50002     0.02     0.03  Class#new
  2.01    44.65      0.99    50002     0.02     0.02  FileUtils.fu_mode
  1.51    45.39      0.74        4   184.89  2664.27  Array#map
  1.01    45.88      0.50    50002     0.01     0.01  File.lstat
  0.90    46.32      0.44    50002     0.01     0.01  File.chmod
  0.88    46.75      0.43   100004     0.00     0.00  FileUtils::Entry_#dereference?
  0.60    47.05      0.29    50006     0.01     0.01  File.path
  0.57    47.33      0.28    50002     0.01     0.01  FileUtils::Entry_#initialize
  0.53    47.58      0.26    50002     0.01     0.01  File::Stat#symlink?
  0.53    47.84      0.26    50003     0.01     0.01  Array#pop
  0.51    48.09      0.25    50002     0.01     0.01  File::Stat#directory?
  0.49    48.33      0.24    50001     0.00     0.00  FileUtils::Entry_#rel
  0.48    48.57      0.24    50001     0.00     0.00  Kernel#untaint
  0.45    48.79      0.22    50002     0.00     0.00  Kernel#is_a?
  0.43    49.00      0.21    50001     0.00     0.00  FileUtils::Entry_#prefix
  0.04    49.02      0.02        1    18.47    18.49  Dir.children
  0.03    49.03      0.01        1    13.31    59.77  Kernel#require_relative
  0.02    49.04      0.01      320     0.03     0.08  nil#
  0.01    49.05      0.01       86     0.08     0.08  Module#module_eval
  0.01    49.06      0.01      180     0.03     0.10  FileUtils.collect_method
  0.01    49.06      0.00        2     1.29     3.07  Kernel#require
  0.01    49.06      0.00        4     0.64     2.52  Array#select
  0.01    49.06      0.00       44     0.06     0.14  Array#map!
  0.00    49.07      0.00        3     0.70     1.10  Module#public
  0.00    49.07      0.00        5     0.41  9808.82  Array#each
  0.00    49.07      0.00      226     0.01     0.01  BasicObject#singleton_method_added
  0.00    49.07      0.00      262     0.01     0.01  Module#method_added
  0.00    49.07      0.00       61     0.02     0.02  Module#module_function
  0.00    49.07      0.00       44     0.02     0.02  Module#instance_method
  0.00    49.07      0.00      176     0.00     0.00  Array#include?
  0.00    49.07      0.00      176     0.00     0.00  Hash#[]
  0.00    49.08      0.00      168     0.00     0.00  Symbol#==
  0.00    49.08      0.00       17     0.03     0.06  FileUtils.private_module_function
  0.00    49.08      0.00       22     0.02     0.03  Module#alias_method
  0.00    49.08      0.00       86     0.00     0.00  Integer#+
  0.00    49.08      0.00       44     0.01     0.01  UnboundMethod#parameters
  0.00    49.08      0.00        7     0.03     0.04  Module#include
  0.00    49.08      0.00       44     0.01     0.01  Hash#[]=
  0.00    49.08      0.00        1     0.20     0.20  Array#reverse
  0.00    49.08      0.00       44     0.00     0.00  Array#compact!
  0.00    49.08      0.00       44     0.00     0.00  Symbol#to_s
  0.00    49.08      0.00        1     0.16     0.16  Array#concat
  0.00    49.08      0.00        4     0.04     0.06  Kernel#extend
  0.00    49.08      0.00       28     0.00     0.00  String#intern
  0.00    49.08      0.00       17     0.00     0.00  Module#private_class_method
  0.00    49.08      0.00        2     0.03     0.03  Kernel#singleton_methods
  0.00    49.08      0.00        7     0.01     0.01  Module#private
  0.00    49.08      0.00        1     0.05     0.07  MonitorMixin#mon_enter
  0.00    49.08      0.00        2     0.03     0.05  FileUtils.fu_list
  0.00    49.08      0.00        7     0.01     0.01  Module#append_features
  0.00    49.08      0.00        4     0.01     0.01  Module#extend_object
  0.00    49.08      0.00        1     0.04     0.04  Module#private_instance_methods
  0.00    49.08      0.00        4     0.01     0.01  IO#set_encoding
  0.00    49.08      0.00        7     0.00     0.00  Module#included
  0.00    49.08      0.00        1     0.03     0.06  MonitorMixin#mon_exit
  0.00    49.08      0.00        2     0.01     0.01  Array#-
  0.00    49.08      0.00        1     0.03     0.03  Kernel#methods
  0.00    49.08      0.00        1     0.02     0.03  Numeric#nonzero?
  0.00    49.08      0.00        1     0.02     0.03  FileUtils::StreamUtils_#fu_windows?
  0.00    49.08      0.00        1     0.02    12.57  Enumerable#inject
  0.00    49.08      0.00        1     0.02     0.02  Dir.open
  0.00    49.08      0.00        4     0.00     0.00  Module#extended
  0.00    49.08      0.00        1     0.02     0.02  MonitorMixin#mon_check_owner
  0.00    49.08      0.00        4     0.00     0.00  Hash#keys
  0.00    49.08      0.00        1     0.02     0.02  Array#&
  0.00    49.08      0.00        3     0.01     0.01  Class#inherited
  0.00    49.08      0.00        3     0.00     0.00  Thread.current
  0.00    49.08      0.00        1     0.01     0.01  TracePoint#enable
  0.00    49.08      0.00        1     0.01     0.01  Regexp#=~
  0.00    49.08      0.00        1     0.01     0.01  TracePoint#disable
  0.00    49.08      0.00        1     0.01     0.01  Gem.find_unresolved_default_spec
  0.00    49.08      0.00        1     0.01     0.01  Array#flatten
  0.00    49.08      0.00        1     0.01     0.01  Gem::Specification.unresolved_deps
  0.00    49.08      0.00        1     0.01     0.01  Numeric#zero?
  0.00    49.08      0.00        1     0.01     0.01  Thread::Mutex#lock
  0.00    49.08      0.00        1     0.01     0.01  Thread::Mutex#unlock
  0.00    49.08      0.00        1     0.01     0.01  Kernel#respond_to?
  0.00    49.08      0.00        1     0.00 49079.91  #toplevel

PR at GitHub: https://github.com/ruby/ruby/pull/1754

Associated revisions

Revision 37c08fad
Added by hsbt (Hiroshi SHIBATA) 5 months ago

Dir.children is available since Feature #11302. FileUtils uses
Dir.each on an internal method encapsulated on a private class
Entry_#entry, having no '.' neither '..' entries would make
now superfluous a chained reject filtering.

This change can improve the performance of these FileUtils
methods when the provided path covers thousands of files or
directories:

  • chmod_R
  • chown_R
  • remove_entry
  • remove_entry_secure
  • rm_r
  • remove_dir
  • copy_entry

Related: Feature #13896 https://bugs.ruby-lang.org/issues/13896

[Feature #14109][Fix GH-1754]

Co-Authored-By: esparta esparta@gmail.com

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65610 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Revision 65610
Added by hsbt (Hiroshi SHIBATA) 5 months ago

Dir.children is available since Feature #11302. FileUtils uses
Dir.each on an internal method encapsulated on a private class
Entry_#entry, having no '.' neither '..' entries would make
now superfluous a chained reject filtering.

This change can improve the performance of these FileUtils
methods when the provided path covers thousands of files or
directories:

  • chmod_R
  • chown_R
  • remove_entry
  • remove_entry_secure
  • rm_r
  • remove_dir
  • copy_entry

Related: Feature #13896 https://bugs.ruby-lang.org/issues/13896

[Feature #14109][Fix GH-1754]

Co-Authored-By: esparta esparta@gmail.com

Revision 65610
Added by hsbt (Hiroshi SHIBATA) 5 months ago

Dir.children is available since Feature #11302. FileUtils uses
Dir.each on an internal method encapsulated on a private class
Entry_#entry, having no '.' neither '..' entries would make
now superfluous a chained reject filtering.

This change can improve the performance of these FileUtils
methods when the provided path covers thousands of files or
directories:

  • chmod_R
  • chown_R
  • remove_entry
  • remove_entry_secure
  • rm_r
  • remove_dir
  • copy_entry

Related: Feature #13896 https://bugs.ruby-lang.org/issues/13896

[Feature #14109][Fix GH-1754]

Co-Authored-By: esparta esparta@gmail.com

History

#1

Updated by esparta (Espartaco Palma) over 1 year ago

  • Description updated (diff)
#2

Updated by hsbt (Hiroshi SHIBATA) 5 months ago

  • Status changed from Open to Closed

Applied in changeset trunk|r65610.


Dir.children is available since Feature #11302. FileUtils uses
Dir.each on an internal method encapsulated on a private class
Entry_#entry, having no '.' neither '..' entries would make
now superfluous a chained reject filtering.

This change can improve the performance of these FileUtils
methods when the provided path covers thousands of files or
directories:

  • chmod_R
  • chown_R
  • remove_entry
  • remove_entry_secure
  • rm_r
  • remove_dir
  • copy_entry

Related: Feature #13896 https://bugs.ruby-lang.org/issues/13896

[Feature #14109][Fix GH-1754]

Co-Authored-By: esparta esparta@gmail.com

Also available in: Atom PDF