Feature #6845

Optimize stat on Windows

Added by Hiroshi Shirosaki over 2 years ago. Updated over 2 years ago.

[ruby-core:47083]
Status:Closed
Priority:Normal
Assignee:Hiroshi Shirosaki

Description

require slowness on startup was pointed out and some improvements were done, but startup seems still slow especially on Windows.
I suggest a patch to optimize require(stat) performance on Windows.

https://gist.github.com/3285446#file_1_winstat.patch

This patch was originally created by Dušan D. Majkić. I reviewed it and added some fixes.

Rationale:

When using require, stat() is called in realpath_rec(). Currently FindFirstFile() is used for stat() implementation on Windows.
According to our benchmark, GetFileAttributesEx() is the fastest way to get file attributes.

We changed stat implementation to use GetFileAttributesEx() and fallback to FindFirstFile().
GetFileAttributesEx() has possibility to get ERROR_SHARING_VIOLATION. For example against 'C:\pagefile.sys'.[1]
In such cases, we use FindFirstFile() to get file info. I added a test for this.

check_valid_dir() in winnt_stat() was introduced due to the following issue.
http://bugs.ruby-lang.org/issues/1005

This function uses FindFirstFile() after GetFileAttributes() to check if the path is valid directory or not.
This is for checking paths like "...". So I limited FindFirstFile() call only if the path contains "..." for performance.

Benchmark:

I can see small performance improvement with this patch. Here are benchmark results on Windows 7.
I used measurements [2] and empty rails app.

https://gist.github.com/3285446#file_2_bench.md
https://gist.github.com/3285446#gistcomment-393221 (chart)

Notable improvement is this.

core_require_nested 3.41s to 2.96s (-0.45s) (13% faster)
(require of deeply nested path)

One behavior difference:

This patch changes stat result of root path. Trunk ruby doesn't get root path's time information. Times are zero.
But patched ruby can get times of root path as expected. It's because FindFirstFile() cannot get root attributes.

trunk

C:\Users\hiroshi\work>ruby -e "p File.stat('/')"
#

patched

C:\Users\hiroshi\work>ruby -e "p File.stat('/')"
#

I confirmed make test and make test-all. It seems fine.

Also Luis Lavena, I and other people are testing TCS ruby 1.9.3 which includes Dušan's patch [3] and didn't see any issues while several months.

Thank you.

[1] http://hg.openjdk.java.net/icedtea/jdk7/jdk/rev/e2d9696aa701
[2] https://github.com/jonforums/measurements
[3] https://github.com/thecodeshop/ruby/commits/winstat/ruby_1_9_3

winnt_stat.patch Magnifier (3.7 KB) Hiroshi Shirosaki, 08/09/2012 07:46 AM


Related issues

Related to Backport193 - Backport #6846: Please backport Feature #6845 Closed 08/10/2012

Associated revisions

Revision 36668
Added by shirosaki over 2 years ago

Optimize winnt_stat

  • test/ruby/test_file_exhaustive.rb
    (TestFileExhaustive#test_stat_special_file): add a test.
    GetFileAttributesExW fails to get attributes of special files
    such as pagefile.sys.

  • win32/win32.c (check_valid_dir): for performance, check the path
    by FindFirstFileW only if the path containts "..."

  • win32/win32.c (winnt_stat): use GetFileAttributesExW instead of
    FindFirstFileW since GetFileAttributesExW is faster.
    Based on the patch by Dusan D. Majkic.
    [Feature #6845]

Revision 36668
Added by shirosaki over 2 years ago

Optimize winnt_stat

  • test/ruby/test_file_exhaustive.rb
    (TestFileExhaustive#test_stat_special_file): add a test.
    GetFileAttributesExW fails to get attributes of special files
    such as pagefile.sys.

  • win32/win32.c (check_valid_dir): for performance, check the path
    by FindFirstFileW only if the path containts "..."

  • win32/win32.c (winnt_stat): use GetFileAttributesExW instead of
    FindFirstFileW since GetFileAttributesExW is faster.
    Based on the patch by Dusan D. Majkic.
    [Feature #6845]

History

#1 Updated by Usaku NAKAMURA over 2 years ago

  • Category set to core
  • Status changed from Open to Assigned
  • Assignee set to Hiroshi Shirosaki
  • Target version set to 2.0.0

seems good.
commit it, please.

#2 Updated by Anonymous over 2 years ago

  • Status changed from Assigned to Closed
  • % Done changed from 0 to 100

This issue was solved with changeset r36668.
Hiroshi, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.


Optimize winnt_stat

  • test/ruby/test_file_exhaustive.rb
    (TestFileExhaustive#test_stat_special_file): add a test.
    GetFileAttributesExW fails to get attributes of special files
    such as pagefile.sys.

  • win32/win32.c (check_valid_dir): for performance, check the path
    by FindFirstFileW only if the path containts "..."

  • win32/win32.c (winnt_stat): use GetFileAttributesExW instead of
    FindFirstFileW since GetFileAttributesExW is faster.
    Based on the patch by Dusan D. Majkic.
    [Feature #6845]

Also available in: Atom PDF