Feature #6845: Optimize stat on Windows

• Status: Closed
• Assignee: h.shirosaki (Hiroshi Shirosaki)
• Target version: 2.0.0

[h.shirosaki (Hiroshi Shirosaki)]

require slowness on startup was pointed out and some improvements were done, but startup seems still slow especially on Windows.
I suggest a patch to optimize require(stat) performance on Windows.

https://gist.github.com/3285446#file_1_winstat.patch

This patch was originally created by Dušan D. Majkić. I reviewed it and added some fixes.

Rationale:

When using require, stat() is called in realpath_rec(). Currently FindFirstFile() is used for stat() implementation on Windows.
According to our benchmark, GetFileAttributesEx() is the fastest way to get file attributes.

We changed stat implementation to use GetFileAttributesEx() and fallback to FindFirstFile().
GetFileAttributesEx() has possibility to get ERROR_SHARING_VIOLATION. For example against 'C:\pagefile.sys'.[1]
In such cases, we use FindFirstFile() to get file info. I added a test for this.

check_valid_dir() in winnt_stat() was introduced due to the following issue.
http://bugs.ruby-lang.org/issues/1005

This function uses FindFirstFile() after GetFileAttributes() to check if the path is valid directory or not.
This is for checking paths like "...". So I limited FindFirstFile() call only if the path contains "..." for performance.

Benchmark:

I can see small performance improvement with this patch. Here are benchmark results on Windows 7.
I used measurements [2] and empty rails app.

https://gist.github.com/3285446#file_2_bench.md
https://gist.github.com/3285446#gistcomment-393221 (chart)

Notable improvement is this.

core_require_nested 3.41s to 2.96s (-0.45s) (13% faster)
(require of deeply nested path)

One behavior difference:

This patch changes stat result of root path. Trunk ruby doesn't get root path's time information. Times are zero.
But patched ruby can get times of root path as expected. It's because FindFirstFile() cannot get root attributes.

trunk

C:\Users\hiroshi\work>ruby -e "p File.stat('/')"
#<File::Stat dev=0x2, ino=0, mode=040755, nlink=1, uid=0, gid=0, rdev=0x2, size=0, blksize=nil, bloc
ks=nil, atime=1970-01-01 09:00:00 +0900, mtime=1970-01-01 09:00:00 +0900, ctime=1970-01-01 09:00:00
+0900>

patched

C:\Users\hiroshi\work>ruby -e "p File.stat('/')"
#<File::Stat dev=0x2, ino=0, mode=040755, nlink=1, uid=0, gid=0, rdev=0x2, size=0, blksize=nil, bloc
ks=nil, atime=2012-08-07 21:55:09 +0900, mtime=2012-08-07 21:55:09 +0900, ctime=2009-07-14 11:38:56
+0900>

I confirmed make test and make test-all. It seems fine.

Also Luis Lavena, I and other people are testing TCS ruby 1.9.3 which includes Dušan's patch [3] and didn't see any issues while several months.

Thank you.

[1] http://hg.openjdk.java.net/icedtea/jdk7/jdk/rev/e2d9696aa701
[2] https://github.com/jonforums/measurements
[3] https://github.com/thecodeshop/ruby/commits/winstat/ruby_1_9_3

[usa (Usaku NAKAMURA)]

• Category set to core
• Status changed from Open to Assigned
• Assignee set to h.shirosaki (Hiroshi Shirosaki)
• Target version set to 2.0.0

seems good.
commit it, please.

• Status changed from Assigned to Closed
• % Done changed from 0 to 100

This issue was solved with changeset r36668.
Hiroshi, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

---

Optimize winnt_stat

• test/ruby/test_file_exhaustive.rb
  (TestFileExhaustive#test_stat_special_file): add a test.
  GetFileAttributesExW fails to get attributes of special files
  such as pagefile.sys.

• win32/win32.c (check_valid_dir): for performance, check the path
  by FindFirstFileW only if the path containts "..."

• win32/win32.c (winnt_stat): use GetFileAttributesExW instead of
  FindFirstFileW since GetFileAttributesExW is faster.
  Based on the patch by Dusan D. Majkic.
  [ruby-core:47083] [Feature #6845]