Project

General

Profile

Feature #5999

Optimize for faster loading time and win32/file.c refactoring

Added by h.shirosaki (Hiroshi Shirosaki) about 6 years ago. Updated about 6 years ago.

Status:
Closed
Priority:
Normal
Target version:
[ruby-core:42480]

Description

This was originally discussed at Best way to separate implementation specific code?.

Ruby startup time (loading time) is slow when loading a lot of libraries such as Rails. Some works were done to improve performance such as #3924, but startup is still not fast. Especially ruby on Windows seems much slower than on Unix like OS.
This work is an effort to improve startup time on Windows.

I have updated the patch with help from Luis Lavena. This patch improves loading time a little.

https://gist.github.com/1605263#file_0001_move_rb_file_load_ok_to_win32_file.c.patch

If load_path and loaded_features are large numbers such as rails app, file_load_ok() is called many times against non-existing path.

On Windows, GetFileAttributes() is faster than open(). And open() performance is affected by AntiVirus software's realtime scan.
I confirmed this with NOD32. Microsoft Security Essentials doesn't affect so badly as NOD32. It seems to depend on kinds of antivirus softwares.

Benchmark:
https://gist.github.com/1605263#gistcomment-75650
https://gist.github.com/1448652

Avoiding to call open() against non-existing path by using GetFileAttributes() seems to improve performance.
I used CreateFile() instead of open() because CreateFile() seems a little faster.

Here are rails startup benchmarks.

On Windows 7:

200ms improvement
https://gist.github.com/1605263#gistcomment-82137

On Windows XP:

800ms improvement with NOD32 OFF
1800ms improvement with NOD32 ON
https://gist.github.com/1605263#gistcomment-75644

This fix would be helpful for windows users. I guess most windows users use antivirus softwares.

I confirmed make, make test are OK. make test-all result was same as before patch at the following revision.
ruby 2.0.0dev (2012-02-09 trunk 34511) [i386-mingw32]

At , Matz agreed with replacing file_load_ok.

This is the first work in win32/file.c. The following works (optimize expand_path etc.) into win32/file.c will be done by Luis.
This patch changed configure and Makefile. I also confirmed make with nmake(mswin32).

Is this acceptable way?

Associated revisions

Revision 4b1e9f0c
Added by shirosaki about 6 years ago

  • Makefile.in (PLATFORM_DIR): add a variable for win32 directory.
  • Makefile.in (clean-platform): add new target.
    It cleans win32 directory.

  • common.mk (clean): add a dependency for win32 directory.

  • common.mk (distclean): ditto.

  • common.mk (distclean-platform): add new target.
    It cleans win32 directory.

  • common.mk ($(PLATFORM_D)): add new target to make win32 directory.

  • common.mk (win32/win32.$(OBJEXT)): move win32.o into win32
    directory.

  • common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

  • configure.in: move win32.o into win32 directory and add
    win32/file.o to MISSING.

  • file.c (file_load_ok, rb_file_load_ok): replace static
    file_load_ok() with public rb_file_load_ok().
    It's to link Windows implementation in win32/file.c.

  • file.c (rb_find_file_ext_safe): ditto.

  • file.c (rb_find_file_safe): ditto.

  • win32/file.c (rb_file_load_ok): new file. Add Windows specific
    optimized implementation of rb_file_load_ok(). We created a
    separated file to avoid too many #ifdef macro which is unreadable.

  • win32/Makefile.sub (PLATFORM_DIR): add a variable for win32
    directory.

  • win32/Makefile.sub (MISSING): move win32.obj into win32
    directory and add win32/file.obj to MISSING.

  • win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
    It's because miniruby doesn't exist when making win32 directory.

  • win32/Makefile.sub (clean-platform): add new target to clean win32
    directory.

  • win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
    win32/file.c to build properly.

  • win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
    win32 directory.

Patch created with Luis Lavena.
[Feature #5999]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@34849 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Revision 34849
Added by shirosaki about 6 years ago

  • Makefile.in (PLATFORM_DIR): add a variable for win32 directory.
  • Makefile.in (clean-platform): add new target.
    It cleans win32 directory.

  • common.mk (clean): add a dependency for win32 directory.

  • common.mk (distclean): ditto.

  • common.mk (distclean-platform): add new target.
    It cleans win32 directory.

  • common.mk ($(PLATFORM_D)): add new target to make win32 directory.

  • common.mk (win32/win32.$(OBJEXT)): move win32.o into win32
    directory.

  • common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

  • configure.in: move win32.o into win32 directory and add
    win32/file.o to MISSING.

  • file.c (file_load_ok, rb_file_load_ok): replace static
    file_load_ok() with public rb_file_load_ok().
    It's to link Windows implementation in win32/file.c.

  • file.c (rb_find_file_ext_safe): ditto.

  • file.c (rb_find_file_safe): ditto.

  • win32/file.c (rb_file_load_ok): new file. Add Windows specific
    optimized implementation of rb_file_load_ok(). We created a
    separated file to avoid too many #ifdef macro which is unreadable.

  • win32/Makefile.sub (PLATFORM_DIR): add a variable for win32
    directory.

  • win32/Makefile.sub (MISSING): move win32.obj into win32
    directory and add win32/file.obj to MISSING.

  • win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
    It's because miniruby doesn't exist when making win32 directory.

  • win32/Makefile.sub (clean-platform): add new target to clean win32
    directory.

  • win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
    win32/file.c to build properly.

  • win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
    win32 directory.

Patch created with Luis Lavena.
[Feature #5999]

Revision 34849
Added by shirosaki about 6 years ago

  • Makefile.in (PLATFORM_DIR): add a variable for win32 directory.
  • Makefile.in (clean-platform): add new target.
    It cleans win32 directory.

  • common.mk (clean): add a dependency for win32 directory.

  • common.mk (distclean): ditto.

  • common.mk (distclean-platform): add new target.
    It cleans win32 directory.

  • common.mk ($(PLATFORM_D)): add new target to make win32 directory.

  • common.mk (win32/win32.$(OBJEXT)): move win32.o into win32
    directory.

  • common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

  • configure.in: move win32.o into win32 directory and add
    win32/file.o to MISSING.

  • file.c (file_load_ok, rb_file_load_ok): replace static
    file_load_ok() with public rb_file_load_ok().
    It's to link Windows implementation in win32/file.c.

  • file.c (rb_find_file_ext_safe): ditto.

  • file.c (rb_find_file_safe): ditto.

  • win32/file.c (rb_file_load_ok): new file. Add Windows specific
    optimized implementation of rb_file_load_ok(). We created a
    separated file to avoid too many #ifdef macro which is unreadable.

  • win32/Makefile.sub (PLATFORM_DIR): add a variable for win32
    directory.

  • win32/Makefile.sub (MISSING): move win32.obj into win32
    directory and add win32/file.obj to MISSING.

  • win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
    It's because miniruby doesn't exist when making win32 directory.

  • win32/Makefile.sub (clean-platform): add new target to clean win32
    directory.

  • win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
    win32/file.c to build properly.

  • win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
    win32 directory.

Patch created with Luis Lavena.
[Feature #5999]

Revision 34849
Added by shirosaki about 6 years ago

  • Makefile.in (PLATFORM_DIR): add a variable for win32 directory.
  • Makefile.in (clean-platform): add new target.
    It cleans win32 directory.

  • common.mk (clean): add a dependency for win32 directory.

  • common.mk (distclean): ditto.

  • common.mk (distclean-platform): add new target.
    It cleans win32 directory.

  • common.mk ($(PLATFORM_D)): add new target to make win32 directory.

  • common.mk (win32/win32.$(OBJEXT)): move win32.o into win32
    directory.

  • common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

  • configure.in: move win32.o into win32 directory and add
    win32/file.o to MISSING.

  • file.c (file_load_ok, rb_file_load_ok): replace static
    file_load_ok() with public rb_file_load_ok().
    It's to link Windows implementation in win32/file.c.

  • file.c (rb_find_file_ext_safe): ditto.

  • file.c (rb_find_file_safe): ditto.

  • win32/file.c (rb_file_load_ok): new file. Add Windows specific
    optimized implementation of rb_file_load_ok(). We created a
    separated file to avoid too many #ifdef macro which is unreadable.

  • win32/Makefile.sub (PLATFORM_DIR): add a variable for win32
    directory.

  • win32/Makefile.sub (MISSING): move win32.obj into win32
    directory and add win32/file.obj to MISSING.

  • win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
    It's because miniruby doesn't exist when making win32 directory.

  • win32/Makefile.sub (clean-platform): add new target to clean win32
    directory.

  • win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
    win32/file.c to build properly.

  • win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
    win32 directory.

Patch created with Luis Lavena.
[Feature #5999]

Revision 34849
Added by shirosaki about 6 years ago

  • Makefile.in (PLATFORM_DIR): add a variable for win32 directory.
  • Makefile.in (clean-platform): add new target.
    It cleans win32 directory.

  • common.mk (clean): add a dependency for win32 directory.

  • common.mk (distclean): ditto.

  • common.mk (distclean-platform): add new target.
    It cleans win32 directory.

  • common.mk ($(PLATFORM_D)): add new target to make win32 directory.

  • common.mk (win32/win32.$(OBJEXT)): move win32.o into win32
    directory.

  • common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

  • configure.in: move win32.o into win32 directory and add
    win32/file.o to MISSING.

  • file.c (file_load_ok, rb_file_load_ok): replace static
    file_load_ok() with public rb_file_load_ok().
    It's to link Windows implementation in win32/file.c.

  • file.c (rb_find_file_ext_safe): ditto.

  • file.c (rb_find_file_safe): ditto.

  • win32/file.c (rb_file_load_ok): new file. Add Windows specific
    optimized implementation of rb_file_load_ok(). We created a
    separated file to avoid too many #ifdef macro which is unreadable.

  • win32/Makefile.sub (PLATFORM_DIR): add a variable for win32
    directory.

  • win32/Makefile.sub (MISSING): move win32.obj into win32
    directory and add win32/file.obj to MISSING.

  • win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
    It's because miniruby doesn't exist when making win32 directory.

  • win32/Makefile.sub (clean-platform): add new target to clean win32
    directory.

  • win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
    win32/file.c to build properly.

  • win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
    win32 directory.

Patch created with Luis Lavena.
[Feature #5999]

Revision 34849
Added by shirosaki about 6 years ago

  • Makefile.in (PLATFORM_DIR): add a variable for win32 directory.
  • Makefile.in (clean-platform): add new target.
    It cleans win32 directory.

  • common.mk (clean): add a dependency for win32 directory.

  • common.mk (distclean): ditto.

  • common.mk (distclean-platform): add new target.
    It cleans win32 directory.

  • common.mk ($(PLATFORM_D)): add new target to make win32 directory.

  • common.mk (win32/win32.$(OBJEXT)): move win32.o into win32
    directory.

  • common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

  • configure.in: move win32.o into win32 directory and add
    win32/file.o to MISSING.

  • file.c (file_load_ok, rb_file_load_ok): replace static
    file_load_ok() with public rb_file_load_ok().
    It's to link Windows implementation in win32/file.c.

  • file.c (rb_find_file_ext_safe): ditto.

  • file.c (rb_find_file_safe): ditto.

  • win32/file.c (rb_file_load_ok): new file. Add Windows specific
    optimized implementation of rb_file_load_ok(). We created a
    separated file to avoid too many #ifdef macro which is unreadable.

  • win32/Makefile.sub (PLATFORM_DIR): add a variable for win32
    directory.

  • win32/Makefile.sub (MISSING): move win32.obj into win32
    directory and add win32/file.obj to MISSING.

  • win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
    It's because miniruby doesn't exist when making win32 directory.

  • win32/Makefile.sub (clean-platform): add new target to clean win32
    directory.

  • win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
    win32/file.c to build properly.

  • win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
    win32 directory.

Patch created with Luis Lavena.
[Feature #5999]

Revision 2c400078
Added by usa (Usaku NAKAMURA) over 5 years ago

merge revision(s) 34849,34853,34854,34855,34859,34862,35384,35385,36811,36812,36850,36907,36908: [Backport #7174]

* Makefile.in (PLATFORM_DIR): add a variable for `win32` directory.

* Makefile.in (clean-platform): add new target.
  It cleans `win32` directory.

* common.mk (clean): add a dependency for `win32` directory.

* common.mk (distclean): ditto.

* common.mk (distclean-platform): add new target.
  It cleans `win32` directory.

* common.mk ($(PLATFORM_D)): add new target to make `win32` directory.

* common.mk (win32/win32.$(OBJEXT)): move win32.o into `win32`
  directory.

* common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

* configure.in: move win32.o into `win32` directory and add
  win32/file.o to MISSING.

* file.c (file_load_ok, rb_file_load_ok): replace static
  file_load_ok() with public rb_file_load_ok().
  It's to link Windows implementation in win32/file.c.

* file.c (rb_find_file_ext_safe): ditto.

* file.c (rb_find_file_safe): ditto.

* win32/file.c (rb_file_load_ok): new file. Add Windows specific
  optimized implementation of rb_file_load_ok(). We created a
  separated file to avoid too many #ifdef macro which is unreadable.

* win32/Makefile.sub (PLATFORM_DIR): add a variable for `win32`
  directory.

* win32/Makefile.sub (MISSING): move win32.obj into `win32`
  directory and add win32/file.obj to MISSING.

* win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
  It's because miniruby doesn't exist when making `win32` directory.

* win32/Makefile.sub (clean-platform): add new target to clean `win32`
  directory.

* win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
  win32/file.c to build properly.

* win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
 `win32` directory.
  Patch created with Luis Lavena.
   [Feature #5999]

* win32/Makefile.sub (MAKEDIRS): use mkdir of cmd.exe instead of ruby.
  [Bug #6103] 

* win32/README.win32: added a notice about command extension of cmd.exe.

* win32/makedirs.bat: new command to make intermediate
  directories, and not to report any errors if the directory
  already exists.

* win32/Makefile.sub (MAKEDIRS): enable command extensions.

* win32/file.c (INVALID_FILE_ATTRIBUTES): define for old SDK.

* configure.in (mingw): add shlwapi to the list of dependency
  libs for Windows.

* win32/Makefile.sub (EXTSOLIBS): ditto.

* internal.h: declare internal functions rb_w32_init_file,
  rb_file_expand_path_internal and rb_file_expand_path_fast.

* file.c (Init_File): invoke Windows initialization rb_w32_init_file

* win32/file.c (rb_file_load_path_internal): new function.
  Windows-specific implementation that replaces file_expand_path.
  [Bug #6836]

* win32/file.c (rb_w32_init_file): new function. Initialize codepage
  cache for faster conversion encodings lookup.

* file.c (file_expand_path): rename to rb_file_expand_path_internal.
  Conditionally exclude from Windows.

* file.c (rb_file_expand_path_fast): new function. delegates to
  rb_file_expand_path_internal without performing a hit to the
  filesystem.

* file.c (file_expand_path_1): use rb_file_expand_path_internal without
  path expansion (used by require).

* file.c (rb_find_file_ext_safe): ditto.

* file.c (rb_find_file_safe): ditto.

* load.c (rb_get_expanded_load_path): use rb_file_expand_path_fast.

* load.c (rb_feature_provided): ditto.

* file.c (rb_file_expand_path): use rb_file_expand_path_internal with
  path expansion.

* file.c (rb_file_absolute_path): ditto.

* test/ruby/test_file_exhaustive.rb: new tests to exercise
  rb_file_expand_path_internal implementation and compliance with
  existing behaviors.

* test/ruby/test_file_exhaustive.rb: fix test introduced in r36811 for
  posix environments where HOME is not defined.  

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_1_9_3@37321 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

History

#1 [ruby-core:42481] Updated by luislavena (Luis Lavena) about 6 years ago

  • Category set to core
  • Assignee set to usa (Usaku NAKAMURA)
  • Target version set to 2.0.0

#2 [ruby-core:42482] Updated by luislavena (Luis Lavena) about 6 years ago

Usa,

Do you approve this be merged?

Thank you.

#3 [ruby-core:42976] Updated by jonforums (Jon Forums) about 6 years ago

As there's been no negative feedback in the past 17 days, is this patch acceptable as-is and can it now be committed to trunk?

#4 [ruby-core:42978] Updated by usa (Usaku NAKAMURA) about 6 years ago

  • Status changed from Open to Assigned
  • Assignee changed from usa (Usaku NAKAMURA) to h.shirosaki (Hiroshi Shirosaki)

I don't object.
Shirosaki-san, please commit it.

#5 Updated by Anonymous about 6 years ago

  • Status changed from Assigned to Closed
  • % Done changed from 0 to 100

This issue was solved with changeset r34849.
Hiroshi, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.


  • Makefile.in (PLATFORM_DIR): add a variable for win32 directory.
  • Makefile.in (clean-platform): add new target.
    It cleans win32 directory.

  • common.mk (clean): add a dependency for win32 directory.

  • common.mk (distclean): ditto.

  • common.mk (distclean-platform): add new target.
    It cleans win32 directory.

  • common.mk ($(PLATFORM_D)): add new target to make win32 directory.

  • common.mk (win32/win32.$(OBJEXT)): move win32.o into win32
    directory.

  • common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

  • configure.in: move win32.o into win32 directory and add
    win32/file.o to MISSING.

  • file.c (file_load_ok, rb_file_load_ok): replace static
    file_load_ok() with public rb_file_load_ok().
    It's to link Windows implementation in win32/file.c.

  • file.c (rb_find_file_ext_safe): ditto.

  • file.c (rb_find_file_safe): ditto.

  • win32/file.c (rb_file_load_ok): new file. Add Windows specific
    optimized implementation of rb_file_load_ok(). We created a
    separated file to avoid too many #ifdef macro which is unreadable.

  • win32/Makefile.sub (PLATFORM_DIR): add a variable for win32
    directory.

  • win32/Makefile.sub (MISSING): move win32.obj into win32
    directory and add win32/file.obj to MISSING.

  • win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
    It's because miniruby doesn't exist when making win32 directory.

  • win32/Makefile.sub (clean-platform): add new target to clean win32
    directory.

  • win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
    win32/file.c to build properly.

  • win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
    win32 directory.

Patch created with Luis Lavena.
[Feature #5999]

#6 [ruby-core:43004] Updated by h.shirosaki (Hiroshi Shirosaki) about 6 years ago

I don't object.
Shirosaki-san, please commit it.

Thank you. I committed it at r34849.

--
Hiroshi Shirosaki

Also available in: Atom PDF