Project

General

Profile

Actions

Bug #20641

closed

`lib/bundled_gems.rb` makes `Kernel.require` over 100x slower

Added by byroot (Jean Boussier) 5 months ago. Updated 3 months ago.

Status:
Closed
Assignee:
-
Target version:
-
[ruby-core:118637]

Description

I just discovered this while profiling Active Record's test suite, and I noticed 40% of the runtime was in $LOAD_PATH.resolve_feature_path, so much I thought it was a profiler bug.

But it turns out it's real. Various APIs do call require late, for instance Psych calls require 'date' every single time it parses a date: https://github.com/ruby/psych/blob/be0ba74e5613c20f213403e15914d24944c2652d/lib/psych/scalar_scanner.rb#L64

I've put together a quick benchmark:

# frozen_string_literal: true

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"

  gem "benchmark-ips"
end

Benchmark.ips do |x|
  x.report(RUBY_VERSION) { require "erb" }
  x.save! "/tmp/require.bench"
  x.compare!
end

And the difference is massive:

Calculating -------------------------------------
               3.2.2      6.450M (± 1.2%) i/s -     32.822M in   5.089538s

Comparison:
               3.2.2:  6449899.4 i/s
               3.3.3:    46996.8 i/s - 137.24x  slower

And that's with a small $LOAD_PATH, the bigger the application, the worse it is, if I add:

100.times do |i|
  $LOAD_PATH.unshift("/tmp/empty-#{i}")
end
Calculating -------------------------------------
               3.3.3      6.198k (± 6.1%) i/s -     30.968k in   5.018955s

Comparison:
               3.2.2:  3380939.6 i/s
               3.3.3:     6198.4 i/s - 545.46x  slower

I'm looking at a way to speed this up, but this is big enough that I believe we should backport the fix.

cc @hsbt (Hiroshi SHIBATA)

Updated by byroot (Jean Boussier) 5 months ago

Some extra context. This codepath is only active if using Bundler >= 2.5.0.

Actions #2

Updated by byroot (Jean Boussier) 5 months ago

  • Status changed from Open to Closed

Applied in changeset git|82aee1a9467c0f1bd33eb0247c5a0a8b8b9a5049.


bundled_gems.rb: Add a fast path

[Bug #20641] Gem::BUNDLED_GEMS.warning? adds a lot of extra
work on top of require. When the call end up atually loading code
the overhead is somewhat marginal.

However it's not uncommon for code to go some late require in some
paths, so it's expected that calling require with something already
required is somewhat fast, and bundled_gems.rb breaks this assumption.

To avoid this, we can have a fast path that in most case allow to
short-circuit all the heavy computations. If we extract the feature
basename and it doesn't match any of the bundled gems we care about
we can return very early.

With this change require 'date' is now only 1.33x slower on Ruby
3.3.3, than it was on Ruby 3.2.2, whereas before this change it
was at least 100x slower.

Updated by k0kubun (Takashi Kokubun) 3 months ago

  • Backport changed from 3.1: DONTNEED, 3.2: DONTNEED, 3.3: REQUIRED to 3.1: DONTNEED, 3.2: DONTNEED, 3.3: DONE
Actions

Also available in: Atom PDF

Like0
Like0Like0Like0