Bug #7716: Readdressing Autoload

• Status: Closed
• Assignee: matz (Yukihiro Matsumoto)
• Target version: -
• ruby -v: ruby 2.0.0dev (2013-01-18 trunk 38877) [x86_64-linux]

[trans (Thomas Sawyer)]

=begin
It has been known for a long time that Autoload does not utilize the normal require method, and thus is unaffected when require is overridden. This makes it impossible to fully implement a custom load system. Obviously, customizing the load system is a fairly rare usecase and thus a low priority. But it can and does occur. The most notable example is RubyGems itself. Without the ability to customize the load system, RubyGems would have been much more difficult, if not impossible, to create. It can be thankful that its particular design does not suffer greatly from the "autoload hole", nonetheless it does have a reported bug b/c of it, #4233. I too have a custom load system that I have been using in a development setting for many years. I use it to simplify vendoring between dependent projects. I would have liked to made public releases of the project, but the autoload issue has always been too much of a show-stopper to make it generally usable.

This issue with autoload has been reported since at least as far back as October 2007. That's over 5 years ago. Over these years, I have asked many times that this issue be addressed and why I needed it addressed. It has been very frustrating to watch this issue languish, year after year, going unresolved. I understand low priority issues, but 5 years!? Finally, just ((over a year ago)) Matz declared that "autoload will be dead" in issue #5653. "Well", I thought at the time, "that's one way to fix it". While I still wished autoload would actually be fixed, at least I could rest assured that in due course the problem would dissipate as new releases of projects are made deprecating their use of autoload.

In the last few days, I have asked a number of notable projects, including Bundler and RubyGems (a standard library mind you!) when we might expect autoload to be removed from their code. After all it's already been over a year since Matz' declaration. But, in all cases, they reported that they had no intentions of removing autoload in the foreseeable future. So much for strong discouragements from our benevolent dictator.

This week I revisited my custom load manager and re-factored the code a great deal. I'm very happy with the new code and think that it's about as good as it's going to get (minor improvements aside). I suppose I can at least be happy that I've had all this time to think about and improve my code. That's good. But at this point I'd would really like to release it all proper-like.

So... since this autoload issue was obviously still not going anywhere, I took it upon myself to fix. Mind you, I am not a C programmer and this is basically right at the limits of my ability to do anything with Ruby's C code. But, I was able to wrangle out how to make a fix and now offer it up here with an enclosed patch. The patch simply adds a Kernel singleton method called #require_autoload which can be overridden to customize it's operation. The patch includes a test to ensure it works. Perhaps there is some preference to handle this differenetly. That's fine, please adjust it as seen fit. But please do something! This issue has gone unaddressed long enough. And really, it is only a small bit of code. I ask, and pray, we will see this fix make it into the next release of 2.0 and back-ported tho 1.9. Please.
=end

[trans (Thomas Sawyer)]

Priority: High

[ko1 (Koichi Sasada)]

• Assignee set to matz (Yukihiro Matsumoto)
• Target version changed from 2.6 to 2.1.0

[naruse (Yui NARUSE)]

Could you summarize this?
This is hard to read for me.

Or Abstract/Background/Detail/Use case/Conclusion structure may help me.

[trans (Thomas Sawyer)]

Sure. I'll lay it out in it's more basic terms, step by step:

• Ruby allows us to override the Kernel #require and #load methods.

• That may seem dangerous, but used carefully it allows us to do useful and/or interesting things with the load system.

• The most famous example of such a use is RubyGems itself, but there are others.

• I have developed such a system myself. I have used in house for many years, but have never been able to release it.

• A severe limitation for such systems is that #autoload can't be hooked into because it uses an internal require method (in the C code) rather than an exposed method.

• The inability to override autoload's require prevents any alternate load system from fully functioning as it should.

• For RubyGems it is not a serious issue b/c the gem method can be called first it mitigates the problem. Nonetheless it still remains a minor issue (see #4233).

• For other systems, such as my own, it is a huge issue. My system simply cannot be used generally b/c it will not be able to work with any library that uses autoload.

• The fix is fairly simple. The require method that autoload uses simply needs to be made accessible via Ruby. Alternatively, autoload can be made to use the standard require method instead.

HTH

[trans (Thomas Sawyer)]

So it wasn't discussed. Thanks bunches.

[naruse (Yui NARUSE)]

I have DevelopersMeeting20130831Japan and we can discuss about this there.
So could you prepare a slide? and we discuss about this and provide our view.

Anyway thank you for summarize but it is still confusing.
I wrote your proposal in doc/contributing.rdoc's style as far as I understand.
Could you edit if your thought is not reflected to this.
http://www.ruby-doc.org/core-2.0/doc/contributing_rdoc.html#label-How+To+Request+Features

[Abstract]
autoload's require should be hookable

[Background]
I'm making my own packaging system.
But I'm in trouble because current autoload uses ruby's internal require and I cannnot hook it.
I want to hook it and use my package system's custom require.

[Proposal]
call Ruby's require method in autoload

[Details]

If it has complicated feature, describe it

[Usecase]
Packaging system which handles auto-loading.

[Discussion]

Discuss about this proposal. A list of pros and cons will help start discussion.

[Limitation]

Limitation of your proposal

[Another alternative proposal]

If there are alternative proposals, show them.

[See also]

Links to the other related resources

[trans (Thomas Sawyer)]

= Abstract
Autoload's require should be hookable.

= Background
I'm making my own load system. It is badly handicapped because current autoload
uses Ruby's internal require and I cannot hook it. I want to hook it to my
load system's custom require. I have other projects that are effected by this
issue too.

= Proposal
Call Ruby's require method in autoload.

= Details
The proposal is fairly straight-forward. The only complication is whether
autoload should have it's own special require method, rather then use
Ruby's regular require method. I can't think of any reasons for it have it's
own, but maybe there are. If so Ruby would need a new method,
e.g. Kernel.autoload_require, so it can be hooked into.

= Usecase
Developing alternate load or packaging systems which handle auto-loading.
RubyGems would also benefit a little.

= Discussion
The pros of this proposal is that it allows developers to fully explore
alternatives in load/package systems for Ruby. I can't think of any cons.

= Limitation
No limitations that I can think of.

= Another alternative proposal
As mentioned in Detail, if for some reason there is an issue with autoload using Ruby's require method,
the alternative is to create a special require method for it.

= See Also

• Issues
  • {Latest issue}[https://bugs.ruby-lang.org/issues/7716]
  • {Gem issue}[https://bugs.ruby-lang.org/issues/4233]
• Effected Projects
  • {Realms}[https://github.com/rubyworks/realms]
  • {Loadable}[https://github.com/rubyworks/loadable]
  • {Backload}[https://github.com/rubyworks/backload]
  • {RubyGems}[https://github.com/rubygems/rubygems]

[matz (Yukihiro Matsumoto)]

• Status changed from Open to Feedback

I understand your motivation.

But autoload is invoked asynchronously so hooking it may be dangerous sometimes especially under threading environment.
How do you think?

Matz.

[trans (Thomas Sawyer)]

Hmmm.. require and load are never invoked asynchronously?

I am not sure it matters though. Its use is for very specific case. Can it be left to the programmer to ensure thread safety? I don't think the hook can be made thread safe automatically, can it?

[hsbt (Hiroshi SHIBATA)]

• Target version changed from 2.1.0 to 2.2.0

[naruse (Yui NARUSE)]

• Target version deleted (2.2.0)

[jeremyevans0 (Jeremy Evans)]

• Status changed from Feedback to Closed

autoload was modified to call require in cd465d552c3a00341f4cb7f1d7a793d0ebcb6cde.