Feature #5653

"I strongly discourage the use of autoload in any standard libraries" (Re: autoload will be dead)

Added by Yukihiro Matsumoto over 2 years ago. Updated 3 months ago.

[ruby-core:41149]
Status:Assigned
Priority:Normal
Assignee:Yukihiro Matsumoto
Category:lib
Target version:current: 2.2.0

Description

Hi,

Today, I talked with NaHi about enhancing const_missing to enable
autoload-like feature with nested modules. But autoload itself has
fundamental flaw under multi-thread environment. I should have remove
autoload when I added threads to the language (threads came a few
months after autoload).

So I hereby declare the future deprecation of autoload. Ruby will
keep autoload for a while, since 2.0 should keep compatibility to 1.9.
But you don't expect it will survive further future, e.g. 3.0.

I strongly discourage the use of autoload in any standard libraries.

                        matz.

noname (500 Bytes) Anonymous, 11/22/2011 04:53 AM

5653.pdf (38.3 KB) Hiroshi Nakamura, 07/01/2012 12:24 AM

History

#1 Updated by Hiroshi Nakamura over 2 years ago

  • Subject changed from autoload will be dead to "I strongly discourage the use of autoload in any standard libraries" (Re: autoload will be dead)
  • Category set to lib
  • Target version set to 2.0.0

This ticket is for discussion about removing autoload from stdlib (or not)

% grep autoload {ext//,}lib// | wc -l
442

#2 Updated by Anonymous over 2 years ago

On Mon, Nov 21, 2011 at 05:28:25PM +0900, Hiroshi Nakamura wrote:

Issue #5653 has been updated by Hiroshi Nakamura.

Subject changed from autoload will be dead to "I strongly discourage the use of autoload in any standard libraries" (Re: autoload will be dead)
Category set to lib
Target version set to 2.0.0

This ticket is for discussion about removing autoload from stdlib (or not)

% grep autoload {ext//,}lib// | wc -l
442

Looks like tk has most of them:

[aaron@higgins ruby (trunk)]$ git grep autoload ext | grep 'ext/tk' | wc -l
417
[aaron@higgins ruby (trunk)]$

I removed them from psych, and I'll do the same with dl.

--
Aaron Patterson
http://tenderlovemaking.com/

#3 Updated by jonathan rochkind over 2 years ago

My understanding was that plain old 'require' had much the same flaw in a multi-threaded environment as autoload. No?

#4 Updated by Yukihiro Matsumoto over 2 years ago

Hi,

In message "Re: [ruby-trunk - Feature #5653] "I strongly discourage the use of autoload in any standard libraries" (Re: autoload will be dead)"
on Tue, 22 Nov 2011 09:57:54 +0900, jonathan rochkind jonathan@dnil.net writes:

|My understanding was that plain old 'require' had much the same flaw in a multi-threaded environment as autoload. No?

Since calling #require is explicit, there's plenty of chance to manage
them. Introducing single lock for #require loading is another issue.

                        matz.

#5 Updated by Eric Hodel over 2 years ago

On Nov 21, 2011, at 11:51 AM, Aaron Patterson wrote:

On Mon, Nov 21, 2011 at 05:28:25PM +0900, Hiroshi Nakamura wrote:

Issue #5653 has been updated by Hiroshi Nakamura.

Subject changed from autoload will be dead to "I strongly discourage the use of autoload in any standard libraries" (Re: autoload will be dead)
Category set to lib
Target version set to 2.0.0

This ticket is for discussion about removing autoload from stdlib (or not)

% grep autoload {ext//,}lib// | wc -l
442

Looks like tk has most of them:

[aaron@higgins ruby (trunk)]$ git grep autoload ext | grep 'ext/tk' | wc -l
417
[aaron@higgins ruby (trunk)]$

I removed them from psych, and I'll do the same with dl.

The newest version of rdoc has them, but I can remove them.

#6 Updated by Eric Wong over 2 years ago

Yukihiro Matsumoto matz@ruby-lang.org wrote:

Today, I talked with NaHi about enhancing const_missing to enable
autoload-like feature with nested modules. But autoload itself has
fundamental flaw under multi-thread environment. I should have remove
autoload when I added threads to the language (threads came a few
months after autoload).

Hi, many of my Ruby scripts/apps are single-threaded and I would
like to keep memory usage down.

How about keeping autoload unchanged for single-threaded use and have
modules registered via autoload instantly require everything registered
when Thread.new is called?

#7 Updated by Yehuda Katz over 2 years ago

It is common to use autoload to register a number of incompatible options.
For instance, Rack registers all possible server adapters, and loading the
adapters has side-effects.

I looked into this when I worked on Rails threadsafetiness and it is simply
incorrect to preload anything registered as an autoload.

Yehuda Katz
(ph) 718.877.1325

On Tue, Nov 22, 2011 at 3:51 PM, Eric Wong normalperson@yhbt.net wrote:

Yukihiro Matsumoto matz@ruby-lang.org wrote:

Today, I talked with NaHi about enhancing const_missing to enable
autoload-like feature with nested modules. But autoload itself has
fundamental flaw under multi-thread environment. I should have remove
autoload when I added threads to the language (threads came a few
months after autoload).

Hi, many of my Ruby scripts/apps are single-threaded and I would
like to keep memory usage down.

How about keeping autoload unchanged for single-threaded use and have
modules registered via autoload instantly require everything registered
when Thread.new is called?

#8 Updated by Luis Lavena over 2 years ago

On Nov 19, 2011 4:11 AM, "Yukihiro Matsumoto" matz@ruby-lang.org wrote:

Hi,

Today, I talked with NaHi about enhancing const_missing to enable
autoload-like feature with nested modules. But autoload itself has
fundamental flaw under multi-thread environment. I should have remove
autoload when I added threads to the language (threads came a few
months after autoload).

So I hereby declare the future deprecation of autoload. Ruby will
keep autoload for a while, since 2.0 should keep compatibility to 1.9.
But you don't expect it will survive further future, e.g. 3.0.

I strongly discourage the use of autoload in any standard libraries.

Thank you for the details matz, hope this means require and
$LOADEDFEATURES along $LOADPATH will get a performance refactoring.

Saying because lazy loading no longer be an option, startup times will
become a problem in some scenarios.

--
Luis Lavena
AREA 17

#9 Updated by Nikolai Weibull over 2 years ago

On Wed, Nov 23, 2011 at 04:34, Luis Lavena luislavena@gmail.com wrote:

On Nov 19, 2011 4:11 AM, "Yukihiro Matsumoto" matz@ruby-lang.org wrote:

Today, I talked with NaHi about enhancing const_missing to enable
autoload-like feature with nested modules.  But autoload itself has
fundamental flaw under multi-thread environment.  I should have remove
autoload when I added threads to the language (threads came a few
months after autoload).

So I hereby declare the future deprecation of autoload.  Ruby will
keep autoload for a while, since 2.0 should keep compatibility to 1.9.
But you don't expect it will survive further future, e.g. 3.0.

I strongly discourage the use of autoload in any standard libraries.

Thank you for the details matz, hope this means require and $LOADEDFEATURES
along $LOAD
PATH will get a performance refactoring.

Saying because lazy loading no longer be an option, startup times will
become a problem in some scenarios.

I second that. I have mostly switched to using load and
require_relative, as they are the fastest alternative on Windows.
They still do an insane amount of (seemingly) unnecessary work, but at
least they’re better than require. I currently still use autoload for
loading bigger features that aren’t immediately needed at start-up. I
think that this use-case for autoload is still valid and perhaps
should be in the future, in the same way Yehuda mentioned regarding
Rack’s server adapters.

#10 Updated by Thomas Sawyer over 2 years ago

"autoloading" can still be done by putting the require within a method that
is called only as needed.

The downside of this is that requires get pushed down into deeper levels of
code, making requirements less obvious to developers. Documentation not
withstanding, it's also not hard to work around. Just link us something
like:

 $AUTOREQ = Hash.new{|h,k|h[k]=[]}

 def req(key, path)
   $AUTOREQ[key.to_sym] << path
 end

 def use(key)
   $AUTOREQ[key.to_sym].each{ |path| require path }
 end

Then

 req :rdoc, 'rdoc'
 req :markdown, 'redcarpet'

 class Tmpl
   initialize(type)
     use type

I'm sure this can be greatly improved upon, maybe even generalized (and
thread safe?) to make a useful library gem.

#11 Updated by Stephen Touset over 2 years ago

=begin
One thing to keep in mind is that (({const_missing})) cannot be used to replicate (({autoload})) currently due to Ruby cascading constant lookup to the Object namespace. Example:

>> class Foo; end
>> class Bar; end
>> class Baz
|    autoload :Foo, 'baz/foo'
|
|    def self.const_missing(name)
|      require "baz/#{name.downcase}"
|    end  
|  end  
>> Baz::Foo
LoadError: cannot load such file -- baz/foo
>> Baz::Bar
=> Bar

You can see here that Baz::Foo wasn't detected (even though there's a Foo in the Object namespace) and the autoload triggers. Baz::Bar, however, resolves to Object::Bar and the (({const_missing}))-based autoload does not fire as expected.
=end

#12 Updated by Stephen Touset over 2 years ago

=begin
@ThomasSawyer That kind of approach falls apart when you have multiple entry points into your library that require various features.

I've long considered it (perhaps incorrectly) a best practice to organize my hierarchy as this gist:

https://gist.github.com/1412552

It has several advantages. Users can require everything in my library with only using the top-level file ((({require 'foo'}))), but without incurring an immediate performance penalty for loading the entire library. It becomes loaded progressively as modules are needed, and only needed modules are ever loaded. It also allows a user to require only a nested component of my library ((({require 'foo/baz/qux'}))) and have everything above it automatically pulled in; again, without pre-loading unnecessary parts of the library.

Of course, the disadvantage is that (({autoload})) is being removed. So what alternative exists that lets library authors be considerate to their users, while still ensuring thread-safety?
=end

#13 Updated by Yehuda Katz over 2 years ago

Stephen: There is an open request, together with a proposal and working patch to address the issue. Check it out: http://redmine.ruby-lang.org/issues/2740

Make sure to read down towards the bottom as the proposal changed.

#14 Updated by Stephen Touset over 2 years ago

=begin
After discussion last night with Yehuda, we both agreed that this issue isn't resolved by #2740. Since (({const_missing})) is never called when Ruby resolves a constant like (({Foo::Bar})) to (({Object::Bar})), it cannot be used as a replacement to (({autoload})), which ((does)) trigger before the constant lookup is delegated to (({Object})).

This is a more common occurrence than you might think. Requiring any gem or outside library that defines a top-level constant named the same as a nested constant you've autoloaded (via (({const_missing}))) in your project will prevent that nested constant from ever being visible.
=end

#15 Updated by Yusuke Endoh about 2 years ago

  • Status changed from Open to Assigned
  • Assignee set to Hiroshi Nakamura

Hello, NaHi-san

Hiroshi Nakamura wrote:

This ticket is for discussion about removing autoload from stdlib (or not)

It would be good to open tickets for each library that uses autoload.

And, what do you think about Stephen and Yehuda's opinion?
It looks reasonable to me.

Stephen Touset wrote:

After discussion last night with Yehuda, we both agreed that this issue isn't resolved by #2740. Since const_missing is never called when Ruby resolves a constant like Foo::Bar to Object::Bar, it cannot be used as a replacement to autoload, which does trigger before the constant lookup is delegated to Object.

This is a more common occurrence than you might think. Requiring any gem or outside library that defines a top-level constant named the same as a nested constant you've autoloaded (via const_missing) in your project will prevent that nested constant from ever being visible.

Yusuke Endoh mame@tsg.ne.jp

#16 Updated by Thomas Sawyer almost 2 years ago

If autoload is still going to be around for awhile, can we at least get it modified to call Kernel#require? As I've said before, I use a customized load system and b/c of this inability to effect how autoload requires, I can't use my load system with any library that uses autoload.

#17 Updated by Hiroshi Nakamura almost 2 years ago

Endoh-san, here's "slide-show" of this proposal.

#18 Updated by Rodrigo Rosenfeld Rosas almost 2 years ago

I totally support this slide. The differences in start-up time of a typical Rails application will reveal very noticeable with and without auto-loading. It would be great to include this demonstration during the slide shows. Would that be possible?

#19 Updated by Hiroshi Nakamura almost 2 years ago

Slightly updated version. You can comment it, too: https://docs.google.com/presentation/d/1pP8XZBzoA5HehA5xyWBHda6qgpvFvmdMsVWaWauyr2o/edit

I'm not sure if we need to add the demonstration at this moment because the most important part is 'the problem does not exist' :)

#20 Updated by Rodrigo Rosenfeld Rosas almost 2 years ago

sorry, but I didn't get it. It is pretty easy to demonstrate the difference in performance between using autoload and require.

Just change "config.cache_classes = false" to "true" in config/environments/development.rb.

Then, time rails r 'p 1'. For a fresh Rails application here:

with cacheclasses set to false: 1.5s
with cache
classes set to true: 1.9s

In my actual app it goes from 4.5s to 6.3s.

#21 Updated by Yusuke Endoh almost 2 years ago

NaHi-san,
Please make a presentation yourself at the meeting!

Yusuke Endoh mame@tsg.ne.jp

#22 Updated by Thomas Sawyer almost 2 years ago

I just want to note that start-up times should be able to be improve somewhat by optimizing #require_relative.

Also, by putting #require_relative in #initialize methods of classes that need library instead of at toplevel and speed improvement is likewise gained.

Nonetheless, I agree that autoload is nice feature and is unfortunate loss.

#23 Updated by Yura Sokolov almost 2 years ago

rosenfeld (Rodrigo Rosenfeld Rosas) wrote:

sorry, but I didn't get it. It is pretty easy to demonstrate the difference in performance between using autoload and require.

Just change "config.cache_classes = false" to "true" in config/environments/development.rb.

Then, time rails r 'p 1'. For a fresh Rails application here:

with cacheclasses set to false: 1.5s
with cache
classes set to true: 1.9s

In my actual app it goes from 4.5s to 6.3s.

Considering start-up time: many people uses patch #5767 ( https://github.com/ruby/ruby/pull/68 ) to speedup startup time.
(Currently, it is part of falcon.patch in rvm. It could be easily installed with rvm reinstall 1.9.3-falcon --patch falcon)

With this patch difference becomes much lesser. time rails r 'p 1' for fresh rails app:

with cacheclasses set to false: 1.43s
with cache
classes set to true: 1.58s

#24 Updated by Rodrigo Rosenfeld Rosas almost 2 years ago

Yura, thanks for letting me know. I'll give it a try when I find some time.

#25 Updated by Yusuke Endoh over 1 year ago

  • Assignee changed from Hiroshi Nakamura to Yukihiro Matsumoto

matz, how do you feel about this ticket?
autoload should be marked as "deprecated" in 2.0.0?

Yusuke Endoh mame@tsg.ne.jp

#26 Updated by Rodrigo Rosenfeld Rosas about 1 year ago

+1 for removing autoload. It not only has troubles with thread safety, which I believe is fixable (although I confess I haven't thought much about it).

There is a more serious issue in my opinion that is unfixable with current specs.

For a while I've been experiencing a subtle bug in my Rails application that only happened on development mode (when autoload is enabled) and only for the first request. Recently I decided to understand what was happening and Xavier Noria showed me what was happening.

The problem is that I had some custom classes that relied on ActionView::Helpers::NumberHelper. Since all code in number_helper.rb was self-contained and it had no extra dependencies on ActionView, I just used it directly and my unit tests passed and my application always worked except for the first request when my custom class was used by the controller.

The problem is that once Rails is loaded it will declare an auto-load dependency on ActionView. Then when I "require 'actionview/helpers/numberhelper'" that file will be something like:

module ActionView
module Helpers
module NumberHelper

But since Helpers is set up to use autoload, once "module Helpers" is found, it will load "actionview/helpers.rb". And this file will then require "actionview/helpers/number_helper.rb" which causes a circular dependency problem.

Here is a complete Ruby-only code exemplifying the issue:

./test.rb:
autoload :A, 'a'
require 'a/b'

./lib/a.rb:
require 'a/b'

./lib/a/b.rb:
module A
module B
end
end

ruby -I lib test.rb

This kind of bug may be hard to track, so I agree with Matz that autoload should be dead.

#27 Updated by Koichi Sasada about 1 year ago

  • Target version changed from 2.0.0 to 2.1.0

time up. 2.0.0 was fixed.
Matz, could you consider it on 2.1.0?
Or please reject it.

#28 Updated by Jamie Winsor 11 months ago

Matz,

It appears that a patch was accepted into the soon to be released 1.9.4 branch (http://bugs.ruby-lang.org/issues/921) to fix the thread safety issues with autoload. Is this something that you still wish to see deprecated and removed from the language?

I agree with Yehuda's point regarding intelligent loading of modular, and possibly incompatible, components (http://bugs.ruby-lang.org/issues/5653#note-7).

Are there other reasons aside from the thread safety issues which would still make deprecating this feature a good choice?

#29 Updated by Sébastien Durand 10 months ago

Where are we now ? Will the autoload be deprecated in the future ?

#30 Updated by Boris Stitnicky 8 months ago

I would also be interested in the answer to se8's question.

#31 Updated by Hiroshi SHIBATA 3 months ago

  • Target version changed from 2.1.0 to current: 2.2.0

Also available in: Atom PDF