Bug #16680: Symlink folder in $LOAD_PATH does not work with autoload

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: 2.7.0
• Backport: 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: REQUIRED

[zw963 (Wei Zheng)]

The following is a full reproducible process. I will use a gem named 'looksee' to reproduce this.

1. Do gem install looksee

2. Go to gem folder and copy lib content into /tmp/test_looksee so that we have a folder like this:

    ╰─ $ tree /tmp/test_looksee
   /tmp/test_looksee
   └── lib
       ├── looksee
       │   ├── adapter
       │   │   ├── base.rb
       │   │   └── rubinius.rb
       │   ├── adapter.rb
       │   ├── clean.rb
       │   ├── columnizer.rb
       │   ├── core_ext.rb
       │   ├── editor.rb
       │   ├── help.rb
       │   ├── inspector.rb
       │   ├── lookup_path.rb
       │   ├── mri.so
       │   └── version.rb
       └── looksee.rb
   

3. Create a new symlink to this folder.

    ╰─ $ln -s /tmp/test_looksee /tmp/test_looksee1
   
    ╰─ $ ls -alhd /tmp/test_looksee*
   drwxr-xr-x 3 zw963 zw963 60 2020-03-08 01:39 /tmp/test_looksee/
   lrwxrwxrwx 1 zw963 zw963 13 2020-03-08 02:08 /tmp/test_looksee1 -> 
   test_looksee//
   

4. Run the following command to reproduce this issue.

    ╰─ $ ruby -I/tmp/test_looksee1/lib/ -rlooksee -e 'puts 100'
   Traceback (most recent call last):
           10: from /home/zw963/others/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
        9: from /home/zw963/others/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
        8: from /tmp/test_looksee1/lib/looksee.rb:1:in `<top (required)>'
        7: from /home/zw963/others/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
        6: from /home/zw963/others/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
        5: from /tmp/test_looksee1/lib/looksee/clean.rb:4:in `<top (required)>'
        4: from /tmp/test_looksee1/lib/looksee/clean.rb:171:in `<module:Looksee>'
        3: from /home/zw963/others/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
        2: from /home/zw963/others/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
        1: from /tmp/test_looksee1/lib/looksee/adapter.rb:1:in `<top (required)>'
   

/tmp/test_looksee1/lib/looksee/adapter.rb:2:in `module:Looksee': uninitialized constant Looksee::Adapter (NameError)
```

You may want to ask why use symlinks folder in $LOAD_PATH. I think this is a quite common case; many ruby developers have a local gem not managed by Rubygems or Bundler. They just use $LOAD_PATH to require/load it, and it is hard to ensure everyone's folder is not a symlink.

I think this is a breaking change. All those codes worked quite well before 2.7 because old codes always
follow symlinks. The following is an example from ruby 2.6.3, which works.

 ╰─ $ ruby -I/tmp/test_looksee1/lib/ -rlooksee -e 'puts 100'
Traceback (most recent call last):
        8: from /home/zw963/others/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        7: from /home/zw963/others/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        6: from /tmp/test_looksee/lib/looksee.rb:1:in `<top (required)>'
        5: from /home/zw963/others/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        4: from /home/zw963/others/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        3: from /tmp/test_looksee/lib/looksee/clean.rb:4:in `<top (required)>'
        2: from /tmp/test_looksee/lib/looksee/clean.rb:171:in `<module:Looksee>'
        1: from /home/zw963/others/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/home/zw963/others/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': incompatible library version - /tmp/test_looksee/lib/looksee/mri.so (LoadError)

As we can seen, those breaking codes come from autoload :Adapter, 'looksee/adapter'. I suspect
maybe this is caused by nested autoload. But this is only a guess, please check. Thank you.

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #16462: Ruby 2.7 autoload not working with $RUBYLIB (maybe circular dependency error) added

[zw963 (Wei Zheng)]

Sorry, any update on this?

[Eregon (Benoit Daloze)]

Maybe a solution is to always resolve symlinks in the cache of $LOAD_PATH?
One issue is that $LOAD_PATH entries might not necessarily exist.

[zw963 (Wei Zheng)]

Eregon (Benoit Daloze) wrote in #note-3:

One issue is that $LOAD_PATH entries might not necessarily exist.

Sorry, not really understood what your means about reference comment.

I think maybe it acceptable for only support non-symlink $LOAD_PATH when update to ruby 2.7.
but, it should be mentions on release change log and mention why we change this behavior
before ruby 2.7 release.

[nobu (Nobuyoshi Nakada)]

I can't reproduce it with the master.

$ ruby -v
ruby 2.8.0dev (2020-06-13T14:33:40Z master 2496bdb28f) [x86_64-linux]

First, copied looksee/lib directory to /tmp/test_looksee, and linked it to /tmp/test_looksee1, correct?

$ ls -lRA /tmp/test_looksee*
lrwxrwxrwx 1 ubuntu ubuntu   17 Jun 14 07:49 /tmp/test_looksee1 -> /tmp/test_looksee/

/tmp/test_looksee:
total 4
drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 14 07:55 lib/

/tmp/test_looksee/lib:
total 8
drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 14 07:42 looksee/
-rw-r--r-- 1 ubuntu ubuntu   51 Jun 14 07:42 looksee.rb

/tmp/test_looksee/lib/looksee:
total 44
drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 14 07:42 adapter/
-rw-r--r-- 1 ubuntu ubuntu  246 Jun 14 07:42 adapter.rb
-rw-r--r-- 1 ubuntu ubuntu 5276 Jun 14 07:42 clean.rb
-rw-r--r-- 1 ubuntu ubuntu 2107 Jun 14 07:42 columnizer.rb
-rw-r--r-- 1 ubuntu ubuntu  732 Jun 14 07:42 core_ext.rb
-rw-r--r-- 1 ubuntu ubuntu 1522 Jun 14 07:42 editor.rb
-rw-r--r-- 1 ubuntu ubuntu 1875 Jun 14 07:42 help.rb
-rw-r--r-- 1 ubuntu ubuntu 2046 Jun 14 07:42 inspector.rb
-rw-r--r-- 1 ubuntu ubuntu 2292 Jun 14 07:42 lookup_path.rb
-rw-r--r-- 1 ubuntu ubuntu  128 Jun 14 07:42 version.rb

/tmp/test_looksee/lib/looksee/adapter:
total 8
-rw-r--r-- 1 ubuntu ubuntu 2018 Jun 14 07:42 base.rb
-rw-r--r-- 1 ubuntu ubuntu  629 Jun 14 07:42 rubinius.rb

Second, run with the real path, and got a LoadError on mri.so as expected.

$ ruby -I/tmp/test_looksee/lib/ -rlooksee -e 'puts 100'
/home/ubuntu/.rbenv/versions/master-2020-06-13/lib/ruby/2.8.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- looksee/mri.so (LoadError)
	from /home/ubuntu/.rbenv/versions/master-2020-06-13/lib/ruby/2.8.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	from /tmp/test_looksee/lib/looksee/clean.rb:171:in `<module:Looksee>'
	from /tmp/test_looksee/lib/looksee/clean.rb:4:in `<top (required)>'
	from /home/ubuntu/.rbenv/versions/master-2020-06-13/lib/ruby/2.8.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	from /home/ubuntu/.rbenv/versions/master-2020-06-13/lib/ruby/2.8.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	from /tmp/test_looksee/lib/looksee.rb:1:in `<top (required)>'
	from /home/ubuntu/.rbenv/versions/master-2020-06-13/lib/ruby/2.8.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	from /home/ubuntu/.rbenv/versions/master-2020-06-13/lib/ruby/2.8.0/rubygems/core_ext/kernel_require.rb:83:in `require'
bash: exit 1

Then, tried the symlink path, got the same error.

$ ruby -I/tmp/test_looksee1/lib/ -rlooksee -e 'puts 100'
/home/ubuntu/.rbenv/versions/master-2020-06-13/lib/ruby/2.8.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- looksee/mri.so (LoadError)
	from /home/ubuntu/.rbenv/versions/master-2020-06-13/lib/ruby/2.8.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	from /tmp/test_looksee/lib/looksee/clean.rb:171:in `<module:Looksee>'
	from /tmp/test_looksee/lib/looksee/clean.rb:4:in `<top (required)>'
	from /home/ubuntu/.rbenv/versions/master-2020-06-13/lib/ruby/2.8.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	from /home/ubuntu/.rbenv/versions/master-2020-06-13/lib/ruby/2.8.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	from /tmp/test_looksee/lib/looksee.rb:1:in `<top (required)>'
	from /home/ubuntu/.rbenv/versions/master-2020-06-13/lib/ruby/2.8.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	from /home/ubuntu/.rbenv/versions/master-2020-06-13/lib/ruby/2.8.0/rubygems/core_ext/kernel_require.rb:83:in `require'
bash: exit 1

[nobu (Nobuyoshi Nakada)]

I could reproduce it with 2.7.1 and 2.7 head.

[zw963 (Wei Zheng)]

nobu (Nobuyoshi Nakada) wrote in #note-6:

I can't reproduce it with the master.

I could reproduce it with 2.7.1 and 2.7 head.

Cool, good news, so, it was fixed on our master, right?

[sawa (Tsuyoshi Sawada)]

• Subject changed from [Breaking Change] Ruby 2.7 not support symlinks folder in $LOAD_PATH to work with autoload. to Symlink folder in $LOAD_PATH does not work with autoload
• Description updated (diff)

[jeremyevans0 (Jeremy Evans)]

• Status changed from Open to Closed
• Backport changed from 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN to 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: REQUIRED

I tried this with Ruby 2.6, 2.7, and 3.0, and got the same results with all three:

$ ls -ld /home/jeremy/tmp/looksee1
lrwxr-xr-x  1 jeremy  jeremy  7 Apr 28 13:19 /home/jeremy/tmp/looksee1 -> looksee
$ ruby26 -I /home/jeremy/tmp/looksee1/lib -r looksee -e 'puts 100'
/usr/local/lib/ruby/2.6/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- looksee/mri.so (LoadError)
        from /usr/local/lib/ruby/2.6/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /home/jeremy/tmp/looksee/lib/looksee/clean.rb:171:in `<module:Looksee>'
        from /home/jeremy/tmp/looksee/lib/looksee/clean.rb:4:in `<top (required)>'
        from /usr/local/lib/ruby/2.6/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /usr/local/lib/ruby/2.6/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /home/jeremy/tmp/looksee/lib/looksee.rb:1:in `<top (required)>'
        from /usr/local/lib/ruby/2.6/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /usr/local/lib/ruby/2.6/rubygems/core_ext/kernel_require.rb:54:in `require'
$ ruby27 -I /home/jeremy/tmp/looksee1/lib -r looksee -e 'puts 100'
Traceback (most recent call last):
        8: from /usr/local/lib/ruby/2.7/rubygems/core_ext/kernel_require.rb:83:in `require'
        7: from /usr/local/lib/ruby/2.7/rubygems/core_ext/kernel_require.rb:83:in `require'
        6: from /home/jeremy/tmp/looksee/lib/looksee.rb:1:in `<top (required)>'
        5: from /usr/local/lib/ruby/2.7/rubygems/core_ext/kernel_require.rb:83:in `require'
        4: from /usr/local/lib/ruby/2.7/rubygems/core_ext/kernel_require.rb:83:in `require'
        3: from /home/jeremy/tmp/looksee/lib/looksee/clean.rb:4:in `<top (required)>'
        2: from /home/jeremy/tmp/looksee/lib/looksee/clean.rb:171:in `<module:Looksee>'
        1: from /usr/local/lib/ruby/2.7/rubygems/core_ext/kernel_require.rb:83:in `require'
/usr/local/lib/ruby/2.7/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- looksee/mri.so (LoadError)
$ ruby30 -I /home/jeremy/tmp/looksee1/lib -r looksee -e 'puts 100'
<internal:/usr/local/lib/ruby/3.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- looksee/mri.so (LoadError)
        from <internal:/usr/local/lib/ruby/3.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /home/jeremy/tmp/looksee/lib/looksee/clean.rb:171:in `<module:Looksee>'
        from /home/jeremy/tmp/looksee/lib/looksee/clean.rb:4:in `<top (required)>'
        from <internal:/usr/local/lib/ruby/3.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/usr/local/lib/ruby/3.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /home/jeremy/tmp/looksee/lib/looksee.rb:1:in `<top (required)>'
        from <internal:/usr/local/lib/ruby/3.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/usr/local/lib/ruby/3.0/rubygems/core_ext/kernel_require.rb>:85:in `require'

I tried with both the master branch and the v4.2.0 tag for looksee with the same results, just in case looksee had worked around this problem another way.

So I think this problem has been resolved by Ruby 2.7.3. If you can still recreate the issue on Ruby 2.7.3, please bisect to find the failing commit and fixing commit, and note it in this ticket, and we can mark this for backporting.