Bug #15461: ruby-2.6.0.tar.gz seems incorrectly packed

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: 2.6.0
• Backport: 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN

[valich (Valentin Fondaratov)]

1. Download ftp.ruby-lang.org/pub/ruby/2.6/ruby-2.6.0.tar.gz

2. Run from the console:

ruby -e 'require "zlib";require "minitar";Minitar.unpack(Zlib::GzipReader.new(File.open("ruby-2.6.0.tar.gz", "rb")), "x")'

It fails with an exception:

Traceback (most recent call last):
	17: from -e:1:in `<main>'
	16: from /home/user/.rvm/gems/ruby-2.5.0/gems/minitar-0.7/lib/archive/tar/minitar.rb:249:in `unpack'
	15: from /home/user/.rvm/gems/ruby-2.5.0/gems/minitar-0.7/lib/archive/tar/minitar/input.rb:27:in `open'
	14: from /home/user/.rvm/gems/ruby-2.5.0/gems/minitar-0.7/lib/archive/tar/minitar.rb:256:in `block in unpack'
	13: from /home/user/.rvm/gems/ruby-2.5.0/gems/minitar-0.7/lib/archive/tar/minitar/input.rb:90:in `each_entry'
	12: from /home/user/.rvm/gems/ruby-2.5.0/gems/minitar-0.7/lib/archive/tar/minitar/reader.rb:207:in `each_entry'
	11: from /home/user/.rvm/gems/ruby-2.5.0/gems/minitar-0.7/lib/archive/tar/minitar/reader.rb:207:in `loop'
	10: from /home/user/.rvm/gems/ruby-2.5.0/gems/minitar-0.7/lib/archive/tar/minitar/reader.rb:226:in `block in each_entry'
	 9: from /home/user/.rvm/gems/ruby-2.5.0/gems/minitar-0.7/lib/archive/tar/minitar/input.rb:91:in `block in each_entry'
	 8: from /home/user/.rvm/gems/ruby-2.5.0/gems/minitar-0.7/lib/archive/tar/minitar.rb:258:in `block (2 levels) in unpack'
	 7: from /home/user/.rvm/gems/ruby-2.5.0/gems/minitar-0.7/lib/archive/tar/minitar/input.rb:162:in `extract_entry'
	 6: from /home/user/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/fileutils.rb:193:in `mkdir_p'
	 5: from /home/user/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/fileutils.rb:193:in `each'
	 4: from /home/user/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/fileutils.rb:208:in `block in mkdir_p'
	 3: from /home/user/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/fileutils.rb:208:in `reverse_each'
	 2: from /home/user/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/fileutils.rb:210:in `block (2 levels) in mkdir_p'
	 1: from /home/user/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/fileutils.rb:229:in `fu_mkdir'
/home/user/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/fileutils.rb:229:in `mkdir': File exists @ dir_s_mkdir - x/ruby-2.6.0/libexec (Errno::EEXIST)

It seems flags are set incorrectly in the package. Same command for previous rubies tgzs works well.

[shevegen (Robert A. Heiler)]

This confuses me a bit though; using "tar" from
ftp://ftp.gnu.org/gnu/tar/tar-1.30.tar.xz repackaging works for me. Is this not rather
a behaviour problem of minitar if it chokes whereas tar works fine? I just repackaged
the .tar.gz into .tar.xz locally here via tar and that works.

[knu (Akinori MUSHA)]

Maybe this issue should be escalated to the minitar project.

One difference I found between the 2.5.3 and 2.6.0 tarballs is that the latter starts with the entry ruby-2.6.0/libexec/ whereas the former starts with ruby-2.5.3/, and minitar stucks at trying mkdir libexec when somehow there's already an empty regular file named libexec there.

[naruse (Yui NARUSE)]

• Status changed from Open to Third Party's Issue

It's considered a minitar's issue.

[valich (Valentin Fondaratov)]

OK, I've made the same experiment from the command line:

user@UNIT-867:~/Downloads$ gunzip ruby-2.6.0.tar.gz 
user@UNIT-867:~/Downloads$ tar -xf ruby-2.6.0.tar ruby-2.6.0
tar: ruby-2.6.0: Not found in archive
tar: Exiting with failure status due to previous errors
user@UNIT-867:~/Downloads$ gunzip ruby-2.5.3.tar.gz 
user@UNIT-867:~/Downloads$ tar -xf ruby-2.5.3.tar ruby-2.5.3

command-line tar fails, too, so I am not sure this is minitar issue.

Interestingly, normal tar -xzf ruby-2.6.0.tar.gz works.

[mame (Yusuke Endoh)]

What OS are you using? What version of tar are you using?

[valich (Valentin Fondaratov)]

mame (Yusuke Endoh) wrote:

What OS are you using? What version of tar are you using?

tar (GNU tar) 1.29,
Linux UNIT-867 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux aka Ubuntu 18.04.1

[mame (Yusuke Endoh)]

I tried docker run --rm -ti ubuntu:18.04 and it works. Why...

root@07f1505cc1d7:/# uname -a
Linux 07f1505cc1d7 4.18.0-13-generic #14-Ubuntu SMP Wed Dec 5 09:04:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@07f1505cc1d7:/# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"
root@07f1505cc1d7:/# tar --version
tar (GNU tar) 1.29
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.
root@07f1505cc1d7:/# wget https://ftp.ruby-lang.org/pub/ruby/2.6/ruby-2.6.0.tar.gz
--2018-12-25 13:02:15--  https://ftp.ruby-lang.org/pub/ruby/2.6/ruby-2.6.0.tar.gz
Resolving ftp.ruby-lang.org (ftp.ruby-lang.org)... 151.101.129.178, 151.101.193.178, 151.101.1.178, ...
Connecting to ftp.ruby-lang.org (ftp.ruby-lang.org)|151.101.129.178|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16687800 (16M) [application/x-tar]
Saving to: 'ruby-2.6.0.tar.gz'

ruby-2.6.0.tar.gz         100%[=====================================>]  15.91M  3.09MB/s    in 7.4s    

2018-12-25 13:02:23 (2.15 MB/s) - 'ruby-2.6.0.tar.gz' saved [16687800/16687800]

root@07f1505cc1d7:/# rm -rf ruby-2.6.0 ruby-2.6.0.tar
root@07f1505cc1d7:/# gunzip ruby-2.6.0.tar.gz 
root@07f1505cc1d7:/# tar -xf ruby-2.6.0.tar ruby-2.6.0

[knu (Akinori MUSHA)]

After investigation, I found the 2.6.0 tar file has the "typeflag" of '0' for each directory entry instead of '5' which modern tar implementations should use.
The reason major implementations like GNU tar and libarchive-based tar can extract the directories with typeflag '0' is because they have special treatment (for interoperability and compatibility) to treat a file with a trailing slash as directory regardless of the typeflag, which minitar is seemingly missing in its implementation.

libarchive: https://github.com/libarchive/libarchive/blob/master/libarchive/archive_read_support_format_tar.c#L566
GNU tar: http://git.savannah.gnu.org/cgit/tar.git/tree/src/extract.c#n1543

So, my conclusion is that this can be treated as a packaging issue while major tar implementations can deal with it.

[naruse (Yui NARUSE)]

• Status changed from Third Party's Issue to Open

[mame (Yusuke Endoh)]

knu (Akinori MUSHA) wrote:

After investigation, I found the 2.6.0 tar file has the "typeflag" of '0' for each directory entry instead of '5' which modern tar implementations should use.

This issue has been fixed at r66553. 2.6.1 will be a sane tarball.

[valich (Valentin Fondaratov)]

Thank you for swift fixing!

[naruse (Yui NARUSE)]

• Status changed from Open to Closed

Applied in changeset commit:ruby-git|477fe4ac23592fe0a835b3b8c15594315bf39d22.

---

merge revision(s) 66553:

    Set typeflag [Bug #15461]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_6@66562 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

[knu (Akinori MUSHA)]

Just submitted a PR to the third party: https://github.com/halostatue/minitar/pull/36

[knu (Akinori MUSHA)]

I confirmed the above PR fixes the problem as below:

% gem specific_install https://github.com/knu/minitar dir_with_typeflag_0
...
http installing from https://github.com/knu/minitar
...
Successfully installed
% ruby -e 'require "zlib";require "minitar";Minitar.unpack(Zlib::GzipReader.new(File.open("ruby-2.6.0.tar.gz", "rb")), "x")'
% ls -al x/ruby-2.6.0
total 12016
drwxr-xr-x  208 knu  staff     6656 Dec 26 13:41 .
drwxr-xr-x    3 knu  staff       96 Dec 26 13:41 ..
-rw-r--r--    1 knu  staff      506 Dec 26 13:41 .document
-rw-r--r--    1 knu  staff      336 Dec 26 13:41 .editorconfig
...

[austin (Austin Ziegler)]

I have released an update to minitar that should fix this. Thanks, knu! https://rubygems.org/gems/minitar/versions/0.8