Bug #19233: Failed to install sqlite3 gem since 7f1ca666424849134990d022266bcd4d6636465f using Docker

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 3.2.0dev (2022-10-02T06:19:14Z :detached: 7f1ca66642) [x86_64-linux]
• Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN

[yahonda (Yasuo Honda)]

Failed to install sqlite3 gem since 7f1ca666424849134990d022266bcd4d6636465f using Docker. This issue comes from https://github.com/rails/rails/pull/46711 .

Steps to reproduce

1. Install Docker
2. Follow this step

git clone https://github.com/yahonda/rep19189
cd rep19189
docker build .

Expected behavior

It should pass.

Actual behavior

It fails Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

$ docker build .
Sending build context to Docker daemon  69.63kB
Step 1/5 : FROM rubylang/rubyfarm:7f1ca666424849134990d022266bcd4d6636465f
 ---> cdd68058177d
Step 2/5 : RUN apt-get update -qq && apt-get install -y sqlite3 libsqlite3-dev pkg-config
 ---> Using cache
 ---> b8928c4f6425
Step 3/5 : WORKDIR /myapp
 ---> Using cache
 ---> bae259f8abf3
Step 4/5 : COPY Gemfile /myapp/Gemfile
 ---> Using cache
 ---> a9fe0822ad64
Step 5/5 : RUN bundle install
 ---> Running in 00cd9fc0e691
Don't run Bundler as root. Installing your bundle as root will break this
application for all non-root users on this machine.
Fetching gem metadata from https://rubygems.org/.....
Resolving dependencies...
Using bundler 2.4.0.dev
Fetching mini_portile2 2.8.0
Installing mini_portile2 2.8.0
Fetching sqlite3 1.5.4
Installing sqlite3 1.5.4 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ext/sqlite3
/opt/ruby/bin/ruby -I /opt/ruby/lib/ruby/3.2.0+2 extconf.rb
Ignoring debug-1.6.2 because its extensions are not built. Try: gem pristine
debug --version 1.6.2
Building sqlite3-ruby using packaged sqlite3.
Extracting sqlite-autoconf-3400000.tar.gz into
tmp/x86_64-linux-gnu/ports/sqlite3/3.40.0... OK
Running 'configure' for sqlite3 3.40.0... OK
Running 'compile' for sqlite3 3.40.0... OK
Running 'install' for sqlite3 3.40.0... OK
Activating sqlite3 3.40.0 (from
/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0)...

Could not configure the build properly (pkg_config). Please install either the
`pkg-config` utility or the `pkg-config` rubygem.

*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/opt/ruby/bin/$(RUBY_BASE_NAME)
	--help
	--download-dependencies
	--with-sqlcipher
	--without-sqlcipher
	--with-sqlcipher-dir
	--without-sqlcipher-dir
	--with-sqlcipher-include
	--without-sqlcipher-include
	--with-sqlcipher-lib
	--without-sqlcipher-lib
	--enable-system-libraries
	--disable-system-libraries
	--with-sqlcipher
	--without-sqlcipher
	--with-sqlcipher-dir
	--without-sqlcipher-dir
	--with-sqlcipher-include
	--without-sqlcipher-include
	--with-sqlcipher-lib
	--without-sqlcipher-lib
	--with-sqlite-source-dir
--with-/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-dir
--without-/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-dir
--with-/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-include
--without-/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-include=${/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-dir}/include
--with-/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-lib
--without-/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-lib=${/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-dir}/lib
--with-/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-config
--without-/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-config
	--with-pkg-config
	--without-pkg-config

To see why this extension failed to compile, please check the mkmf.log which can
be found here:

/opt/ruby/lib/ruby/gems/3.2.0+2/extensions/x86_64-linux/3.2.0+2/sqlite3-1.5.4/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in
/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4 for inspection.
Results logged to
/opt/ruby/lib/ruby/gems/3.2.0+2/extensions/x86_64-linux/3.2.0+2/sqlite3-1.5.4/gem_make.out

  /opt/ruby/lib/ruby/3.2.0+2/rubygems/ext/builder.rb:102:in `run'
  /opt/ruby/lib/ruby/3.2.0+2/rubygems/ext/ext_conf_builder.rb:28:in `build'
  /opt/ruby/lib/ruby/3.2.0+2/rubygems/ext/builder.rb:171:in `build_extension'
/opt/ruby/lib/ruby/3.2.0+2/rubygems/ext/builder.rb:205:in `block in
build_extensions'
  /opt/ruby/lib/ruby/3.2.0+2/rubygems/ext/builder.rb:202:in `each'
  /opt/ruby/lib/ruby/3.2.0+2/rubygems/ext/builder.rb:202:in `build_extensions'
  /opt/ruby/lib/ruby/3.2.0+2/rubygems/installer.rb:843:in `build_extensions'
/opt/ruby/lib/ruby/3.2.0+2/bundler/rubygems_gem_installer.rb:72:in
`build_extensions'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/rubygems_gem_installer.rb:28:in `install'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/source/rubygems.rb:202:in `install'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/installer/gem_installer.rb:54:in `install'
/opt/ruby/lib/ruby/3.2.0+2/bundler/installer/gem_installer.rb:16:in
`install_from_spec'
/opt/ruby/lib/ruby/3.2.0+2/bundler/installer/parallel_installer.rb:186:in
`do_install'
/opt/ruby/lib/ruby/3.2.0+2/bundler/installer/parallel_installer.rb:177:in
`block in worker_pool'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/worker.rb:62:in `apply_func'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/worker.rb:57:in `block in process_queue'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/worker.rb:54:in `loop'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/worker.rb:54:in `process_queue'
/opt/ruby/lib/ruby/3.2.0+2/bundler/worker.rb:91:in `block (2 levels) in
create_threads'

An error occurred while installing sqlite3 (1.5.4), and Bundler cannot continue.

In Gemfile:
  sqlite3
The command '/bin/sh -c bundle install' returned a non-zero code: 5

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #19189: Ruby 3.1.3/3.2.x can no longer find pkg-config if not present at buildtime added

[yahonda (Yasuo Honda)]

There are other findings so far:

• It works fine with Ruby installed locally without Docker.
• It also works fine using gem "sqlite3", "< 1.5"

[S_H_ (Shun Hiraoka)]

Hi.

As a test, I also built a container as follows

FROM rubylang/rubyfarm:7f1ca666424849134990d022266bcd4d6636465f
# FROM rubylang/rubyfarm:88c12a29372e3f908190c6af0aa1b2ac6b78fbd1 # passes
# FROM rubylang/rubyfarm:7f1ca666424849134990d022266bcd4d6636465f # fails
# FROM rubylang/ruby:master-nightly-focal # fails
RUN apt-get update -qq && apt-get install -y sqlite3 libsqlite3-dev pkg-config
WORKDIR /myapp
COPY Gemfile /myapp/Gemfile

Then I ran bundle install in the container and got the same error.

sh@DESKTOP-L0NI312:~/rubydev/rep19189$ docker build . -t rep19189:without-bundle-install

...

sh@DESKTOP-L0NI312:~/rubydev/rep19189$ docker run -it rep19189:without-bundle-install
root@0f6123678405:/myapp# bundle install
Don't run Bundler as root. Installing your bundle as root will break this application for all non-root users on this machine.
Fetching gem metadata from https://rubygems.org/.....
Resolving dependencies...
Fetching mini_portile2 2.8.0
Using bundler 2.4.0.dev
Installing mini_portile2 2.8.0
Fetching sqlite3 1.5.4
Installing sqlite3 1.5.4 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ext/sqlite3
/opt/ruby/bin/ruby -I /opt/ruby/lib/ruby/3.2.0+2 extconf.rb
Ignoring debug-1.6.2 because its extensions are not built. Try: gem pristine debug --version 1.6.2
Building sqlite3-ruby using packaged sqlite3.
Extracting sqlite-autoconf-3400000.tar.gz into tmp/x86_64-linux-gnu/ports/sqlite3/3.40.0... OK
Running 'configure' for sqlite3 3.40.0... OK
Running 'compile' for sqlite3 3.40.0... OK
Running 'install' for sqlite3 3.40.0... OK
Activating sqlite3 3.40.0 (from /opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0)...

Could not configure the build properly (pkg_config). Please install either the `pkg-config` utility or the `pkg-config` rubygem.

*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/opt/ruby/bin/$(RUBY_BASE_NAME)
        --help
        --download-dependencies
        --with-sqlcipher
        --without-sqlcipher
        --with-sqlcipher-dir
        --without-sqlcipher-dir
        --with-sqlcipher-include
        --without-sqlcipher-include
        --with-sqlcipher-lib
        --without-sqlcipher-lib
        --enable-system-libraries
        --disable-system-libraries
        --with-sqlcipher
        --without-sqlcipher
        --with-sqlcipher-dir
        --without-sqlcipher-dir
        --with-sqlcipher-include
        --without-sqlcipher-include
        --with-sqlcipher-lib
        --without-sqlcipher-lib
        --with-sqlite-source-dir
        --with-/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-dir
        --without-/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-dir
        --with-/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-include
--without-/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-include=${/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-dir}/include
        --with-/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-lib
--without-/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-lib=${/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-dir}/lib
        --with-/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-config
        --without-/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc-config
        --with-pkg-config
        --without-pkg-config

To see why this extension failed to compile, please check the mkmf.log which can be found here:

  /opt/ruby/lib/ruby/gems/3.2.0+2/extensions/x86_64-linux/3.2.0+2/sqlite3-1.5.4/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in /opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4 for inspection.
Results logged to /opt/ruby/lib/ruby/gems/3.2.0+2/extensions/x86_64-linux/3.2.0+2/sqlite3-1.5.4/gem_make.out

  /opt/ruby/lib/ruby/3.2.0+2/rubygems/ext/builder.rb:102:in `run'
  /opt/ruby/lib/ruby/3.2.0+2/rubygems/ext/ext_conf_builder.rb:28:in `build'
  /opt/ruby/lib/ruby/3.2.0+2/rubygems/ext/builder.rb:171:in `build_extension'
  /opt/ruby/lib/ruby/3.2.0+2/rubygems/ext/builder.rb:205:in `block in build_extensions'
  /opt/ruby/lib/ruby/3.2.0+2/rubygems/ext/builder.rb:202:in `each'
  /opt/ruby/lib/ruby/3.2.0+2/rubygems/ext/builder.rb:202:in `build_extensions'
  /opt/ruby/lib/ruby/3.2.0+2/rubygems/installer.rb:843:in `build_extensions'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/rubygems_gem_installer.rb:72:in `build_extensions'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/rubygems_gem_installer.rb:28:in `install'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/source/rubygems.rb:202:in `install'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/installer/gem_installer.rb:54:in `install'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/installer/parallel_installer.rb:186:in `do_install'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/installer/parallel_installer.rb:177:in `block in worker_pool'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/worker.rb:62:in `apply_func'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/worker.rb:57:in `block in process_queue'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/worker.rb:54:in `loop'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/worker.rb:54:in `process_queue'
  /opt/ruby/lib/ruby/3.2.0+2/bundler/worker.rb:91:in `block (2 levels) in create_threads'

An error occurred while installing sqlite3 (1.5.4), and Bundler cannot continue.

In Gemfile:
  sqlite3

The mkmf error log at that time was as follows.

root@0f6123678405:/myapp# cat /opt/ruby/lib/ruby/gems/3.2.0+2/extensions/x86_64-linux/3.2.0+2/sqlite3-1.5.4/mkmf.log
package configuration for /opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc is not found

[mdalessio (Mike Dalessio)]

Hi, I'm a co-maintainer of the sqlite3 gem and I wanted to add a little more context.

In v1.5, we simplified the extconf.rb by using MakeMakefile.pkg_config to read the sqlite3.pc file generated earlier during installation. Prior to v1.5 we only used the pkg_config method to find system libraries.

[mdalessio (Mike Dalessio)]

When I examine the files left in the docker container, I see this error in the mkmf.log:

package configuration for /opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc is not found

and that pc file contains:

root@8da2dee444e4:/myapp# cat /opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc
# Package Information for pkg-config

prefix=/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: SQLite
Description: SQL database engine
Version: 3.40.0
Libs: -L${libdir} -lsqlite3
Libs.private: -lz -lm -ldl -lpthread 
Cflags: -I${includedir}

This all seems as expected, so I don't think this is a problem with the gem. I'm not sure I understand why pkg_config is raising an exception, so I'll continue investigating.

[mdalessio (Mike Dalessio)]

When sqlite3-ruby's (v1.5+) extconf calls pkg_config(), the argument pkg is "/opt/ruby/lib/ruby/gems/3.2.0+2/gems/sqlite3-1.5.4/ports/x86_64-linux-gnu/sqlite3/3.40.0/lib/pkgconfig/sqlite3.pc".

In this docker container, RbConfig::CONFIG["PKG_CONFIG"] is set to "".

Previously I believe that this method set the variable pkgconfig to "pkg-config". However, after 7f1ca66 it falls through the if conditions and is set to nil, causing this error.

[nobu (Nobuyoshi Nakada)]

Isn’t this same as #19189?

[yahonda (Yasuo Honda)]

I have opened a new one because it does not reproduce against Ruby 3.1.3 Docker images.

FROM ruby:3.1.3
FROM rubylang/ruby:3.1.3-focal

#19189 says:

Starting with Ruby 3.1.3, mkmf.rb calls to pkg-config will fail if pkg-config wasn't present on the system at the time it was built.

[yahonda (Yasuo Honda)]

Confirmed this issue has been fixed since 613fca01486e47dee9364a2fd86b5f5e77fe23c8

[hsbt (Hiroshi SHIBATA)]

• Status changed from Open to Closed

[nobu (Nobuyoshi Nakada)]

• Related to deleted (Bug #19189: Ruby 3.1.3/3.2.x can no longer find pkg-config if not present at buildtime)

[nobu (Nobuyoshi Nakada)]

• Is duplicate of Bug #19189: Ruby 3.1.3/3.2.x can no longer find pkg-config if not present at buildtime added