Misc #22206: Replace `tool/update-deps` with source-based dependency generation

• Status: Closed
• Assignee: nobu (Nobuyoshi Nakada)

[nobu (Nobuyoshi Nakada)]

I propose replacing tool/update-deps with tool/mkdepend.rb, a dependency generator written in Ruby that scans the source files directly.

The complete dependency lists would be generated into the build directory instead of being committed. The repository would retain only the source mappings and manual rules needed to determine which source builds each object.

Problems with tool/update-deps

tool/update-deps has several practical problems.

1. It depends on GNU make and GCC-specific output.

   It obtains build information from make -p and requires GCC’s -save-temps=obj option. This also means that ccache and similar compiler wrappers must be disabled. The dependency update procedure therefore cannot run directly on every platform supported by Ruby.

2. Updating dependencies is expensive.

   The current procedure requires a configured tree and a compiler build before dependencies can be inspected. A dependency-only change should not require compiling Ruby and its extensions first.

3. The generated depend files are very large.

   Most of their contents are mechanically generated header dependencies.
   They make reviews noisy, cause large diffs when a commonly included header changes, and are easy to leave stale. In the prototype, approximately 70,000 generated lines can be removed from 98 depend files.

4. The checked-in files duplicate information available from the sources.

   Most header dependencies can be obtained by recursively following #include directives. Only the object-to-source mappings and a small number of rules for generated or substituted sources need to be recorded explicitly.

Proposal

Add tool/mkdepend.rb and use it instead of tool/update-deps.

tool/mkdepend.rb scans C and parser sources recursively, resolves quoted and angle-bracket includes using Ruby’s source layout, and writes Make dependency rules. It does not require a configured build, GNU make, or a C compiler, and it runs with Ruby 3.1.

Unknown preprocessor conditions are handled conservatively by scanning both branches. Source-specific define and undef declarations are available when one branch must be selected. Small declarations in depend files also describe generated source templates, logical dependencies, and macro-based include names that cannot be inferred from ordinary file lookup.

Targets declared in the manual part of each depend file are treated as generated files. This avoids maintaining a second list of generated files.

Repository and build directory layout

The sections between these markers remain in the repository:

# AUTOGENERATED DEPENDENCIES START
array.$(OBJEXT): {$(VPATH)}array.c
# AUTOGENERATED DEPENDENCIES END

They contain only the minimal object-to-source mappings. Manual Make rules outside the section remain unchanged.

When a base Ruby is available, configure expands these mappings and writes the complete dependency files under .deps in the build directory. The generated Makefile includes those files. This applies to out-of-tree builds, GNU make builds, and Windows NMAKE builds.

Release snapshots expand the dependencies before packaging. A release tarball can therefore still be configured without a base Ruby and use the dependency files included in the source tree.

Commands

The proposed commands are:

# Check the committed source mappings.
ruby tool/mkdepend.rb --scope=all --sources --check

# Update the committed source mappings.
ruby tool/mkdepend.rb --scope=all --sources --inplace

# Generate complete dependencies.
ruby tool/mkdepend.rb --scope=all --output=.deps

The existing make check-depends and make fix-depends targets invoke these operations, so contributors do not need to remember the command-line details.

Benefits

• Dependency generation works without GNU make or GCC.
• It runs before the main build and does not require compiler output.
• Complete dependencies are generated for the current source tree on every build instead of being kept as a large generated snapshot in Git.
• Dependency updates become small and reviewable, usually changing only an object-to-source mapping or an explicit declaration.
• The same generator can produce paths appropriate for GNU make and NMAKE.
• Out-of-tree builds keep generated dependency data out of the source tree.

Limitations

This is a dependency scanner, not a complete C preprocessor. Conditions that cannot be decided safely are scanned in both directions, which may produce extra dependencies but should not omit required ones.

The scanner also cannot infer that a newly added C file should become a build object when there is no object rule for it. That information belongs to the Makefile or the minimal source mapping and must still be added explicitly.

Prototype

A working implementation is available at:

https://github.com/nobu/ruby/tree/mkdepend

It includes the source scanner, dependency declarations, configure and NMAKE integration, snapshot handling, dependency checks, and tests for Ruby 3.1.

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset git|7d4461e16298ca29b1409423d99f202d336a1f9d.

---

[Misc #22206] Generate dependencies without compiler builds

Replace tool/update-deps with a Ruby source scanner so dependency
updates no longer require GNU make, a configured build, or compiler
preprocessor output.

Keep compact source mappings and scanner declarations in dependency
files, and expand complete rules into the build directory. Avoid the
temporary fixture headers required by the compiler-driven prototype.

Use the generated rules from GNU make, BSD make, and NMake builds.
Support out-of-tree and read-only source trees, and retain expanded
dependencies shipped in release archives when baseruby is unavailable.

[nobu (Nobuyoshi Nakada)]

Currently, the upstreams’ depend files and the corresponding files are not maintained well.
For example, json/parser includes no longer used fbuffer.h, while not including the vendored headers.

[nobu (Nobuyoshi Nakada)]

• Status changed from Closed to Open
• Assignee set to nobu (Nobuyoshi Nakada)

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset git|b67f6129e3793bf53be28ba1c066b31f1037de5f.

---

[Misc #22206] Add dependency updater to mkmf

Allow extension projects to update marked dependency sections after
create_makefile has resolved their source and object lists.

Share the dependency scanner with tool/mkdepend.rb so bundled and
upstream extensions can generate the same dependency files.