Makefile: make testpack.tar.gz from (almost) all changed files.
What the developer wrote
Authored by Rusty Russell
Makefile: make testpack.tar.gz from (almost) all changed files.
Here are the speeds and sizes of different compressors (just
compressing the raw tar file):
bzip2: 34.15s 412MB
gzip -1: 9.96s 420MB
gzip -2: 10.39s 417MB
gzip -3: 11.63s 414MB
gzip -4: 12.29s 409MB
gzip -5: 14.41s 406MB
gzip -6: 19.02s 405MB
gzip -7: 22.96s 405MB
gzip -8: 39.63s 404MB
gzip -9: 72.19s 404MB
xz: 72.92s 407MB
Uncompressing time:
xz: 2.3 seconds
bzip2: 20 seconds
gzip -5: 3 seconds
Note that this target is only useful on a *freshly built* tree: it
will include everything config.vars and newer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
What changed, and why it matters
This commit changes how the project packages built files for testing in continuous integration (CI). It switches from a hand-maintained list of files bundled into a bzip2 archive to a rule that automatically collects almost all files newer than a build marker and compresses them with gzip. There is no user-facing behavior change, no network code, no cryptography, and no security-sensitive logic. It is purely a build/CI tooling change.
No security action required. Review as a normal build-system change if desired.
Security signals we found
No strong security signals were identified.
Evidence from the diff
The Makefile target testpack.tar.bz2 is replaced by testpack.tar.gz. The old target listed explicit built artifacts, test binaries, man pages, docs, and config.vars. The new target depends on aggregate targets (all-programs, all-fuzz-programs, all-test-programs, default-targets) and then uses find to include all files newer than config.vars while pruning external/ and target/, plus a small set of extras (config.vars, ccan/config.h, header_versions_gen.h, DEFAULT_TARGETS, EXTERNAL_LIBS). Compression is changed from bzip2 to gzip -5 based on the commit message’s benchmark of size/speed tradeoffs. The archive format is switched to POSIX for subsecond timestamp support.
Changed components
MakefileCI testpack packaging targetInspect captured patch +13 / −17
diff --git a/Makefile b/Makefile
index c18f4838..b6f28825 100644
--- a/Makefile
+++ b/Makefile
@@ -964,25 +964,21 @@ install-data: installdirs $(MAN1PAGES) $(MAN5PAGES) $(MAN7PAGES) $(MAN8PAGES) $(
install: install-program install-data
-# Non-artifacts that are needed for testing. These are added to the
-# testpack.tar, used to transfer things between builder and tester
-# phase. If you get a missing file/executable while testing on CI it
-# is likely missing from this variable.
-TESTBINS = \
- $(CLN_PLUGIN_EXAMPLES) \
- tests/plugins/test_libplugin \
- tests/plugins/channeld_fakenet \
- tests/plugins/test_selfdisable_after_getmanifest
+# We exclude most of target/ and external, but we need:
+# 1. config files (we only tar up files *newer* than these)
+# 2. $(DEFAULT_TARGETS) for rust stuff.
+# 3. $(EXTERNAL_LIBS) for prebuild external libraries.
+TESTPACK_EXTRAS := \
+ config.vars ccan/config.h \
+ header_versions_gen.h \
+ $(DEFAULT_TARGETS) \
+ $(EXTERNAL_LIBS)
# The testpack is used in CI to transfer built artefacts between the
-# build and the test phase. This is necessary because the fixtures in
-# `tests/` explicitly use the binaries built in the current directory
-# rather than using `$PATH`, as that may pick up some other installed
-# version of `lightningd` leading to bogus results. We bundle up all
-# built artefacts here, and will unpack them on the tester (overlaying
-# on top of the checked out repo as if we had just built it in place).
-testpack.tar.bz2: $(BIN_PROGRAMS) $(PKGLIBEXEC_PROGRAMS) $(PLUGINS) $(PY_PLUGINS) $(MAN1PAGES) $(MAN5PAGES) $(MAN7PAGES) $(MAN8PAGES) $(DOC_DATA) config.vars $(TESTBINS) $(DEVTOOLS) $(TOOLS)
- tar -caf $@ $^
+# build and the test phase. Only useful on a freshly build tree!
+# We use Posix format for timestamps with subsecond accuracy.
+testpack.tar.gz: all-programs all-fuzz-programs all-test-programs default-targets
+ (find * -path external -prune -o -path target -prune -o -newer config.vars -type f -print; ls $(TESTPACK_EXTRAS)) | tar --verbatim-files-from -T- -c --format=posix -f - | gzip -5 > $@
uninstall:
@$(NORMAL_UNINSTALL)
Why this scored 15/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.