Core Lightning

lightning-hsmtool: Fix getsecret does not show mnemonic after typing passphrase

Changelog-Fixed: `lightning-hsmtool`: `getsecret` does not show mnemonic after typing passphrase

wally: update libwally to v1.5.6

This got fixed in libwally v1.5.2: https://github.com/ElementsProject/libwally-core/commit/7e483c049b0a4405801f010e60c9f0335d2a617f

```
def test_sign_signed_psbt(node_factory, bitcoind, chainparams):
l1 = node_factory.get_node()
l1.fundwallet(10**6)

psbt = l1.rpc.txprepare([{l1.rpc.newaddr('bech32')['bech32']: 10000}])['psbt']
signed_psbt = l1.rpc.signpsbt(psbt)['signed_psbt']

if TEST_NETWORK != 'liquid-regtest':
# FIXME: ideally this would succeed, as a noop. But it shouldn't crash
> with pytest.raises(RpcError):
^^^^^^^^^^^^^^^^^^^^^^^
E Failed: DID NOT RAISE <class 'pyln.client.lightning.RpcError'>
```

Changelog-Changed: Update libwally to v1.5.6
Changelog-Fixed: JSON-RPC: `signpsbt` on an already-signed PSBT with taproot paths now succeeds as a no-op instead of failing.

common: saturate marginal_feerate() instead of overflowing

marginal_feerate() computed current_feerate * 1.1 as a double and
converted the result back to u32. Since current_feerate is chosen by
the peer in open_channel or update_fee, they could choose an absurdly
high value that overflows u32 after the computation. UBSan reports:

common/fee_states.c:179:10: runtime error: 4.72446e+09 is outside the range of representable values of type 'unsigned int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior common/fee_states.c:179:10

Do the arithmetic with u64 and saturate at UINT32_MAX to avoid the
undefined behavior.

Found by fuzzing with smite.

Changelog-Fixed: JSON-RPC: `listpeerchannels` no longer derives `receivable_msat` from an overflowed fee estimate when the peer sets an absurd `feerate_per_kw`.

wallet: rename b32script to scriptpubkey in PSBT change paths

Change addresses are no longer bech32-only (p2tr is the default form),
so the old name was misleading. Pure rename, no functional change;
also drop the unused txfilter.h include.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: register scriptpubkey watches at startup

init_wallet_scriptpubkey_watches walks every HD key (BIP32 + BIP86) up
to {bip32,bip86}_max_index + keyscan_gap and arms a watch for each
form, so bwatch can report deposits from the very first block it
scans. wallet_get_newindex does the same for fresh keys, keeping
coverage as the wallet grows.

The per-UTXO watch on unconfirmed change is now redundant (the
perennial per-key watch already covers that scriptpubkey), so drop it.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: add migrate_backfill_bwatch_tables

Populate our_outputs/our_txs from the legacy outputs/transactions
tables so the bwatch wallet path starts with the same state as the
existing wallet.

Use outputs rather than utxoset because it already contains only
wallet-owned rows and carries wallet metadata like reservations and
channel-close info.

Downgrade only drops the new tables: later commits keep mirroring
writes into the legacy tables, so no copy-back migration is needed.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: dual-write chaintopology UTXO changes into our_outputs

Before this commit:

chaintopology ──────────────> outputs
bwatch ─────────────────────> our_outputs
└────────────────────> outputs (downgrade mirror)

wallet reads ───────────────> outputs

After this commit:

chaintopology ──────────────> our_outputs
└─────────────> outputs (downgrade mirror)

bwatch ─────────────────────> our_outputs
└────────────────────> outputs (downgrade mirror)

wallet still reads ─────────> outputs

Make each existing chaintopology writer update both representations of
its UTXO state:

- wallet_add_utxo and wallet_add_onchaind_utxo insert into both tables;
- db_set_utxo updates reservations in both tables;
- wallet_confirm_tx updates confirmation heights in both tables; and
- wallet_outpoint_spend marks outputs spent in both tables.

Keeping each pair of writes in the existing helper makes the downgrade
mirror explicit without duplicating SQL at its callers. Reads remain on
legacy `outputs` until the following commit switches them to our_outputs.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: record bwatch-discovered wallet outputs

Add the wallet helpers and watch handlers that turn a bwatch
scriptpubkey match into our_txs and our_outputs rows. They validate
the matching output, notify invoice accounting, record confirmed
deposits, and install watches for later spends.

The watch_revert handler demotes the rows back to unconfirmed (the 0
sentinel) rather than deleting them: a row also carries state a
rediscovery cannot restore (reserved_til, onchaind close metadata),
and the still-armed watches re-promote it if the tx confirms again.
This matches the legacy tables, whose blocks(height) foreign keys
demote rows to NULL when the block row is removed.

Unlike got_utxo(), this path does not write transaction_annotations:
nothing reads per-transaction annotations anymore, so only the legacy
scanner keeps populating them (for downgrade, like the other legacy
tables).

The watchman dispatch entry is wired in the following commit. Keep this
path alongside got_utxo() and wallet_transaction_add(): the legacy
scanner must continue populating outputs and transactions for one release
so downgrades do not require a rescan.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: add our_outputs + our_txs schema migrations

The next commits move wallet UTXO and tx tracking off chaintopology and
onto bwatch. bwatch doesn't maintain a blocks table, but the legacy
utxoset, transactions and channeltxs tables all have FOREIGN KEY
references into blocks(height) (CASCADE / SET NULL), so we can't just
retarget the existing tables.

Instead, introduce parallel tables (our_outputs, our_txs) without the
blocks(height) FK. The new bwatch-driven code writes only to these,
the legacy tables stay populated by the existing code path during this
release so downgrade still works, and a future release can drop them
once we're past the downgrade window.

Losing the FK also changes what NULL means. In the legacy tables a
NULL blockheight was never written by hand: the ON DELETE SET NULL
trigger produced it when a reorg deleted the block row. These tables
have no such trigger, so unconfirmed is stored as blockheight 0
(NOT NULL) instead, for three reasons:

- everything feeding these tables already speaks u32-with-0: watchman
notifications carry blockheight as a required JSON number, and
wallet_transaction_height() has always returned 0 for unconfirmed,
so values bind straight through without a NULL/non-NULL branch at
every read and write site;

- integer comparisons keep working: the unconfirmed->confirmed
promotion is a single "WHERE blockheight < ?" (0 sorts below any
real height) and reorg rollback is "SET blockheight = 0 WHERE
blockheight >= ?", where a NULL row would match neither;

- it removes the footgun the legacy code warned about ("Note:
blockheight=NULL is not the same as is NULL!"), where lookups had
to branch between "= ?" and "IS NULL".

The same logic gives txindex 0 = unconfirmed/unknown (a *confirmed*
txindex of 0 means coinbase, which blockheight disambiguates) and
reserved_til 0 = not reserved. NULL survives only where 0 is a real
value or genuinely ambiguous: spendheight (NULL = unspent),
channel_dbid, commitment_point.

Schema only here — wallet handlers that write into these tables and the
backfill from outputs/transactions land in subsequent commits.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: mirror bwatch writes into legacy outputs table

The bwatch path writes wallet UTXOs to our_outputs while the legacy
`outputs` table is what a downgraded binary reads. Mirror every
our_outputs write (insert, spend, unspend-on-spend-revert, reservation)
into `outputs` so a downgrade for one release needs no copy-back or
rescan. Reorg demotion needs no mirror: the legacy rows are demoted by
their blocks(height) foreign keys when chaintopology removes the block.
The mirroring stops in the release that removes chaintopology, freezing
all the legacy tables at the same height.

The spend mirror guards its spend_height the same way the insert guards
confirmation_height: outputs.spend_height has a foreign key on
blocks(height), which only chaintopology populates, and bwatch can
deliver the spend before chaintopology has processed that block.
Record NULL in that case (status already marks the row spent) and let
chaintopology's own spend pass fill in the height.

The wallet still reads from `outputs`; switching reads over to
our_outputs comes next.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: Update the max_index cache when issuing new addresses

The write-through cache added for the address max_index was only kept in
sync in wallet_can_spend(). wallet_get_newindex() still read and wrote
the bip32_max_index/bip86_max_index db vars directly, leaving the
in-memory cache stale after every newaddr. Since listaddresses and the
rescan window now read from that cache, freshly issued addresses were
invisible until the next restart (and wallet_can_spend could even write a
stale, lower value back to the db).

Route all reads and writes through wallet_max_addr_index() and
wallet_set_max_addr_index() so the cache and the db never diverge.

Changelog-None
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

wallet: Cache the max_index for bip32 and bip86 address indices

We were looking up the index for the addresses to scan, twice, for
every single output, for every single TX for every block. No wonder
that was being slow.

Changelog-Fixed: wallet: Addressed a performance regression during blockchain scans

lightningd: store raw failure message so waitsendpay always has raw_message

If a payment's HTLC failure completes before waitsendpay is called,
wait_payment() rebuilds the error from the database, which did not
persist the raw BOLT4 failure message: a 2019 FIXME in that path set
fail->msg = NULL, silently dropping raw_message from the error data.
This is the cause of the test_error_returns_blockheight CI flake -
the test loses the race occasionally, calls waitsendpay after the
failure has landed, and KeyErrors on the missing raw_message.

Add a failmsg column to the payments table (with a downgrade drop),
persist fail->msg when recording the failure, and read it back in
wait_payment. Local and self-payment failures store NULL as before,
since no onion failure message exists for them; failed payments
recorded before this migration also return NULL, matching the old
behavior.

The flaky test now also calls waitsendpay a second time, which
deterministically takes the database-replay path, so the regression
is covered without any timing dependence.

Changelog-Fixed: JSON-RPC: `waitsendpay` error data now includes `raw_message` even when the payment already failed before the command was called.
Fixes: #9341

autogenerate-rpc-examples.py: disable entropy for generation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

autogenerate-rpc-examples.py: pin the full sorted UTXO set for multiwithdraw examples

doc: generate all examples with newchain

common: add randbytes() wrapper to override cryptographic entropy: $CLN_DEV_ENTROPY_SEED

Only in developer mode, ofc.

Notes:
1. We have to move the initialization before the lightningd main trace_start,
since that uses pseudorand().
2. To make the results stable, we need to use per-caller values to randbytes().
Otherwise external timing changes the call order.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

common: don't consume deterministic randbytes stream for trace ids

Trace/span ids now come from a plain counter when CLN_DEV_ENTROPY_SEED is active, so tracing (which CI's HAVE_USDT=1 build enables by default) no longer shifts the deterministic RNG stream that PSBT serial ids and locktime fuzz are drawn from.

common: don't trigger siphash_seed() init from span hashing either.

Previous commit stopped trace ids consuming the deterministic
randbytes stream, but tracing still touched it once: span_key_hash()
calls siphash_seed(), whose lazy init draws 16 bytes from the stream.
Since randbytes call sites are assigned their offset blocks in
first-use order, that single draw shifts every later draw in the
process by 1000 when tracing is active (HAVE_USDT builds, as in CI):
keysend preimages and invoicerequest payer keys then differ from a
build with tracing disabled, showing up as an 11-file doc-examples
diff.

Use a cheap multiplicative hash for span keys when the deterministic
override is active, so tracing touches the RNG stream zero times.

common: seed deterministic RNG from basename of argv0, not full path

dev_override_randbytes() hashes argv0 into the CLN_DEV_ENTROPY_SEED
stream, so every plugin and subdaemon gets a distinct seed. But
lightningd execs them with absolute paths, so the "deterministic"
stream silently depended on where the source tree was checked out:
canned blocks generated in one directory would not replay in another
(funding tx output order and nlocktime fuzz diverge, so txids change).
This is why check-doc-examples passed locally but failed in CI.

Hash only the basename: binaries still get distinct seeds, but the
stream no longer depends on the checkout path.

global: replace randombytes_buf() with randbytes() wrapper.

This allows us to override it for deterministic results.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

ci: reduce sanitizer/valgrind worker count to avoid runner OOM

The ASan/UBSan and Valgrind shards intermittently have their hosted
runner killed mid-run with "The runner has received a shutdown signal",
surfaced as a mass test cancellation. The VM itself is being
terminated, most likely from memory exhaustion: `-n $(nproc)+1` runs 5
parallel node-clusters and ASan roughly triples RSS, peaking past the
16GB hosted-runner ceiling on heavy shards (deterministic repro at ~10%
on ASan/UBSan 3/6 due to the fixed --test-group-random-seed=42).

Drop these two jobs to `-n $(nproc)` (one fewer worker) to lower peak
memory, and add an `if: failure()` step that dumps OOM-related dmesg and
kernel-log lines so the next failure records the actual cause.

Changelog-None

BOLTs: update announcement_signatures and next_funding reconnect quotes

Changelog-None

wire/splice: rename messages and TLV fields per updated BOLTs

Rename splice→splice_init, relative_satoshis→funding_contribution_satoshis,
txsigs_tlvs→tx_signatures_tlvs, and splice_info/batch_info→funding_txid/message_type.
Consolidate three CLN-specific shim patches into extracted_peer_cln_batch_element.patch.

Changelog-None

Update outdated BOLT quotes
common/bolt11.c: Some BOLT quotes had changed slightly.
run-bolt11.c: A test was updated to reflect nomenclature changes (from `signature recovery` to `public-key recovery`).