Core Lightning

wallet: handle bwatch wallet/utxo spend notifications

When bwatch_got_utxo() records a wallet output, it arms an outpoint
watch owned by wallet/utxo/<txid>:<outnum>. This adds the receiving
end of that watch: the dispatch entry plus the found/revert handlers.

On watch_found (a tx consumed the outpoint): mark the output spent in
our_outputs, store the spending tx in our_txs, and emit a withdrawal
coin movement. On watch_revert (a reorg removed that tx): clear
spendheight so the UTXO is spendable again.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: handle reorgs for our_outputs/our_txs by hand

The legacy tables rely on their blocks(height) ON DELETE SET NULL
foreign keys to mark rows unconfirmed/unspent when a block is reorged
out. our_outputs/our_txs deliberately carry no blocks FK (bwatch does
not maintain a blocks table), so block disconnect and rollback must
demote their blockheight/spendheight fields explicitly.

Demote, never delete: a row also carries state the chain cannot
re-deliver (reserved_til, onchaind close metadata), and this path runs
not just on real reorgs but on every startup, when chaintopology
invalidates its rescan window. Rediscovery re-promotes the row via
wallet_add_our_output / wallet_transaction_add; a tx that never
re-confirms just stays unconfirmed and unspendable.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: read UTXO state from our_outputs

With every writer dual-writing since the previous two commits, flip
the readers: all UTXO queries (listfunds, coin selection, reservations,
onchaind close info) now come from our_outputs, and the legacy-only
read helpers (wallet_stmt2output, gather_utxos, db_get_unspent_utxos)
are deleted.

our_output_row_to_utxo replaces wallet_stmt2output, using channel_dbid
to discriminate HD outputs from channel-close outputs, and
wallet_get_spendable_utxos centralizes the unspent+unreserved query
that wallet_find_utxo and wallet_has_funds previously duplicated.

migrate_setup_coinmoves keeps reading the legacy table directly: that
migration runs at v25.09, before our_outputs exists.

Co-authored-by: Cursor <cursoragent@cursor.com>

tests: query our_outputs in test_connection.py

The wallet's live UTXO state now lives in our_outputs; point the raw-SQL
assertions at it (spent means spendheight IS NOT NULL, reserved means
reserved_til > 0).

Co-authored-by: Cursor <cursoragent@cursor.com>

tests: query our_outputs in test_misc.py

Same raw-SQL switch as test_connection.py: read the wallet's UTXO state
from our_outputs.

Co-authored-by: Cursor <cursoragent@cursor.com>

lightningd: only run watchman under --experimental-bwatch

The flag is registered by the bwatch plugin, not lightningd, so peek
at the parsed configvars. Without it ld->watchman stays NULL and the
watchman_* entry points are no-ops, leaving chain_topology as the only
chain watcher: bwatch and the legacy path must not race each other.

The bwatch pytests opt in explicitly (with rescan=0) instead of
enabling bwatch globally.

Changelog-Experimental: wallet: Add bwatch-driven wallet transaction and UTXO tracking behind --experimental-bwatch.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: rename b32script to scriptpubkey in PSBT change paths

Change addresses are no longer bech32-only (p2tr is the default form),
so the old name was misleading. Pure rename, no functional change;
also drop the unused txfilter.h include.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: make datastore helpers self-wrap a wallet transaction

The four wallet_datastore_{get,create,update,remove} helpers used to
require the caller to be inside a wallet transaction; otherwise the
underlying db_prepare_v2 fatals at db/utils.c:103 with "Attempting to
prepare a db_stmt outside of a transaction".

watchman persists its pending bwatch ops through these helpers from
plugin callbacks that run outside any transaction, so wrap one on
demand.

Co-authored-by: Cursor <cursoragent@cursor.com>

tests: update test_wallet.py raw-SQL queries for our_outputs

The wallet no longer writes UTXO state to the legacy outputs table's
status/spend_height columns, so tests that peek at the database
directly must read our_outputs instead.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: register scriptpubkey watches at startup

init_wallet_scriptpubkey_watches walks every HD key (BIP32 + BIP86) up
to {bip32,bip86}_max_index + keyscan_gap and arms a watch for each
form, so bwatch can report deposits from the very first block it
scans. wallet_get_newindex does the same for fresh keys, keeping
coverage as the wallet grows.

The per-UTXO watch on unconfirmed change is now redundant (the
perennial per-key watch already covers that scriptpubkey), so drop it.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: dual-write chaintopology UTXO changes into our_outputs

Before this commit:

chaintopology ──────────────> outputs
bwatch ─────────────────────> our_outputs
└────────────────────> outputs (downgrade mirror)

wallet reads ───────────────> outputs

After this commit:

chaintopology ──────────────> our_outputs
└─────────────> outputs (downgrade mirror)

bwatch ─────────────────────> our_outputs
└────────────────────> outputs (downgrade mirror)

wallet still reads ─────────> outputs

Make each existing chaintopology writer update both representations of
its UTXO state:

- wallet_add_utxo and wallet_add_onchaind_utxo insert into both tables;
- db_set_utxo updates reservations in both tables;
- wallet_confirm_tx updates confirmation heights in both tables; and
- wallet_outpoint_spend marks outputs spent in both tables.

Keeping each pair of writes in the existing helper makes the downgrade
mirror explicit without duplicating SQL at its callers. Reads remain on
legacy `outputs` until the following commit switches them to our_outputs.

Co-authored-by: Cursor <cursoragent@cursor.com>

tests: add our_tables backfill upgrade test

Upgrade a pre-bwatch snapshot db and check the migration mirrored
outputs/transactions into our_outputs/our_txs, listfunds still reports
the old UTXOs, and the wallet can receive and withdraw new funds.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: record bwatch-discovered wallet outputs

Add the wallet helpers and watch handlers that turn a bwatch
scriptpubkey match into our_txs and our_outputs rows. They validate
the matching output, notify invoice accounting, record confirmed
deposits, and install watches for later spends.

The watch_revert handler demotes the rows back to unconfirmed (the 0
sentinel) rather than deleting them: a row also carries state a
rediscovery cannot restore (reserved_til, onchaind close metadata),
and the still-armed watches re-promote it if the tx confirms again.
This matches the legacy tables, whose blocks(height) foreign keys
demote rows to NULL when the block row is removed.

Unlike got_utxo(), this path does not write transaction_annotations:
nothing reads per-transaction annotations anymore, so only the legacy
scanner keeps populating them (for downgrade, like the other legacy
tables).

The watchman dispatch entry is wired in the following commit. Keep this
path alongside got_utxo() and wallet_transaction_add(): the legacy
scanner must continue populating outputs and transactions for one release
so downgrades do not require a rescan.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: route transaction reads/writes through our_txs

Now that the bwatch wallet path records every relevant transaction in
our_txs, point the wallet's transaction readers at that table.

Since our_txs.blockheight is NOT NULL with 0 = unconfirmed, the legacy
NULL handling disappears: wallet_transactions_by_height no longer needs
its "IS NULL vs = ?" query split, and wallet_transaction_height reads
the column unconditionally.

wallet_transaction_add keeps dual-writing the legacy transactions
table: the close path still inserts into channeltxs, whose
transaction_id foreign key points at transactions(id), and
wallet_get_funding_spend joins it. That legacy write can only go away
with channeltxs itself.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: add our_outputs + our_txs schema migrations

The next commits move wallet UTXO and tx tracking off chaintopology and
onto bwatch. bwatch doesn't maintain a blocks table, but the legacy
utxoset, transactions and channeltxs tables all have FOREIGN KEY
references into blocks(height) (CASCADE / SET NULL), so we can't just
retarget the existing tables.

Instead, introduce parallel tables (our_outputs, our_txs) without the
blocks(height) FK. The new bwatch-driven code writes only to these,
the legacy tables stay populated by the existing code path during this
release so downgrade still works, and a future release can drop them
once we're past the downgrade window.

Losing the FK also changes what NULL means. In the legacy tables a
NULL blockheight was never written by hand: the ON DELETE SET NULL
trigger produced it when a reorg deleted the block row. These tables
have no such trigger, so unconfirmed is stored as blockheight 0
(NOT NULL) instead, for three reasons:

- everything feeding these tables already speaks u32-with-0: watchman
notifications carry blockheight as a required JSON number, and
wallet_transaction_height() has always returned 0 for unconfirmed,
so values bind straight through without a NULL/non-NULL branch at
every read and write site;

- integer comparisons keep working: the unconfirmed->confirmed
promotion is a single "WHERE blockheight < ?" (0 sorts below any
real height) and reorg rollback is "SET blockheight = 0 WHERE
blockheight >= ?", where a NULL row would match neither;

- it removes the footgun the legacy code warned about ("Note:
blockheight=NULL is not the same as is NULL!"), where lookups had
to branch between "= ?" and "IS NULL".

The same logic gives txindex 0 = unconfirmed/unknown (a *confirmed*
txindex of 0 means coinbase, which blockheight disambiguates) and
reserved_til 0 = not reserved. NULL survives only where 0 is a real
value or genuinely ambiguous: spendheight (NULL = unspent),
channel_dbid, commitment_point.

Schema only here — wallet handlers that write into these tables and the
backfill from outputs/transactions land in subsequent commits.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: add migrate_backfill_bwatch_tables

Populate our_outputs/our_txs from the legacy outputs/transactions
tables so the bwatch wallet path starts with the same state as the
existing wallet.

Use outputs rather than utxoset because it already contains only
wallet-owned rows and carries wallet metadata like reservations and
channel-close info.

Downgrade only drops the new tables: later commits keep mirroring
writes into the legacy tables, so no copy-back migration is needed.

Co-authored-by: Cursor <cursoragent@cursor.com>

wallet: mirror bwatch writes into legacy outputs table

The bwatch path writes wallet UTXOs to our_outputs while the legacy
`outputs` table is what a downgraded binary reads. Mirror every
our_outputs write (insert, spend, unspend-on-spend-revert, reservation)
into `outputs` so a downgrade for one release needs no copy-back or
rescan. Reorg demotion needs no mirror: the legacy rows are demoted by
their blocks(height) foreign keys when chaintopology removes the block.
The mirroring stops in the release that removes chaintopology, freezing
all the legacy tables at the same height.

The spend mirror guards its spend_height the same way the insert guards
confirmation_height: outputs.spend_height has a foreign key on
blocks(height), which only chaintopology populates, and bwatch can
deliver the spend before chaintopology has processed that block.
Record NULL in that case (status already marks the row spent) and let
chaintopology's own spend pass fill in the height.

The wallet still reads from `outputs`; switching reads over to
our_outputs comes next.

Co-authored-by: Cursor <cursoragent@cursor.com>

pyln-testing: wait for listen port release before starting a node

A node's connectd is a separate process holding the listen socket,
and on shutdown it exits on its own schedule after lightningd itself
is gone -- under valgrind, its teardown can lag by tens of seconds.
If a test restarts the node in that window, the new connectd fails
with 'Address already in use', lightningd exits, and the test times
out waiting for 'Server started with public key'. Seen in CI in
test_emergencyrecoverpenaltytxn, where the port was still held 37
seconds after the old connectd began shutting down.

The filesystem port locks don't cover this: they keep other workers
from reserving the port, but the restarting node owns its reservation
both times; nothing waits for the old connectd to actually release
the socket.

Before launching lightningd, bind-probe the node's port (with
SO_REUSEADDR, matching connectd, so TIME_WAIT sockets don't count as
in use) and only proceed once the bind succeeds.

Fixes: #9354
Changelog-None

plugins: defer spenderp awaiting-channel recovery

plugins: recover awaiting channels after init

wallet: Read the max_index for addresses from the in-memory cache

wallet: Update the max_index cache when issuing new addresses

The write-through cache added for the address max_index was only kept in
sync in wallet_can_spend(). wallet_get_newindex() still read and wrote
the bip32_max_index/bip86_max_index db vars directly, leaving the
in-memory cache stale after every newaddr. Since listaddresses and the
rescan window now read from that cache, freshly issued addresses were
invisible until the next restart (and wallet_can_spend could even write a
stale, lower value back to the db).

Route all reads and writes through wallet_max_addr_index() and
wallet_set_max_addr_index() so the cache and the db never diverge.

Changelog-None
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

wallet: Cache the max_index for bip32 and bip86 address indices

We were looking up the index for the addresses to scan, twice, for
every single output, for every single TX for every block. No wonder
that was being slow.

Changelog-Fixed: wallet: Addressed a performance regression during blockchain scans

docs: Fixing Ubuntu installation docs
- updated release tag references
- removed protobuf-compiler dependency
- removed rustup default installation, changed it to only for Ubuntu <25.10 (rust <1.85)
- added note about tzdata config halting execution

CI: set readme branch name to stable

Changelog-None