Electrum

wallet: de-sloppify d2d4251c8 (paid invoice cache)

arghhhhhh

wallet: followup d2d4251c8 (paid invoice cache)

Followup to d2d4251c8 (paid invoice caching) in wallet.py.
* rename cache attribute to indicate it is a in-memory cache
* some minor reordering/simplifications
* also update on adb_removed_tx

add_tx_fee_from_server: check fee_sat type

qml: QEWallet lifecycle fix, remove QEWallet from QEWallet.__instances on destruction.

This never surfaced, as the mobile GUI currently has no option to unload/close a wallet.

constants: add basic sanity check for servers.json

Some users will fiddle with this file and if we don't sanity check the
error will pop out elsewhere and end up as a not immediately obvious
crash report on github

test_lnmsg: add bolt 12 merkle tree test vector

wallet: cache paid-invoice keys to avoid prevout rescan

On wallets with many outgoing invoices that share output scriptpubkeys,
broadcasting a new transaction froze the GUI for several seconds — up
to a minute on bigger wallets. The hot loops in set_broadcasting() and
_update_onchain_invoice_paid_detection() iterate every invoice touched
by the new tx's outputs, which is large when scriptpubkeys are shared
via _invoices_from_scriptpubkey_map. For each touched invoice they call
_is_onchain_invoice_paid(), which scans all of the invoice's output
scripthashes against _prevouts_by_scripthash and does a get_tx_height
per prevout. Both inner dimensions blow up together when invoices share
inputs/outputs across many past payments.

Add _paid_invoice_keys: Set[str], an in-memory cache of outgoing
invoice ids known to be PR_PAID, maintained incrementally:

- populated at wallet load by _prepare_onchain_invoice_paid_detection
- updated in _update_onchain_invoice_paid_detection (discard then
recompute, so reorgs still demote PAID->UNPAID)
- updated in save_invoice, delete_invoice, clear_invoices
- new on_event_invoice_status listener keeps it in sync with
LN-driven transitions from LNWallet.set_invoice_status

get_invoice_status() short-circuits to PR_PAID on cache hit, skipping
the prevout scan. set_broadcasting() skips already-paid invoices
entirely — broadcasting_status has no effect on a paid invoice and the
callback churn is what made the GUI freeze.

(trivial) move StorageEncryptionVersion and StorageReadWriteError to stored_dict.py

storage: add get_path method

lnchan: if funding tx is coinbase tx, wait for maturity

ref https://github.com/lightning/bolts/commit/17ab3f0dfef50ab98c326a56cb4d109e6f733ab8

Modify the 'stored_at' syntax, so that it includes the full path

This may prevent collisions. More importantly, it also makes
it possible to cache the subdict of 'registered_names' that
is relevant to a given StoredDict, so that we do not need to
walk the path on every conversion. (performance optimization,
not part of this commit)

This is a backport from the levelDB branch

simplify prev

tx_from_any: make sanitization optional

re.sub is expensive (calls re.sub), it is not necessary
to call it when instantiating transactions from the wallet file.

unit tests: do not mock save_db

Create in-memory wallets instead.

The logic of calling save_db() at specific places is specific to
the current JsonDB backend, and it does not generalize well.

tests: move create_test_channels to lnhelpers.py

test_lnwallet: unittest channel rebalancing

Add some unittest coverage for the channel rebalance flow.

Co-authored-by: SomberNight <somber.night@protonmail.com>

lnwatcher: introduce loop to trigger callbacks

Introduce a taskgroup and polling loop to LNWatcher to guarantee
the callbacks get called at least once every
LNWatcher.MAX_CALLBACK_TRIGGER_DELAY_SEC (10 min).
This should prevent callbacks that operate on time instead of
blockheight from becoming (very) stale if there are no blockchain
events triggering the callbacks for a longer time.
Not entirely set about the 10 min delay, might as well be 5/2/1 min?

new 'stored_at' syntax

backported from levelDB branch

build: support blacklisting dependencies for deterministic builds

The method works as follows:
1. for every blacklisted dependency, as listed in ghost.txt,
create and install an empty ghost package which will satisfy
the dependency resolver
2. before hash resolution step, remove those ghosts to make hashin happy

This required converting find_restricted_dependencies to use locally
installed package metadata instead of looking it up online on pypi. But
that seems to be a good idea anyway.

Move StoredDict class into new 'stored_dict' module

This commit is only moving code, no functional change

wallet: encrypt the keystore before adding it to db

adding keystore to db may result in a disk write, and we should
never write unencrypted keys to disk (except lightning_xprv)

qewallet: replace some threads with coroutines

Allows to simplify a bit and has less overhead.

wallet: make_unsigned_transaction: nicer error msg for bad outputs

traceback was hard to understand if outputs was a list of TxOutputs
(as opposed to PartialTxOutput)

tests: lnpeer: simplify anchors more

tests: lnpeer: simplify anchors

there were too many degrees of freedom - sometimes the fields were actually inconsistent