rust-lightning

Add support for native async `KVStore` persist to `ChainMonitor`

This finally adds support for full native Rust `async` persistence
to `ChainMonitor`.

Way back when, before we had any other persistence, we added the
`Persist` trait to persist `ChannelMonitor`s. It eventualy grew
homegrown async persistence support via a simple immediate return
and callback upon completion. We later added a persistence trait
in `lightning-background-processor` to persist the few fields that
it needed to drive writes for. Over time, we found more places
where persistence was useful, and we eventually added a generic
`KVStore` trait.

In dc75436c673fad8b5b8ed8d5a9db1ac95650685a we removed the
`lightning-background-processor` `Persister` in favor of simply
using the native `KVStore` directly.

Here we continue that trend, building native `async`
`ChannelMonitor` persistence on top of our native `KVStore` rather
than hacking support for it into the `chain::Persist` trait.
Because `MonitorUpdatingPersister` already exists as a common way
to wrap a `KVStore` into a `ChannelMonitor` persister, we build
exclusively on that (though note that the "monitor updating" part
is now optional), utilizing its new async option as our native
async driver.

Thus, we end up with a `ChainMonitor::new_async_beta` which takes
a `MonitorUpdatingPersisterAsync` rather than a classic
`chain::Persist` and then operates the same as a normal
`ChainMonitor`.

While the requirement that users now use a
`MonitorUpdatingPersister` to wrap their `KVStore` before providing
it to `ChainMonitor` is somewhat awkward, as we move towards a
`KVStore`-only world it seems like `MonitorUpdatingPersister`
should eventually merge into `ChainMonitor`.

Use public `MonitorUpdatingPersister` API in tests

In the coming commits `MonitorUpdatingPersister`'s internal state
will be reworked. To avoid spurious test diff, we instead use the
public API of `MonitorUpdatingPersister` rather than internal bits
in tests.

Clean up and rustfmt `persist.rs`

Add a generic public `FutureSpawner` in LDK directly

In the next commit we'll use this to spawn async persistence
operations in the background, but for now we just move the
`lightning-block-sync` `FutureSpawner` into `lightning`.

Make `TestStore` async writes actually async, with manual complete

`TestStore` recently got the ability to make async writes, but
wasn't a very useful test as all writes actually completed
immediately. Instead, here, we make the writes actually-async,
forcing the test to mark writes complete as required.

Support `maximum_pending_updates` = 0 in `MonitorUpdatingPersister`

In the coming commits, we'll use the `MonitorUpdatingPersister` as
*the* way to do async monitor updating in the `ChainMonitor`.
However, to support folks who don't actually want a
`MonitorUpdatingPersister` in that case, we explicitly support them
setting `maximum_pending_updates` to 0, disabling all of the
update-writing behavior.

Migrate `MonitorUpdatingPersister` to an async + async-sync wrapper

As we've done with several other structs, this adds an async
variant of `MonitorUpdatingPersister` and adds an async-sync
wrapper for those using `KVStoreSync`. Unlike with other structs,
we leave `MonitorUpdatingPersister` as the sync variant and make
the new async logic a `MonitorUpdatingPersisterAsync` as the async
monitor updating flow is still considered beta.

This does not yet expose the async monitor updating logic anywhere,
as doing a standard `Persist` async variant would not work for
ensuring the `ChannelManager` and `ChainMonitor` don't block on
async writes or suddenly require a runtime.

Correct gossip forwarding criteria while doing background sync

If we're doing gossip backfill to a peer, we first forward all our
`channel_announcement`s and `channel_update`s in SCID-order. While
doing so, we don't forward any fresh `channel_update` messages for
any channels which we haven't yet backfilled (as we'll eventually
send the new update anyway, and it might get rejected without the
corresponding `channel_announcement`).

Sadly, our comparison for this was the wrong way, so we actually
*only* forwarded updates which were for channels we haven't yet
backfilled, and dropped updates for channels we already had
backfilled.

Add concrete `NO_*` consts to avoid users having to specify types

When building a background processor without an onion messenger or
liquidity manager, users still need to pass `None` into the
background processor initialization methods. Sadly, Rust will
require a concrete type for the (unused) `Some` variant of the
`Option` passed in, which requires specifying out a lot of type
gook.

Instead, provide some constants which users can pass in trivially.

Fixes #3612

Set HTLC tx version 3 on handling `BumpTransactionEvent::HTLCResolution`

Add forwards-compatibility logic in `LocalHTLCFailureReason` ser

We recently created a `LocalHTLCFailureReason` enum to cover
detailed reasons why an HTLC may have failed, in excess of the
on-the-wire failure encoding.

Sadly, when we did so we introduced a serialization format which
isn't particularly conducive to introducing new failure reasons in
the future. Luckily, `LocalHTLCFailureReason` already has logic
that works for forwards-compatibility - it can be converted to, and
from, the on-the-wire error codes.

Thus, here, we take advantage of that, writing both the on-the-wire
error code as well as a code for the specific case. If we don't
recognize the specific case on deserialization, we'll fall back to
the default case for the on-the-wire code we've written.

We also, of course, include a length-prefixed TLV stream to allow
for additional fields in the future.

Fixes #4087

Document some channel_reestablish spec requirements

Update commitment_signed retransmission for next_funding

The next_funding TLV is included in channel_reestablish when either
tx_signatures or commitment_signed is needed for an interactive-tx
session. This commit largely includes the spec requirements in the
comments, including when to retransmit commitment_signed. But it also
adds an additional check that tx_signatures has not yet been received.

Compute ConstructedTransaction weight from Transaction

Instead of defining a custom function for calculating a transaction's
weight, have ConstructedTransaction hold a Transaction so that its
weight method can be re-used. As a result NegotiatedTxInput no longer
needs to store the transaction inputs.

Remove NegotiatedTxInput::weight

The weight is no longer needed once a ConstructedTransaction is created,
so it doesn't need to be persisted.

Track satisfaction_weight in InteractiveTxInput

Instead of estimating the input weight, track the satisfaction_weight in
InteractiveTxInput. This then can be used compute the transaction weight
without storing individual weights in NegotiatedTxInput in an upcoming
commit.

Use FundingTxInput in InteractiveTxConstructorArgs

InteractiveTxConstructor::new converts inputs given in channel.rs to
InputOwned. Instead of converting FundingTxInput in channel.rs to the
type needed by that constructor, pass in FundingTxInput and have it
converted directly to InputOwned there. This saves one conversion and
Vec allocation. It also lets us use the satisfaction_weight in an
upcoming refactor.

Simplify test_trampoline_unblinded_receive

No need to construct unused blinded hop data or hardcode session privs/prng
seeds.

Compute trampoline session_priv from outer session_priv

This simplifies the code and makes it more straightforward to test unblinded
trampoline receives where we need to compute the trampoline session_priv when
manually creating the inner onion. (The trampoline onion needs to be manually
created because LDK does not natively support sending to unblinded trampolines,
just receiving.)

Debug-Assert that there are no queued messages when a peer connects

In the previous two commits we stopped enqueueing messages for
disconnected peers. Here we add some basic test coverage of these
changes by asserting that there are no queued messages for a peer
when they (re-)connect.

We also add a few assertions that a peer is connected when we push
a message onto the queu, in cases where we expect a peer to be
connected.

Stop enqueueing error messages for disconnected peers

If a channel is failed while a peer is disconnected, we'll still
have a `PeerState` for that peer. Historically, we haven't bothered
to check if a peer is actually connected before we push the `error`
message onto the `PeerState::pending_msg_events` queue, leaving us
sending messages into the void.

This is generally not an issue as
`ChannelManager::get_and_clear_pending_msg_events` should be called
very regularly, removing these messages and then dropping them as
`PeerManager` won't have anything to do with them. Further, when
the the message is an `error`, if a peer happens to connect between
when we push the message and when `get_and_clear_pending_msg_events`
is called the worst that happens is they get the `error` message
we'd end up sending them when they try to reestablish the channel
anyway.

Still, its awkward to leave the `error`s lying around in a message
queue for a disconnected peer, so we remove them here.

Stop enqueueing messages for disconnected peers

If a channel operation occurs while a peer is disconnected, we'll
still have a `PeerState` for that peer. Historically, we haven't
bothered to check if a peer is actually connected before we push
new messages onto the `PeerState::pending_msg_events` field,
leaving us sending messages into the void.

This is generally not an issue as
`ChannelManager::get_and_clear_pending_msg_events` should be called
very regularly, removing these messages and then dropping them as
`PeerManager` won't have anything to do with them.

Still, there is a race condition here - if a peer manages to
connect between the generation of one such message and when
`get_and_clear_pending_msg_events` is called, we may enqueue a
message to the peer which makes no sense and could lead to a
spurious channel closure (especially in the case of an async
`ChannelMonitorUpdate` completion or async signing operation, which
often lead to normal channel message generation).

Further, if a peer is slow to send their `channel_reestablish`
message after connection this race could be substantially more
likely, as such normal channel messages may be nonsense until we've
completed the reestablish dance (i.e. the later reestablish dance
may lead us to re-send the same messages again immediately).

Here we remove most of the cases where we enqueue messages for
disconnected peers.

Note that we differentiate between two different checks for
connected-ness - for cases where we're sending an `error` or gossip
messages, we allow the messages to be enqueued if the peer is
connected at all. For most other cases, we only allow messages to
be enqueued if the peer is connected *and* the channel has
completed its reestablish dance (if required, i.e. the channel is
"connected").

Correct `msgs_sent_since_pong` tracking for gossip forwards

When we forward gossip messages, we use the already-encoded copy
we have, pushing it onto the `gossip_broadcast_buffer`. When we
have free socket buffer space, and there are no non-gossip messages
pending, we'll remove it from the `gossip_broadcast_buffer` and
push it onto the normal `pending_outbound_buffer` socket queue.

Separately, we use `msgs_sent_since_pong` to ensure that our peer
is being responsive and our messages are getting through with
minimal latency. After we send 32 messages (if we've already
received the `pong` for the last `ping` wensent), we send an extra
`ping` and will only queue up an additional 32 forwarded gossip
messages before we start dropping forwarded gossip on the floor.

Previously, we (arguably) incorrectly incremented
`msgs_sent_since_pong` when we pushed a message onto the
`gossip_broadcast_buffer`, not when it was actually queued up in
our `pending_outbound_buffer` immediate socket queue. This is
fairly strange - the point of `msgs_sent_since_pong` is to keep
peer latency low, we already independently enforce memory limits to
ensure we drop forwarded gossip messages if a peer's message queues
have grown too large. This means we potentially disconnect a peer
for not draining the backlog forwarded-gossip socket queue fast
enough, rather than simply dropping gossip.

While this shouldn't be a big deal in practice, I do see (mostly
Tor) peers disconnect on my node occasionally, and while its likely
due to Tor circuits hanging and a disconnect is needed, the
incorrect tracking here makes state analysis difficult as there are
nearly always a flow of enough gossip messages to "send to the
peer". Its also possible that this allows incredibly low bandwidth
connections to stay afloat more durably.

fix: Onion v3 parse/format order and render lowercase

Tor v3 addresses are base32 of 35 bytes laid out as:
[32-byte ed25519 pubkey][2-byte checksum big-endian][1-byte version]

see: https://spec.torproject.org/rend-spec/encoding-onion-addresses.html

* Parsing now slices 0..32 = pubkey, 32..34 = checksum (BE), 34 = version.
* Display now appends pubkey + checksum + version and lowercases base32.
* Adds/updates test vector accordingly.

Fix lint CI job