Eclair

Add previous commitments to ChannelFundingConfirmed event (#3303)

Event stream subscribers could use this field to find extra information about previous states that could have been pruned from the current commitments.

Add liquidity ads metrics (#3301)

We add metrics for liquidity ads (purchased amount and mining fee
refund).

We refactor the interactive-tx metrics to record them *after* tx
validation, instead of *before*. We also remove the input/output
count for the "shared" case, since it's always 0 or 1.

Add metrics on interactive-tx inputs and outputs (#3300)

We track:
- global input/output count by type and distribution of amounts
- per-session input/output count by type

Fix flaky DER signature weight test (#3299)

This test was flaky because for the funding output, we provide two
signatures, which means that we need twice the slack for the variable
DER encoding length.

Use official feature bit for `option_simple_taproot` (#3144)

Use the official feature bit and name for taproot channels and the
corresponding channel types. Activate taproot channels support by
default (without support for announcing such channels yet).

See https://github.com/lightning/bolts/pull/995

Add support for the official splicing protocol (#2887)

We replace our experimental version of `splice_init`, `splice_ack` and
`splice_locked` by their official version. If our peer is using the
experimental feature bit, we convert our outgoing messages to use the
experimental encoding and incoming messages to the official messages.

We also change the TLV fields added to `tx_add_input`, `tx_signatures`
and `splice_locked` to match the spec version. We always write both the
official and experimental TLV to updated nodes (because the experimental
one is odd and will be ignored) but we drop the official TLV if our
peer is using the experimental feature, because it won't understand the
even TLV field.

We do the same thing for the `commit_sig` TLV. For peers who support the
official splicing version, we insert the `start_batch` message before
the batch of `commit_sig` messages.

This guarantees backwards-compatibility with peers who only support the
experimental feature.

We introduce a `retransmit_flags` field to `my_current_funding_locked`
and `next_funding` to ask our peer to retransmit `commitment_signed` or
`announcement_signatures` if we're expecting them. With this change, we
don't need to retransmit `splice_locked` on reconnection anymore to
trigger the exchange of `announcement_signatures`. We don't need to
retransmit it to let our peer know that we've seen enough confirmations
for the splice either, since `my_current_funding_locked` implies that.

This allows us to completely remove retransmission of `splice_locked` on
reconnection, and also get rid of the `your_last_funding_locked` TLV,
which greatly simplifies the reconnection logic. We do keep them for
backwards-compatibility with existing Phoenix users though, but we'll
be able to clean it up once they have updated.

Note that this works with taproot channels since we will simply provide
nonces in `channel_reestablish` when we need our peer to send
announcement signatures (not supported yet since taproot channels are
never announced).

We rollback using the `next_commitment_number` to let our peer know that
we haven't received their `commit_sig` and instead use the retransmit
flags added to the `next_funding` TLV, unless our peer is using the
legacy splicing protocol.

More aggressive peer scorer idle channels management (#3295)

We previously lowered fees from idle channels only if our latest update
was from more than 24 hours ago, and closed them if our latest update
was more than 5 days ago and using our minimal fees.

The issue is that our channel update may have been refreshed without
changing the relay fees, for example if our peer disconnected. We don't
track when our relay fees reached the minimum amount, so we may have
a recent channel update even though we reached our minimum relay fees
weeks ago.

We use smaller durations for those thresholds to see if that helps
reclaim liquidity more aggressively.

We also disable old peers revival by default.
This feature is useful when feerates have been high for a while and
funding was thus put on hold, but when feerates are frequently below
our feerate threshold, this is wasting liquidity. It should be enabled
after a high-fee period.

Store our closing_complete in the simple close session (#3289)

In the simple close session, which is never persisted, we just need to remember our closing_complete and
the partial signatures it contains, instead of keeping our musig2 nonces and generating our partial signatures a second time to build the final closing tx when we receive our peer's partial signatures.
We also check our peer's partial signature and fail early if they're not valid, instead of building a fully sign
closing tx and then verify that it is correct.

Increase start-up ZMQ timeout (#3294)

We connect to bitcoind over ZMQ right after spawning all actors in the
system, which can consume a lot of our available sockets/bandwidth. We
increase the default timeout from 5 seconds to 30 seconds.

Add threshold for disabling `from_future_htlc` (#3293)

When funding a transaction using `from_future_htlc`, we are taking the
risk that the remote node doesn't fulfill the corresponding HTLCs. If
we detect that too many peers fail such HTLCs, this may indicate a bug
on the wallet side, so we automatically disable `from_future_htlc` to
limit our exposure.

We also ensure that nodes that already had one suspicious payment are
blacklisted for other payments. This issue can happen when wallet users
use multiple devices in unsupported scenarios.

Add peer scorer documentation (#3277)

We add release notes about the peer scorer for node operators who like
living on the bleeding edge. We also remove the workaround for the lack
of DB data on restart.

We also add metrics to track rolling profit and payment volume.

Fix flaky zero-conf integration test (#3291)

There's a race condition between Bob and Carol's router states, which
aren't updated exactly at the same time. Retrying the condition should
fix this race condition.

Don't load network graph twice on start-up (#3290)

We previously read the whole network graph from DB (channels and nodes)
twice: once before starting any actor to verify that the DB has not been
corrupted, and then again when initializing the `Router` actor.

We now keep the result of the first DB read instead of throwing it away
and provide it as argument to the `Router`, which doesn't need to read
from the DB on start-up anymore.

Don't try reconnecting automatically to mobile wallets (#3287)

It's useless as they will always have a different IP address, and they
will be offline most of the time anyway. They will initiate connections
either when the app is started or when they receive a push notification
so we never need to initiate connections from the server side.

Add delays when reading past payment events for stats (#3288)

When we restart, we load past payments events into the peer stats
tracker to inform our scoring algorithm. We read them in chunks,
but it can be quite expensive in terms of bandwidth. Since this is
low priority and there are a lot of DB reads already happening when
starting up, we add a large initial delay and a small-ish chunk delay.

With the default values, all past events should be loaded ~20 minutes
after starting up, and the first run of the scoring algorithm happens
1 hour after starting up, so this should work well.

Use noble docker images instead of alpine (#3286)

It fixes compatibility issues with alpine (which uses musl and not glibc), and is easier to maintain.
Note that this makes the image about 190 Mb bigger.

Add fuzz tests for onion, route blinding and lightning message codecs (#3282)

- Expand LightningMessageCodecs fuzz tests to cover all message types
- Add fuzz tests for payment and message onion payload codecs
- Add fuzz tests for route blinding codecs
- Add fuzz tests for onion failure message codecs

Signed-off-by: Nishant Bansal <nishant.bansal.282003@gmail.com>

Fix flaky `PeerStatsTracker` test (#3280)

If the test runs too quickly and the message is processed in less than
1 millisecond, there is a race condition since reading from the database
excludes the upper timestamp bound (`timestamp < now`).

We also remove our usage of the `BETWEEN` keyword for postgres.
The `BETWEEN` keyword is inclusive of both bounds, while we want to
exclude the upper bound to allow pagination.

Add fee field to `findroute` full format response (#3283)

This can be helpful when analysis routes find by path-finding.

Allow reply-less pings (#3284)

BOLT1 says:

```
A node receiving a ping message:
- if num_pong_bytes is less than 65532:
- MUST respond by sending a pong message, with byteslen equal to num_pong_bytes.
- otherwise (num_pong_bytes is not less than 65532):
- MUST ignore the ping.
```

These reply-less pings are used to insert cover traffic to thwart
network-level traffic analysis. We wanted to allow them in #3278
but we missed this ping flood protection that needed to be updated.

We also had an off-by-one in the maximum pong length that
is now fixed.

Add backwards-compatible parts of the official splicing protocol (#3261)

* Add `funding_txid` to `commit_sig`

In https://github.com/lightning/bolts/pull/1160 we add a TLV field to
`commit_sig` messages to let the receiver know to which `funding_txid`
this signature applies. This is more resilient than relying on the
order of the `commit_sig` messages in the batch. This is an odd TLV,
so we can start writing it right now without creating compatibility
issues.

We also slightly refactor existing code to make it easier to introduce
a backwards-compat layer when migrating to the official splicing. We
also increase the default number of RBF attempts allowed.

* Insert a `start_batch` message during splices

In https://github.com/lightning/bolts/pull/1160, we introduce a message
to let our peer know how many `commit_sig` messages they will receive
and treat them as a batch. This replaces our previous version that did
something similar, but by adding a batch TLV in every `commit_sig`
message we send.

We currently do both: we keep inserting the experimental batch TLV, and
we start by sending a `start_batch` message (with the same information).
Since it is an odd message (127), it should be safely ignored if our
peer doesn't understand it.

Follow BOLT1 handling for "no reply" pings: ignore, don't warn. (#3278)

These "oversize ping replies" are how we ask for no replies, such as when
we're sending dummy traffic. From the rationale:

This allows a convenient cutoff for `num_pong_bytes` to indicate that no reply should be sent.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Co-authored-by: t-bast <bastien@acinq.fr>

Add fuzzing infrastructure (#3276)

Add a fuzzing module based on Jazzer to allow fuzzing parts of eclair.
This commit includes fuzz tests for codecs to be used as examples.
It is a separate module (minimal interaction with the other modules).
We include helper scripts to run many fuzz tests in parallel or generate
reports.

Initialize peer stats with past events (#3272)

We initialize peer statistics by reading past events from the `AuditDb`.
We read events in chunks to avoid overloading the DB or the RAM. We only
return peer statistics to callers once we've finished reading past data
from the DB.

Note that on-chain statistics need to be computed whenever statistics
are requested, because they require merging data from two distincts
events (transaction published and transaction confirmed), which requires
keeping state between those events and thus requires the DB (because the
node may restart after publishing a transaction and before it confirms).

Reclaim liquidity from idle channels (#3269)

We more aggressively reclaim liquidity from idle channels. We start by
identifying channels where the liquidity doesn't seem to be moving in
either direction. Then we lower our relay fees until we reach our
minimum configured value. Once we've reached that minimum fee, if the
payment volume doesn't increase, we eventually close those channels
(but we always keep one channel with those peers, we only close the
redundant channels).