Eclair

Fix flaky zero-conf integration test (#3291)

There's a race condition between Bob and Carol's router states, which
aren't updated exactly at the same time. Retrying the condition should
fix this race condition.

Don't load network graph twice on start-up (#3290)

We previously read the whole network graph from DB (channels and nodes)
twice: once before starting any actor to verify that the DB has not been
corrupted, and then again when initializing the `Router` actor.

We now keep the result of the first DB read instead of throwing it away
and provide it as argument to the `Router`, which doesn't need to read
from the DB on start-up anymore.

Don't try reconnecting automatically to mobile wallets (#3287)

It's useless as they will always have a different IP address, and they
will be offline most of the time anyway. They will initiate connections
either when the app is started or when they receive a push notification
so we never need to initiate connections from the server side.

Add delays when reading past payment events for stats (#3288)

When we restart, we load past payments events into the peer stats
tracker to inform our scoring algorithm. We read them in chunks,
but it can be quite expensive in terms of bandwidth. Since this is
low priority and there are a lot of DB reads already happening when
starting up, we add a large initial delay and a small-ish chunk delay.

With the default values, all past events should be loaded ~20 minutes
after starting up, and the first run of the scoring algorithm happens
1 hour after starting up, so this should work well.

Use noble docker images instead of alpine (#3286)

It fixes compatibility issues with alpine (which uses musl and not glibc), and is easier to maintain.
Note that this makes the image about 190 Mb bigger.

Add fuzz tests for onion, route blinding and lightning message codecs (#3282)

- Expand LightningMessageCodecs fuzz tests to cover all message types
- Add fuzz tests for payment and message onion payload codecs
- Add fuzz tests for route blinding codecs
- Add fuzz tests for onion failure message codecs

Signed-off-by: Nishant Bansal <nishant.bansal.282003@gmail.com>

Fix flaky `PeerStatsTracker` test (#3280)

If the test runs too quickly and the message is processed in less than
1 millisecond, there is a race condition since reading from the database
excludes the upper timestamp bound (`timestamp < now`).

We also remove our usage of the `BETWEEN` keyword for postgres.
The `BETWEEN` keyword is inclusive of both bounds, while we want to
exclude the upper bound to allow pagination.

Add fee field to `findroute` full format response (#3283)

This can be helpful when analysis routes find by path-finding.

Allow reply-less pings (#3284)

BOLT1 says:

```
A node receiving a ping message:
- if num_pong_bytes is less than 65532:
- MUST respond by sending a pong message, with byteslen equal to num_pong_bytes.
- otherwise (num_pong_bytes is not less than 65532):
- MUST ignore the ping.
```

These reply-less pings are used to insert cover traffic to thwart
network-level traffic analysis. We wanted to allow them in #3278
but we missed this ping flood protection that needed to be updated.

We also had an off-by-one in the maximum pong length that
is now fixed.

Add backwards-compatible parts of the official splicing protocol (#3261)

* Add `funding_txid` to `commit_sig`

In https://github.com/lightning/bolts/pull/1160 we add a TLV field to
`commit_sig` messages to let the receiver know to which `funding_txid`
this signature applies. This is more resilient than relying on the
order of the `commit_sig` messages in the batch. This is an odd TLV,
so we can start writing it right now without creating compatibility
issues.

We also slightly refactor existing code to make it easier to introduce
a backwards-compat layer when migrating to the official splicing. We
also increase the default number of RBF attempts allowed.

* Insert a `start_batch` message during splices

In https://github.com/lightning/bolts/pull/1160, we introduce a message
to let our peer know how many `commit_sig` messages they will receive
and treat them as a batch. This replaces our previous version that did
something similar, but by adding a batch TLV in every `commit_sig`
message we send.

We currently do both: we keep inserting the experimental batch TLV, and
we start by sending a `start_batch` message (with the same information).
Since it is an odd message (127), it should be safely ignored if our
peer doesn't understand it.

Follow BOLT1 handling for "no reply" pings: ignore, don't warn. (#3278)

These "oversize ping replies" are how we ask for no replies, such as when
we're sending dummy traffic. From the rationale:

This allows a convenient cutoff for `num_pong_bytes` to indicate that no reply should be sent.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Co-authored-by: t-bast <bastien@acinq.fr>

Add fuzzing infrastructure (#3276)

Add a fuzzing module based on Jazzer to allow fuzzing parts of eclair.
This commit includes fuzz tests for codecs to be used as examples.
It is a separate module (minimal interaction with the other modules).
We include helper scripts to run many fuzz tests in parallel or generate
reports.

Initialize peer stats with past events (#3272)

We initialize peer statistics by reading past events from the `AuditDb`.
We read events in chunks to avoid overloading the DB or the RAM. We only
return peer statistics to callers once we've finished reading past data
from the DB.

Note that on-chain statistics need to be computed whenever statistics
are requested, because they require merging data from two distincts
events (transaction published and transaction confirmed), which requires
keeping state between those events and thus requires the DB (because the
node may restart after publishing a transaction and before it confirms).

Reclaim liquidity from idle channels (#3269)

We more aggressively reclaim liquidity from idle channels. We start by
identifying channels where the liquidity doesn't seem to be moving in
either direction. Then we lower our relay fees until we reach our
minimum configured value. Once we've reached that minimum fee, if the
payment volume doesn't increase, we eventually close those channels
(but we always keep one channel with those peers, we only close the
redundant channels).

Use bitcoin-lib 0.47 (#3268)

* Use bitcoin-lib 0.47

* Add checksums for secp256k1-kmp 0.23.0

* der2compact() and compact2der() methods have been removed, new signInputCompact() method has been added

* Use Transaction.encodeWitnessEcdsaSig() instead of low-level calls

* Replace local der() methode with bitcoin-lib's encodeWitnessEcdsaSig()

Improve data stored in `AuditDb` (#3245)

* Improve `channel_events` in the `AuditDb`

We improve the `channel_events` table in the `AuditDb` by:

- adding the `funding_txid` from each event
- adding the `channel_type` from each event
- using hex instead of blobs in sqlite
- providing APIs to list these events
- adding indexes on `channel_id` and `remote_node_id`

The added data isn't available in past events. We decide to simply
rename the older tables and start fresh: previous data will not be
available from the API, but can still be queried directly in SQL if
necessary.

* Improve transaction events in the `AuditDb`

We improve the `transactions_published` and `transactions_confirmed`
tables in the `AuditDb` by:

- adding more details about mining fees and feerates
- adding input and output counts
- using hex instead of blobs in sqlite
- providing APIs to list transactions by `remote_node_id`
- adding indexes for listing APIs

The added data isn't available in past events. We decide to simply
rename the older tables and start fresh: previous data will not be
available from the API, but can still be queried directly in SQL if
necessary.

* Improve payment sent and received events in the `AuditDb`

We improve the `sent` and `received` payment tables in the `AuditDb` by:

- adding `start_at` / `settled_at` timestamps to measure duration
- adding `remote_node_id` for every channel used
- using hex instead of blobs in sqlite
- reordering columns and renaming them wherever useful

The added data isn't available in past events. We decide to simply
rename the older tables and start fresh: previous data will not be
available from the API, but can still be queried directly in SQL if
necessary.

* Improve payment relayed events in the `AuditDb`

We improve the `relayed` and `relayed_trampoline` payment tables in the
`AuditDb` by:

- unifying the way we handle multi-part payments, by having one row per
payment, whether incoming or outgoing, and reconciling based on type
and direction when listing events
- adding `remote_node_id` for every channel used
- using hex instead of blobs in sqlite
- renaming columns for better clarify

The added data isn't available in past events. We decide to simply
rename the older tables and start fresh: previous data will not be
available from the API, but can still be queried directly in SQL if
necessary.

* Improve relay statistics computation

We change the relay statistics we compute, to more accurately reflect
nodes that are good or bad peers and inform liquidity allocation
decisions.

* Add liquidity purchase fees in transaction events

We now track liquidity purchases as well in transaction events, as their
fees must be taken into account when evaluating the overall fees earned
from a node.

We also add the number of on-chain transactions made with a given peer
to its relay stats.

Close connection when receiving malformed messages (#3273)

When receiving a message that we're supposed to be able to decode, for
which decoding fails, we must send a warning and close the connection.
This ensures that the connection state cannot become corrupted, which
may otherwise happen if we fail to deserialize a `commit_sig` message
(for example) and simply ignored it while continuing to process the
next messages.

Fixes #3271

Update Bitcoin Core to v30.2 (#3274)

This release contains improvements for v3 transactions, which we want
for #3192.

Add per-peer profit scoring (#3247)

We create a new set of actors that keep track of payment statistics
across our peers and rank them to identify the top profit earners.
Based on those statistics, the actors issue recommendations to:

- allocate more liquidity towards nodes that are generating revenue and
may run out of liquidity in the next few days
- reclaim liquidity from inactive channels
- change our relay fees to optimize increases or decreases in outgoing
flow and volume

Fix race condition in `Postman` causing flaky `OfferPayment` tests (#3270)

The integration test "send blinded multi-part payment a->b->c (single
channel a->b)" in `OfferPaymentSpec` fails intermittently. The root
cause is a race condition in `Postman` where the subscription for an
onion message reply is registered after the message is sent, allowing
the reply to arrive and be silently dropped before the subscription
exists.

In integration tests where all 3 nodes run on the same JVM, the onion
message round-trip (Alice -> Bob -> Carol -> creates invoice -> Carol
-> Bob -> Alice) can complete in just a few milliseconds - fast enough
to beat the `Subscribe` message to the `Postman`'s mailbox.

This explains why the test is flaky: it usually works (round-trip slower
than subscribe), but occasionally fails (round-trip faster than
subscribe, reply dropped).

We simply move the subscription registration to *before* the message is
sent to the network.

Add bitcoin rpc call to check if an address belongs to our wallet (#3267)

We use the "ismine" property that is returned by the "getaddressinfo" RPC call.

Improve support for plugin-defined features (#3264)

We allow injecting non-official features via plugins, by extending the
`CustomFeaturePlugin` trait. However, we previously added those features
to a set of unknown features: we can do better and add them to the known
features map, which provides pretty-printing in logs and simplifies
debugging.

We also had an issue where `hasFeature` would always return `false` for
plugin features, even when they were supported by both nodes, because we
only looked at our `activated` features map which only contains official
features.

We get rid of the `UnknownFeature` class entirely and keep the encoded
feature bits after deserializing them. We take those feature bits into
account in `hasFeature`, which fixes the issue.

This creates a slightly weird asymmetry because our local features,
which are created based on our `eclair.conf`, will not have this field
set while features that are read from network messages or the DB will
have this field set. A better architecture would be to remove the
`activated` map entirely and directly wrap the feature bits, while
providing helper functions to interact with features without having
to directly read a `BitVector`. This is a larger refactoring though,
because we'd need to add support for filtering `init` / `node` features
using bitmasks, which isn't entirely trivial to handle and requires
careful management of plugin features.

Fix flaky onion message test (#3266)

The "automatically connect to known nodes" test was flaky, because there
is a race with our code that lazily removes node announcements from our
DB after channels close (in `handleChannelSpent` in `Validation.scala`).

We now explicitly wait for gossip to ensure that we cannot run into this
race condition, which should fix the flakyness during CI.

Reject offers with amount set to `0` (#3265)

This doesn't make any sense, the field should be omitted if any amount
is acceptable: setting it to `0` is confusing.

See https://github.com/lightning/bolts/pull/1316

Remove support for zlib encoding for channel queries (#3263)

We removed support for sending compressed data when it was removed from
the spec (see https://github.com/ACINQ/eclair/pull/2244). We meant to
remove support for receiving compressed data as well as a follow-up, but
it slipped for a long time! There's no reason to keep supporting it now.