Bitcoin Core

test: return full keypair from `getnewdestination` helper

This can be useful for spending funds that are received at the created
address. Preparatory for the next commit, which modifies the taproot
functional test to work without the Bitcoin Core wallet compiled in.

private broadcast: add release note for limited cap

private broadcast: limit outstanding txs to count of 10,000

Add a belt-and-suspenders feature, limit the amount of
memory and cpu possible when unlucky or simply misconfigured.
The worst case limit is roughly 400kB * 10,000 = 4GB, regardless
of usage pattern.

Before this change, sheer volume of broadcasts, mismatches in
standardness rules, or simply fee mismatches may result in unbounded
growth of memory usage. As the feature may be expanded in
the future, explicit bounds helps reasoning going forward.

Release cs_main between individual private tx re-attempts

This queue may be hundreds to thousands of items, each of
which are being revalidated at short interval. Allow the
node to be more responsive in this scenario.

fuzz: add coins_view_stacked fuzz harness to test concurrent leveldb reads

doc: add release notes

coins: add ready flag to InputToFetch

Prepares for ProcessInput to be called from multiple threads.

This flag acts as a memory fence around InputToFetch::coin. There is no lock
guarding reads and writes of the coin field.
Instead we use the flag's release/acquire semantics to ensure that when the
main thread reads the coin it will have happened after a worker thread has
finished writing it.

Co-authored-by: l0rinc <pap.lorinc@gmail.com>

coins: fetch inputs in parallel

Leverages the thread pool to fetch inputs on multiple threads, while the overlay
serves inputs on the main thread.

This is a performance improvement over blocking the main thread to fetch inputs.

Co-authored-by: l0rinc <pap.lorinc@gmail.com>

doc: update CoinsViewOverlay docstring to describe parallel fetching

Co-authored-by: l0rinc <pap.lorinc@gmail.com>

test: add unit tests for CoinsViewOverlay::StartFetching

Co-authored-by: l0rinc <pap.lorinc@gmail.com>

fuzz: update harnesses to cover CoinsViewOverlay::StartFetching

Co-authored-by: l0rinc <pap.lorinc@gmail.com>
Co-authored-by: sedited <seb.kung@gmail.com>

validation: collect block inputs in CoinsViewOverlay before ConnectBlock

Introduce CoinsViewOverlay::StartFetching, which maps all input prevouts of a
block to a new m_inputs vector of InputToFetch elements. Returns a ResetGuard
which is lifetime bound to the block, while the InputToFetch elements are
lifetime bound to the block as well.

Inputs spending outputs of an earlier transaction in the same block won't
be in the cache or the db. They also won't be requested by FetchCoinFromBase,
so we filter them out while building m_inputs to not waste time trying to
fetch them. Build an unordered set of seen txids while flattening m_inputs and
skip any prevout whose hash is already in the set.

Introduce StopFetching to clear the m_inputs vector.
CCoinsViewCache::Reset is made virtual and is overridden in CoinsViewOverlay.
StopFetching is called on Reset, so the InputToFetch objects will not
exceed the lifetime of the block.

Introduce ProcessInput to fetch the utxo of an individual input in m_inputs.
Each caller fetches the input at m_input_head and increments it, so each call
will fetch the next input in the queue.

Fetch coins from the m_inputs vector in FetchCoinFromBase by comparing the
requested outpoint against the single input at m_input_tail. ConnectBlock
requests prevouts in the same order StartFetching queued them, and same-block
spends are filtered out, so the coin to serve is always the one at m_input_tail
(aside from BIP30 checks, an invalid block, or when the thread pool is not yet.
These cases fall back to base->PeekCoin).

This is designed deliberately so multiple threads can call ProcessInput independently.

Co-authored-by: l0rinc <pap.lorinc@gmail.com>
Co-authored-by: Hodlinator <172445034+hodlinator@users.noreply.github.com>

validation: add -prevoutfetchthreads configuration option

Add a configuration option for the number of worker threads used for
parallel UTXO prevout prefetching during block connection.

Default is 8 threads, max is 16, 0 disables parallel fetching.

coins: introduce thread pool in CoinsViewOverlay

Introduce a ThreadPool shared pointer to CoinsViewOverlay. A pool managed
externally can be passed in the constructor.

A global thread pool is used in fuzz harnesses since iterations can happen
faster than the OS can create and tear down thread pools.
This can cause a memory leak when fuzzing.

Co-authored-by: l0rinc <pap.lorinc@gmail.com>

bitcoin-util: Add netmagic command

validation: In AcceptBlock, ignore flush result

At the end of `ChainstateManager::AcceptBlock`, errors from
`FlushStateToDisk` (e.g. low disk space during pruning)
are ignored, so that callers can't mistreat a flush failure
as a block validation failure.
The internal fatal error notification still fires, so the node
will shut down on unrecoverable flush errors.
For state a dummy value is used, and the return value is ignored.
Previously the in-out `state` parameter was used, so it could
return a flush error message.

Co-authored-by: Ryan Ofsky <ryan@ofsky.org>

qa: Strip prefix length from NetBSD `ifconfig` output

Modern NetBSD `ifconfig` prints interface addresses in CIDR notation
(e.g. `inet 127.0.0.1/8`), unlike the other supported platforms which
print the netmask as a separate field. The trailing prefix length breaks
functional tests that expect a plain IP address.

This change is a no-op on other platforms.

qa: Ignore `lsof` warnings on NetBSD

On NetBSD, `lsof` can produce the following warnings:
1. "created device cache file: ..." - usually happens in a CI
environment.
2. "compiled for NetBSD release 10.0; this is 10.1." or similar - when
the `lsof` binary package is installed on a point release.

This change suppresses both warnings printed to stderr and fixes the
affected tests.

doc: Add `lsof` to Test Suite Dependencies on NetBSD

qa: Support `get_bind_addrs` and `feature_bind_extra` on illumos

This extends support for `get_bind_addrs` and `feature_bind_extra` to
illumos-based OSes.

doc: Add `lsof` to Test Suite Dependencies on FreeBSD

qa: Drop OpenBSD from supported platforms in `get_bind_addrs` function

qa: Add `skip_if_no_lsof_on_nonlinux` helper and use it where needed

Some functional tests on non-Linux platforms rely on the `lsof` utility.
However, we treat all other functional test dependencies, such as
additional Python modules, as optional, and skip dependent tests if
those are unavailable.

This change makes `lsof` optional as well.

test: introduce a worker thread in http socket error test

test: ensure HTTPServer race condition is fixed

The result of WriteReply() losing the race condition would prevent any
new requests being read from the socket. The socket error test
sent 3 requests all at once after connecting, so in this commit
we separate the the third request to make the losing race
condition more likely, and make its effect more obvious.