Merge bitcoin/bitcoin#34075: fees: Introduce Mempool Based Fee Estimation to reduce overestimation
What changed, and why it matters
This commit merges a major Bitcoin Core change that adds a second, mempool-based fee estimator alongside the existing block-history estimator. By default, `estimatesmartfee` now returns the lower of the two estimates, intended to reduce overpayment during quiet mempool periods. The change also renames files, moves fee-estimate data into a new `fees/` directory, and adds new RPC options so users can pick which estimator to use. It is a deliberate design change, not a hidden bug fix, and the authors explicitly discuss the trade-offs (risk of underestimation during sudden fee spikes).
Treat this as a significant consensus-adjacent behavior change rather than a vulnerability. Operators relying on `estimatesmartfee` should review the new default combined behavior, understand that default estimates may now be lower, and consider using `fee_rate_estimator: "block_policy"` if they require the previous behavior. Reviewers should scrutinize the mempool-health threshold, cache staleness handling, and serialization robustness in follow-up audits. No immediate patch is required.
Security signals we found
Behavior change in widely used fee estimation RPC (`estimatesmartfee`)
New mempool-based estimate can only lower, not raise, the combined result
Added mempool-health gating (75% coverage over last 6 blocks) to mitigate gaming
Authors explicitly acknowledge Finney-attack-style mempool gaming risk and state the design limits the attack to lowering estimates
File migration logic for legacy fee estimates data
New serialization/deserialization paths for persisted mempool estimator data with version checks
Caching layer tied to chain tip; potential stale-tip race noted in comments
Extensive test additions (unit, functional, fuzz) for new estimator behavior and I/O
Evidence from the diff
PR #34075 introduces FeeRateEstimatorManager, which owns both CBlockPolicyEstimator (renamed/moved) and a new MemPoolFeeRateEstimator. The manager subscribes to validation signals and forwards block/mempool events. The mempool estimator builds a block template, computes 50th/75th percentile package feerates, caches the result for 7 seconds, and only returns an estimate when recent mined-block coverage of the local mempool exceeds 75% over a 6-block window. estimatesmartfee gains fee_rate_estimator (none/block_policy/mempool_policy) and verbosity options, and now returns an estimator field. Wallet fee_reason values are split from block-policy internal reasons. Legacy fee_estimates.dat is migrated to fees/block_policy_estimates.dat; mempool stats persist to fees/mempool_policy_estimator.dat.
Changed components
src/policy/fees/block_policy_estimator.cpp/hsrc/policy/fees/estimator_man.cpp/hsrc/policy/fees/mempool_estimator.cpp/hsrc/policy/fees/estimator_args.cpp/hsrc/rpc/fees.cppsrc/rpc/server_util.cpp/hsrc/init.cppsrc/node/context.cpp/hsrc/node/interfaces.cppsrc/interfaces/chain.hsrc/interfaces/wallet.hsrc/wallet/fees.cpp/hsrc/wallet/spend.cppsrc/wallet/rpc/spend.cppsrc/common/messages.cppsrc/validation.cppsrc/validationinterface.cpp/hsrc/txmempool.cpp/hsrc/qt/sendcoinsdialog.cppInspect captured patch +2167 / −523
### doc/files.md
@@ -54,6 +54,7 @@ Subdirectory | File(s) | Description
`blocks/` | `revNNNNN.dat`<sup>[\[2\]](#note2)</sup> | Block undo data (custom format)
`blocks/` | `xor.dat` | Rolling XOR pattern for block and undo data files
`chainstate/` | LevelDB database | Blockchain state (a compact representation of all currently unspent transaction outputs (UTXOs) and metadata about the transactions they are from)
+`fees/` | `block_policy_estimates.dat` and `mempool_policy_estimator.dat` | Stores block policy and mempool policy estimator data
`indexes/txindex/` | LevelDB database | Transaction index; *optional*, used if `-txindex=1`
`indexes/txospenderindex/` | LevelDB database | Transaction spender index; *optional*, used if `-txospenderindex=1`
`indexes/blockfilter/basic/db/` | LevelDB database | Blockfilter index LevelDB database for the basic filtertype; *optional*, used if `-blockfilterindex=basic`
@@ -65,7 +66,6 @@ Subdirectory | File(s) | Description
`./` | `bitcoin.conf` | User-defined [configuration settings](bitcoin-conf.md) for `bitcoind` or `bitcoin-qt`. File is not written to by the software and must be created manually. Path can be specified by `-conf` option
`./` | `bitcoind.pid` | Stores the process ID (PID) of `bitcoind` or `bitcoin-qt` while running; created at start and deleted on shutdown; can be specified by `-pid` option
`./` | `debug.log` | Contains debug information and general logging generated by `bitcoind` or `bitcoin-qt`; can be specified by `-debuglogfile` option
-`./` | `fee_estimates.dat` | Stores statistics used to estimate minimum transaction fees required for confirmation
`./` | `guisettings.ini.bak` | Backup of former [GUI settings](#gui-settings) after `-resetguisettings` option is used
`./` | `mempool.dat` | Dump of the mempool's transactions
`./` | `onion_v3_private_key` | Cached Tor onion service private key for `-listenonion` option
### doc/release-notes-34075.md
@@ -0,0 +1,52 @@
+Updated RPCs
+------------
+
+- The `estimatesmartfee` RPC now combines two fee rate estimators: the existing
+ block policy fee rate estimator and a new mempool fee rate estimator.
+
+- The new mempool fee rate estimator produces conservative and economical fee
+ rate estimates from the current contents of the mempool. It only produces a
+ fee rate estimate when recent blocks indicate a healthy mempool, and falls
+ back to the higher of the minimum relay fee rate and the current mempool
+ minimum fee rate when the mempool is too sparse. Its statistics are persisted
+ to `fees/mempool_policy_estimator.dat` and reloaded on startup.
+
+ `estimatesmartfee` returns the lower of the two fee rate estimators' results,
+ so the mempool fee rate estimator can only lower the block policy fee rate
+ estimate.
+
+- The combined estimate requires both estimators to succeed. If the mempool fee
+ rate estimator cannot produce an estimate, for example, while the mempool is
+ still loading, when too few recent blocks have been observed, or when the
+ mempool is suspected to be unhealthy, an error is returned.
+
+- `estimatesmartfee` accepts an `options` object with `fee_rate_estimator`.
+ Recognized values are `"none"` (the default, combined behavior described
+ above), `"block_policy"` (use only the block policy fee rate estimator),
+ and `"mempool_policy"` (use only the mempool fee rate estimator).
+ All unknown values are treated as `"none"`.
+ Users who want the previous behavior can select the block policy fee rate
+ estimator explicitly.
+
+- The options object also accepts `verbosity`. A verbosity of `2` or higher
+ also returns `mempool_health_statistics`.
+
+- When `fee_rate_estimator` is `"none"` and the estimate succeeds, the response
+ also includes an `estimator` field identifying which fee rate estimator produced
+ the result.
+
+- Block policy fee estimator data is now stored in
+ `fees/block_policy_estimates.dat`. If the new file does not exist, the
+ legacy `fee_estimates.dat` file is moved to the new path during startup. If
+ both files exist, the legacy file is removed.
+
+- Wallet fee rate estimation uses the default combined estimate.
+
+Wallet
+------
+
+- The `fee_reason` field returned by wallet transaction creation RPCs now
+ reports the reason the wallet selected the fee rate (fee rate estimator,
+ mempool minimum, fallback, or minimum required) instead of the block policy
+ fee rate estimator's internal threshold details. Those details remain
+ available in the block policy fee rate estimator debug log.
### src/CMakeLists.txt
@@ -247,7 +247,9 @@ add_library(bitcoin_node STATIC EXCLUDE_FROM_ALL
noui.cpp
policy/ephemeral_policy.cpp
policy/fees/block_policy_estimator.cpp
- policy/fees/block_policy_estimator_args.cpp
+ policy/fees/estimator_args.cpp
+ policy/fees/estimator_man.cpp
+ policy/fees/mempool_estimator.cpp
policy/packages.cpp
policy/rbf.cpp
policy/settings.cpp
### src/bench/mempool_stress.cpp
@@ -156,7 +156,7 @@ static void ComplexMemPool(benchmark::Bench& bench)
// in the same state at the end of the function, so we benchmark both
// mining a block and reorging the block's contents back into the mempool.
bench.run([&]() NO_THREAD_SAFETY_ANALYSIS {
- pool.removeForBlock(tx_remove_for_block, /*nBlockHeight=*/100);
+ pool.removeForBlock(tx_remove_for_block);
for (auto& tx: tx_remove_for_block) {
AddTx(tx, pool, det_rand);
}
### src/common/messages.cpp
@@ -7,7 +7,6 @@
#include <common/types.h>
#include <node/types.h>
-#include <policy/fees/block_policy_estimator.h>
#include <tinyformat.h>
#include <util/check.h>
#include <util/fees.h>
@@ -28,12 +27,9 @@ namespace common {
std::string StringForFeeReason(FeeReason reason)
{
static const std::map<FeeReason, std::string> fee_reason_strings = {
- {FeeReason::NONE, "None"},
- {FeeReason::HALF_ESTIMATE, "Half Target 60% Threshold"},
- {FeeReason::FULL_ESTIMATE, "Target 85% Threshold"},
- {FeeReason::DOUBLE_ESTIMATE, "Double Target 95% Threshold"},
- {FeeReason::CONSERVATIVE, "Conservative Double Target longer horizon"},
+ {FeeReason::FEE_RATE_ESTIMATOR, "Fee Rate Estimator"},
{FeeReason::MEMPOOL_MIN, "Mempool Min Fee"},
+ {FeeReason::USER_SPECIFIED, "User Specified Fee"},
{FeeReason::FALLBACK, "Fallback fee"},
{FeeReason::REQUIRED, "Minimum Required Fee"},
};
@@ -60,13 +56,9 @@ std::string FeeModeInfo(const std::pair<std::string, FeeEstimateMode>& mode, std
case FeeEstimateMode::UNSET:
return strprintf("%s means no mode set (%s). \n", mode.first, default_info);
case FeeEstimateMode::ECONOMICAL:
- return strprintf("%s estimates use a shorter time horizon, making them more\n"
- "responsive to short-term drops in the prevailing fee market. This mode\n"
- "potentially returns a lower fee rate estimate.\n", mode.first);
+ return strprintf("%s mode potentially returns a lower fee rate estimate.\n", mode.first);
case FeeEstimateMode::CONSERVATIVE:
- return strprintf("%s estimates use a longer time horizon, making them\n"
- "less responsive to short-term drops in the prevailing fee market. This mode\n"
- "potentially returns a higher fee rate estimate.\n", mode.first);
+ return strprintf("%s potentially returns a higher fee rate estimate.\n", mode.first);
} // no default case, so the compiler can warn about missing cases
assert(false);
}
### src/common/messages.h
@@ -18,7 +18,6 @@
struct bilingual_str;
enum class FeeEstimateMode;
enum class FeeReason;
-
namespace node {
enum class TransactionError;
} // namespace node
### src/init.cpp
@@ -67,7 +67,9 @@
#include <node/peerman_args.h>
#include <policy/feerate.h>
#include <policy/fees/block_policy_estimator.h>
-#include <policy/fees/block_policy_estimator_args.h>
+#include <policy/fees/estimator_args.h>
+#include <policy/fees/estimator_man.h>
+#include <policy/fees/mempool_estimator.h>
#include <policy/policy.h>
#include <policy/settings.h>
#include <protocol.h>
@@ -368,10 +370,10 @@ void Shutdown(NodeContext& node)
// Drop transactions we were still watching, record fee estimations and unregister
// fee estimator from validation interface.
- if (node.fee_estimator) {
- node.fee_estimator->Flush();
+ if (node.fee_estimator_man) {
+ node.fee_estimator_man->ShutdownFlush();
if (node.validation_signals) {
- node.validation_signals->UnregisterValidationInterface(node.fee_estimator.get());
+ node.validation_signals->UnregisterValidationInterface(node.fee_estimator_man.get());
}
}
@@ -430,8 +432,8 @@ void Shutdown(NodeContext& node)
if (node.validation_signals) {
node.validation_signals->UnregisterAllValidationInterfaces();
}
+ node.fee_estimator_man.reset();
node.mempool.reset();
- node.fee_estimator.reset();
node.chainman.reset();
node.validation_signals.reset();
node.scheduler.reset();
@@ -1679,22 +1681,6 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
rng.rand64(),
*node.addrman, *node.netgroupman, chainparams, args.GetBoolArg("-networkactive", true));
- assert(!node.fee_estimator);
- // Don't initialize fee estimation with old data if we don't relay transactions,
- // as they would never get updated.
- if (!peerman_opts.ignore_incoming_txs) {
- bool read_stale_estimates = args.GetBoolArg("-acceptstalefeeestimates", DEFAULT_ACCEPT_STALE_FEE_ESTIMATES);
- if (read_stale_estimates && (chainparams.GetChainType() != ChainType::REGTEST)) {
- return InitError(strprintf(_("acceptstalefeeestimates is not supported on %s chain."), chainparams.GetChainTypeString()));
- }
- node.fee_estimator = std::make_unique<CBlockPolicyEstimator>(FeeestPath(args), read_stale_estimates);
-
- // Flush estimates to disk periodically
- CBlockPolicyEstimator* fee_estimator = node.fee_estimator.get();
- scheduler.scheduleEvery([fee_estimator] { fee_estimator->FlushFeeEstimates(); }, FEE_FLUSH_INTERVAL);
- validation_signals.RegisterValidationInterface(fee_estimator);
- }
-
for (const std::string& socket_addr : args.GetArgs("-bind")) {
std::string host_out;
uint16_t port_out{0};
@@ -1922,6 +1908,24 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
}
ChainstateManager& chainman = *Assert(node.chainman);
+
+ assert(!node.fee_estimator_man);
+ // Don't initialize fee estimation with old data if we don't relay transactions,
+ // as they would never get updated.
+ if (!peerman_opts.ignore_incoming_txs) {
+ bool read_stale_estimates = args.GetBoolArg("-acceptstalefeeestimates", DEFAULT_ACCEPT_STALE_FEE_ESTIMATES);
+ if (read_stale_estimates && (chainparams.GetChainType() != ChainType::REGTEST)) {
+ return InitError(strprintf(_("acceptstalefeeestimates is not supported on %s chain."), chainparams.GetChainTypeString()));
+ }
+ MaybeMigrateLegacyFeeEstimates(args);
+ node.fee_estimator_man = std::make_unique<FeeRateEstimatorManager>(BlockPolicyFeeEstPath(args), read_stale_estimates, MempoolPolicyEstimatorPath(args), *Assert(node.mempool), chainman);
+
+ // Flush estimates to disk periodically
+ FeeRateEstimatorManager* fee_estimator_man = node.fee_estimator_man.get();
+ scheduler.scheduleEvery([fee_estimator_man] { fee_estimator_man->IntervalFlush(); }, FEE_FLUSH_INTERVAL);
+ validation_signals.RegisterValidationInterface(fee_estimator_man);
+ }
+
auto& kernel_notifications{*Assert(node.notifications)};
assert(!node.peerman);
### src/interfaces/chain.h
@@ -10,6 +10,8 @@
#include <kernel/chain.h> // IWYU pragma: export
#include <node/types.h>
#include <primitives/transaction.h>
+#include <util/expected.h>
+#include <util/fees.h>
#include <util/result.h>
#include <cstddef>
@@ -33,7 +35,6 @@ enum class MemPoolRemovalReason;
enum class RBFTransactionState;
struct bilingual_str;
struct CBlockLocator;
-struct FeeCalculation;
namespace kernel {
struct ChainstateRole;
} // namespace kernel
@@ -256,11 +257,11 @@ class Chain
//! Check if transaction will pass the mempool's chain limits.
virtual util::Result<void> checkChainLimits(const CTransactionRef& tx) = 0;
- //! Estimate smart fee.
- virtual CFeeRate estimateSmartFee(int num_blocks, bool conservative, FeeCalculation* calc = nullptr) = 0;
+ //! Estimate a fee rate.
+ virtual util::Expected<FeeRateEstimation, FeeRateEstimationError> getFeeRateEstimate(int num_blocks, bool conservative) const = 0;
//! Fee estimator max target.
- virtual unsigned int estimateMaxBlocks() = 0;
+ virtual unsigned int maximumFeeEstimationTargetBlocks() const = 0;
//! Mempool minimum fee.
virtual CFeeRate mempoolMinFee() = 0;
### src/interfaces/wallet.h
@@ -22,6 +22,7 @@
#include <functional>
#include <map>
#include <memory>
+#include <optional>
#include <string>
#include <tuple>
#include <type_traits>
@@ -242,7 +243,7 @@ class Wallet
//! Get minimum fee.
virtual CAmount getMinimumFee(unsigned int tx_bytes,
const wallet::CCoinControl& coin_control,
- int* returned_target,
+ std::optional<int>* returned_target,
FeeReason* reason) = 0;
//! Get tx confirm target.
### src/node/context.cpp
@@ -15,7 +15,7 @@
#include <netgroup.h>
#include <node/kernel_notifications.h>
#include <node/warnings.h>
-#include <policy/fees/block_policy_estimator.h>
+#include <policy/fees/estimator_man.h>
#include <scheduler.h>
#include <torcontrol.h>
#include <txmempool.h>
### src/node/context.h
@@ -18,8 +18,8 @@ class ArgsManager;
class AddrMan;
class BanMan;
class BaseIndex;
-class CBlockPolicyEstimator;
class CConnman;
+class FeeRateEstimatorManager;
class ValidationSignals;
class CScheduler;
class CTxMemPool;
@@ -70,7 +70,7 @@ struct NodeContext {
std::unique_ptr<CConnman> connman;
std::unique_ptr<CTxMemPool> mempool;
std::unique_ptr<const NetGroupManager> netgroupman;
- std::unique_ptr<CBlockPolicyEstimator> fee_estimator;
+ std::unique_ptr<FeeRateEstimatorManager> fee_estimator_man;
std::unique_ptr<PeerManager> peerman;
std::unique_ptr<TorController> tor_controller;
std::unique_ptr<ChainstateManager> chainman;
### src/node/interfaces.cpp
@@ -46,7 +46,7 @@
#include <node/types.h>
#include <node/warnings.h>
#include <policy/feerate.h>
-#include <policy/fees/block_policy_estimator.h>
+#include <policy/fees/estimator_man.h>
#include <policy/policy.h>
#include <policy/rbf.h>
#include <primitives/block.h>
@@ -61,6 +61,8 @@
#include <univalue.h>
#include <util/btcsignals.h>
#include <util/check.h>
+#include <util/expected.h>
+#include <util/fees.h>
#include <util/result.h>
#include <util/signalinterrupt.h>
#include <util/string.h>
@@ -736,15 +738,15 @@ class ChainImpl : public Chain
}
return {};
}
- CFeeRate estimateSmartFee(int num_blocks, bool conservative, FeeCalculation* calc) override
+ util::Expected<FeeRateEstimation, FeeRateEstimationError> getFeeRateEstimate(int num_blocks, bool conservative) const override
{
- if (!m_node.fee_estimator) return {};
- return m_node.fee_estimator->estimateSmartFee(num_blocks, calc, conservative);
+ if (!m_node.fee_estimator_man) return EstimationError(FeeRateEstimatorType::NONE, /*returned_target=*/0, /*error=*/{});
+ return m_node.fee_estimator_man->GetFeeRateEstimate(num_blocks, conservative);
}
- unsigned int estimateMaxBlocks() override
+ unsigned int maximumFeeEstimationTargetBlocks() const override
{
- if (!m_node.fee_estimator) return 0;
- return m_node.fee_estimator->HighestTargetTracked(FeeEstimateHorizon::LONG_HALFLIFE);
+ if (!m_node.fee_estimator_man) return 0;
+ return m_node.fee_estimator_man->MaximumTarget();
}
CFeeRate mempoolMinFee() override
{
### src/policy/feerate.h
@@ -47,6 +47,15 @@ class CFeeRate
*/
CFeeRate(const CAmount& nFeePaid, int32_t virtual_bytes);
+ /**
+ * Construct from a fee rate expressed as FeePerVSize.
+ *
+ * Lossless: CFeeRate is internally a FeePerVSize, so the exact
+ * fee/vsize fraction is preserved. A feerate whose size is less than
+ * or equal to 0 results in 0 fee rate per 0 size.
+ */
+ explicit CFeeRate(const FeePerVSize& feerate) : m_feerate{feerate.size > 0 ? feerate : FeePerVSize{}} {}
+
/**
* Return the fee in satoshis for the given vsize in vbytes.
* If the calculated fee would have fractional satoshis, then the
### src/policy/fees/block_policy_estimator.cpp
@@ -30,6 +30,7 @@
#include <cstdint>
#include <exception>
#include <stdexcept>
+#include <system_error>
#include <utility>
// The current format written, and the version required to read. Must be
@@ -48,6 +49,23 @@ std::string StringForFeeEstimateHorizon(FeeEstimateHorizon horizon)
assert(false);
}
+std::string StringForBlockPolicyEstimateReason(BlockPolicyEstimateReason reason)
+{
+ switch (reason) {
+ case BlockPolicyEstimateReason::NONE:
+ return "None";
+ case BlockPolicyEstimateReason::HALF_ESTIMATE:
+ return "Half Target 60% Threshold";
+ case BlockPolicyEstimateReason::FULL_ESTIMATE:
+ return "Target 85% Threshold";
+ case BlockPolicyEstimateReason::DOUBLE_ESTIMATE:
+ return "Double Target 95% Threshold";
+ case BlockPolicyEstimateReason::CONSERVATIVE:
+ return "Conservative Double Target longer horizon";
+ } // no default case, so the compiler can warn about missing cases
+ assert(false);
+}
+
namespace {
struct EncodedDoubleFormatter
@@ -578,21 +596,6 @@ CBlockPolicyEstimator::CBlockPolicyEstimator(const fs::path& estimation_filepath
CBlockPolicyEstimator::~CBlockPolicyEstimator() = default;
-void CBlockPolicyEstimator::TransactionAddedToMempool(const NewMempoolTransactionInfo& tx, uint64_t /*unused*/)
-{
- processTransaction(tx);
-}
-
-void CBlockPolicyEstimator::TransactionRemovedFromMempool(const CTransactionRef& tx, MemPoolRemovalReason /*unused*/, uint64_t /*unused*/)
-{
- removeTx(tx->GetHash());
-}
-
-void CBlockPolicyEstimator::MempoolTransactionsRemovedForBlock(const std::vector<RemovedMempoolTransactionInfo>& txs_removed_for_block, unsigned int nBlockHeight)
-{
- processBlock(txs_removed_for_block, nBlockHeight);
-}
-
void CBlockPolicyEstimator::processTransaction(const NewMempoolTransactionInfo& tx)
{
LOCK(m_cs_fee_estimator);
@@ -872,11 +875,12 @@ CFeeRate CBlockPolicyEstimator::estimateSmartFee(int confTarget, FeeCalculation
{
LOCK(m_cs_fee_estimator);
- if (feeCalc) {
- feeCalc->desiredTarget = confTarget;
- feeCalc->returnedTarget = confTarget;
- feeCalc->best_height = nBestSeenHeight;
- }
+ FeeCalculation temp_fee_calc;
+ if (!feeCalc) feeCalc = &temp_fee_calc;
+
+ feeCalc->desiredTarget = confTarget;
+ feeCalc->returnedTarget = confTarget;
+ feeCalc->best_height = nBestSeenHeight;
double median = -1;
EstimationResult tempResult;
@@ -893,7 +897,7 @@ CFeeRate CBlockPolicyEstimator::estimateSmartFee(int confTarget, FeeCalculation
if ((unsigned int)confTarget > maxUsableEstimate) {
confTarget = maxUsableEstimate;
}
- if (feeCalc) feeCalc->returnedTarget = confTarget;
+ feeCalc->returnedTarget = confTarget;
if (confTarget <= 1) return CFeeRate(0); // error condition
@@ -917,51 +921,76 @@ CFeeRate CBlockPolicyEstimator::estimateSmartFee(int confTarget, FeeCalculation
* See: https://github.com/bitcoin/bitcoin/issues/11800#issuecomment-349697807
*/
double halfEst = estimateCombinedFee(confTarget/2, HALF_SUCCESS_PCT, true, &tempResult);
- if (feeCalc) {
- feeCalc->est = tempResult;
- feeCalc->reason = FeeReason::HALF_ESTIMATE;
- }
+ feeCalc->est = tempResult;
+ feeCalc->reason = BlockPolicyEstimateReason::HALF_ESTIMATE;
median = halfEst;
double actualEst = estimateCombinedFee(confTarget, SUCCESS_PCT, true, &tempResult);
if (actualEst > median) {
median = actualEst;
- if (feeCalc) {
- feeCalc->est = tempResult;
- feeCalc->reason = FeeReason::FULL_ESTIMATE;
- }
+ feeCalc->est = tempResult;
+ feeCalc->reason = BlockPolicyEstimateReason::FULL_ESTIMATE;
}
double doubleEst = estimateCombinedFee(2 * confTarget, DOUBLE_SUCCESS_PCT, !conservative, &tempResult);
if (doubleEst > median) {
median = doubleEst;
- if (feeCalc) {
- feeCalc->est = tempResult;
- feeCalc->reason = FeeReason::DOUBLE_ESTIMATE;
- }
+ feeCalc->est = tempResult;
+ feeCalc->reason = BlockPolicyEstimateReason::DOUBLE_ESTIMATE;
}
if (conservative || median == -1) {
double consEst = estimateConservativeFee(2 * confTarget, &tempResult);
if (consEst > median) {
median = consEst;
- if (feeCalc) {
- feeCalc->est = tempResult;
- feeCalc->reason = FeeReason::CONSERVATIVE;
- }
+ feeCalc->est = tempResult;
+ feeCalc->reason = BlockPolicyEstimateReason::CONSERVATIVE;
}
}
if (median < 0) return CFeeRate(0); // error condition
+ LogDebug(BCLog::ESTIMATEFEE, "estimateSmartFee Selected feerate: %g Tgt: %d (requested %d) Reason: \"%s\" Decay %.5f: Estimation: (%g - %g) %.2f%% %.1f/(%.1f %d mem %.1f out) Fail: (%g - %g) %.2f%% %.1f/(%.1f %d mem %.1f out)",
+ median, feeCalc->returnedTarget, feeCalc->desiredTarget, StringForBlockPolicyEstimateReason(feeCalc->reason), feeCalc->est.decay,
+ feeCalc->est.pass.start, feeCalc->est.pass.end,
+ (feeCalc->est.pass.totalConfirmed + feeCalc->est.pass.inMempool + feeCalc->est.pass.leftMempool) > 0.0 ? 100 * feeCalc->est.pass.withinTarget / (feeCalc->est.pass.totalConfirmed + feeCalc->est.pass.inMempool + feeCalc->est.pass.leftMempool) : 0.0,
+ feeCalc->est.pass.withinTarget, feeCalc->est.pass.totalConfirmed, feeCalc->est.pass.inMempool, feeCalc->est.pass.leftMempool,
+ feeCalc->est.fail.start, feeCalc->est.fail.end,
+ (feeCalc->est.fail.totalConfirmed + feeCalc->est.fail.inMempool + feeCalc->est.fail.leftMempool) > 0.0 ? 100 * feeCalc->est.fail.withinTarget / (feeCalc->est.fail.totalConfirmed + feeCalc->est.fail.inMempool + feeCalc->est.fail.leftMempool) : 0.0,
+ feeCalc->est.fail.withinTarget, feeCalc->est.fail.totalConfirmed, feeCalc->est.fail.inMempool, feeCalc->est.fail.leftMempool);
+
return CFeeRate(llround(median));
}
+util::Expected<FeeRateEstimation, FeeRateEstimationError> CBlockPolicyEstimator::EstimateFeeRate(int target, bool conservative) const
+{
+ FeeCalculation fee_calc;
+ CFeeRate feerate{estimateSmartFee(target, &fee_calc, conservative)};
+ if (feerate == CFeeRate(0)) {
+ return EstimationError(FeeRateEstimatorType::BLOCK_POLICY, fee_calc.returnedTarget, "Insufficient data or no feerate found");
+ }
+ return FeeRateEstimation{FeeRateEstimatorType::BLOCK_POLICY, feerate.GetFeePerVSize(), fee_calc.returnedTarget};
+}
+
+unsigned int CBlockPolicyEstimator::MaximumTarget() const
+{
+ return HighestTargetTracked(FeeEstimateHorizon::LONG_HALFLIFE);
+}
+
void CBlockPolicyEstimator::Flush() {
FlushUnconfirmed();
FlushFeeEstimates();
}
void CBlockPolicyEstimator::FlushFeeEstimates()
{
+ if (!m_estimation_filepath.parent_path().empty()) {
+ std::error_code error;
+ fs::create_directories(m_estimation_filepath.parent_path(), error);
+ if (error) {
+ LogWarning("Failed to create fee estimates directory %s: %s. Continue anyway.", fs::PathToString(m_estimation_filepath.parent_path()), error.message());
+ return;
+ }
+ }
+
AutoFile est_file{fsbridge::fopen(m_estimation_filepath, "wb")};
if (est_file.IsNull() || !Write(est_file)) {
LogWarning("Failed to write fee estimates to %s. Continue anyway.", fs::PathToString(m_estimation_filepath));
### src/policy/fees/block_policy_estimator.h
@@ -7,11 +7,13 @@
#include <consensus/amount.h>
#include <policy/feerate.h>
+#include <primitives/transaction_identifier.h>
#include <random.h>
#include <sync.h>
#include <uint256.h>
+#include <util/expected.h>
+#include <util/fees.h>
#include <util/fs.h>
-#include <validationinterface.h>
#include <array>
#include <chrono>
@@ -22,10 +24,7 @@
#include <vector>
-// How often to flush fee estimates to fee_estimates.dat.
-inline constexpr std::chrono::hours FEE_FLUSH_INTERVAL{1};
-
-/** fee_estimates.dat that are more than 60 hours (2.5 days) old will not be read,
+/** Block policy estimate files that are more than 60 hours (2.5 days) old will not be read,
* as fee estimates are based on historical data and may be inaccurate if
* network activity has changed.
*/
@@ -56,17 +55,16 @@ inline constexpr auto ALL_FEE_ESTIMATE_HORIZONS = std::array{
std::string StringForFeeEstimateHorizon(FeeEstimateHorizon horizon);
/* Enumeration of reason for returned fee estimate */
-enum class FeeReason {
+enum class BlockPolicyEstimateReason {
NONE,
HALF_ESTIMATE,
FULL_ESTIMATE,
DOUBLE_ESTIMATE,
CONSERVATIVE,
- MEMPOOL_MIN,
- FALLBACK,
- REQUIRED,
};
+std::string StringForBlockPolicyEstimateReason(BlockPolicyEstimateReason reason);
+
/* Used to return detailed information about a feerate bucket */
struct EstimatorBucket
{
@@ -90,7 +88,7 @@ struct EstimationResult
struct FeeCalculation
{
EstimationResult est;
- FeeReason reason = FeeReason::NONE;
+ BlockPolicyEstimateReason reason = BlockPolicyEstimateReason::NONE;
int desiredTarget = 0;
int returnedTarget = 0;
unsigned int best_height{0};
@@ -144,7 +142,7 @@ struct FeeCalculation
* a certain number of blocks. Every time a block is added to the best chain, this class records
* stats on the transactions included in that block
*/
-class CBlockPolicyEstimator : public CValidationInterface
+class CBlockPolicyEstimator
{
private:
/** Track confirm delays up to 12 blocks for short horizon */
@@ -263,15 +261,15 @@ class CBlockPolicyEstimator : public CValidationInterface
/** Calculates the age of the file, since last modified */
std::chrono::hours GetFeeEstimatorFileAge();
-protected:
- /** Overridden from CValidationInterface. */
- void TransactionAddedToMempool(const NewMempoolTransactionInfo& tx, uint64_t /*unused*/) override
- EXCLUSIVE_LOCKS_REQUIRED(!m_cs_fee_estimator);
- void TransactionRemovedFromMempool(const CTransactionRef& tx, MemPoolRemovalReason /*unused*/, uint64_t /*unused*/) override
+ /** Return the highest confirmation target for which an estimate can be provided. */
+ unsigned int MaximumTarget() const
EXCLUSIVE_LOCKS_REQUIRED(!m_cs_fee_estimator);
- void MempoolTransactionsRemovedForBlock(const std::vector<RemovedMempoolTransactionInfo>& txs_removed_for_block, unsigned int nBlockHeight) override
+
+ /** Estimate the feerate needed to confirm within @p target blocks; wraps estimateSmartFee into a FeeRateEstimation. */
+ util::Expected<FeeRateEstimation, FeeRateEstimationError> EstimateFeeRate(int target, bool conservative) const
EXCLUSIVE_LOCKS_REQUIRED(!m_cs_fee_estimator);
+
private:
mutable Mutex m_cs_fee_estimator;
### src/policy/fees/block_policy_estimator_args.cpp
@@ -1,16 +0,0 @@
-// Copyright (c) 2021-present The Bitcoin Core developers
-// Distributed under the MIT software license, see the accompanying
-// file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-#include <policy/fees/block_policy_estimator_args.h>
-
-#include <common/args.h>
-
-namespace {
-const char* FEE_ESTIMATES_FILENAME = "fee_estimates.dat";
-} // namespace
-
-fs::path FeeestPath(const ArgsManager& argsman)
-{
- return argsman.GetDataDirNet() / FEE_ESTIMATES_FILENAME;
-}
### src/policy/fees/block_policy_estimator_args.h
@@ -1,15 +0,0 @@
-// Copyright (c) 2022-present The Bitcoin Core developers
-// Distributed under the MIT software license, see the accompanying
-// file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-#ifndef BITCOIN_POLICY_FEES_BLOCK_POLICY_ESTIMATOR_ARGS_H
-#define BITCOIN_POLICY_FEES_BLOCK_POLICY_ESTIMATOR_ARGS_H
-
-#include <util/fs.h>
-
-class ArgsManager;
-
-/** @return The fee estimates data file path. */
-fs::path FeeestPath(const ArgsManager& argsman);
-
-#endif // BITCOIN_POLICY_FEES_BLOCK_POLICY_ESTIMATOR_ARGS_H
### src/policy/fees/estimator_args.cpp
@@ -0,0 +1,60 @@
+// Copyright (c) The Bitcoin Core developers
+// Distributed under the MIT software license, see the accompanying
+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#include <policy/fees/estimator_args.h>
+
+#include <common/args.h>
+#include <util/log.h>
+
+#include <system_error>
+
+namespace {
+constexpr const char* FEES_BASE_DIR{"fees"};
+constexpr const char* BLOCK_POLICY_ESTIMATES_FILENAME{"block_policy_estimates.dat"};
+constexpr const char* LEGACY_FEE_ESTIMATES_FILENAME{"fee_estimates.dat"};
+constexpr const char* MEMPOOL_POLICY_ESTIMATOR_FILENAME{"mempool_policy_estimator.dat"};
+
+fs::path LegacyFeeEstPath(const ArgsManager& argsman)
+{
+ return argsman.GetDataDirNet() / LEGACY_FEE_ESTIMATES_FILENAME;
+}
+} // namespace
+
+void MaybeMigrateLegacyFeeEstimates(const ArgsManager& argsman)
+{
+ const fs::path legacy_path{LegacyFeeEstPath(argsman)};
+ const fs::path block_policy_path{BlockPolicyFeeEstPath(argsman)};
+ if (!fs::exists(legacy_path)) return;
+ std::error_code error;
+ if (fs::exists(block_policy_path)) {
+ fs::remove(legacy_path, error);
+ if (error) {
+ LogWarning("Failed to remove legacy fee estimates file %s: %s. Continuing anyway.", fs::PathToString(legacy_path), error.message());
+ return;
+ }
+ LogInfo("Removed legacy fee estimates file %s.", fs::PathToString(legacy_path));
+ return;
+ }
+ fs::create_directories(block_policy_path.parent_path(), error);
+ if (error) {
+ LogWarning("Failed to create block policy fee estimates directory %s: %s. Continuing without migration.", fs::PathToString(block_policy_path.parent_path()), error.message());
+ return;
+ }
+ fs::rename(legacy_path, block_policy_path, error);
+ if (error) {
+ LogWarning("Failed to migrate fee estimates from %s to %s: %s. Continuing with fresh estimates.", fs::PathToString(legacy_path), fs::PathToString(block_policy_path), error.message());
+ return;
+ }
+ LogInfo("Migrated fee estimates from %s to %s.", fs::PathToString(legacy_path), fs::PathToString(block_policy_path));
+}
+
+fs::path BlockPolicyFeeEstPath(const ArgsManager& argsman)
+{
+ return argsman.GetDataDirNet() / FEES_BASE_DIR / BLOCK_POLICY_ESTIMATES_FILENAME;
+}
+
+fs::path MempoolPolicyEstimatorPath(const ArgsManager& argsman)
+{
+ return argsman.GetDataDirNet() / FEES_BASE_DIR / MEMPOOL_POLICY_ESTIMATOR_FILENAME;
+}
### src/policy/fees/estimator_args.h
@@ -0,0 +1,21 @@
+// Copyright (c) The Bitcoin Core developers
+// Distributed under the MIT software license, see the accompanying
+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#ifndef BITCOIN_POLICY_FEES_ESTIMATOR_ARGS_H
+#define BITCOIN_POLICY_FEES_ESTIMATOR_ARGS_H
+
+#include <util/fs.h>
+
+class ArgsManager;
+
+/** Move a legacy fee_estimates.dat file to the current block policy fee estimator path, if needed. */
+void MaybeMigrateLegacyFeeEstimates(const ArgsManager& argsman);
+
+/** @return The block policy fee estimator data file path. */
+fs::path BlockPolicyFeeEstPath(const ArgsManager& argsman);
+
+/** @return The mempool policy estimator data file path. */
+fs::path MempoolPolicyEstimatorPath(const ArgsManager& argsman);
+
+#endif // BITCOIN_POLICY_FEES_ESTIMATOR_ARGS_H
### src/policy/fees/estimator_man.cpp
@@ -0,0 +1,105 @@
+// Copyright (c) The Bitcoin Core developers
+// Distributed under the MIT software license. See the accompanying
+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#include <policy/fees/estimator_man.h>
+
+#include <logging.h>
+#include <policy/feerate.h>
+#include <policy/fees/block_policy_estimator.h>
+#include <policy/fees/mempool_estimator.h>
+#include <util/fees.h>
+
+FeeRateEstimatorManager::~FeeRateEstimatorManager() = default;
+
+FeeRateEstimatorManager::FeeRateEstimatorManager(const fs::path& block_policy_path,
+ bool read_stale_estimates,
+ const fs::path& mempool_estimator_path,
+ const CTxMemPool& mempool,
+ ChainstateManager& chainman)
+ : m_block_policy_estimator(std::make_unique<CBlockPolicyEstimator>(block_policy_path, read_stale_estimates)),
+ m_mempool_estimator(std::make_unique<MemPoolFeeRateEstimator>(mempool_estimator_path, mempool, chainman))
+{
+}
+
+util::Expected<FeeRateEstimation, FeeRateEstimationError> FeeRateEstimatorManager::GetFeeRateEstimate(int target, bool conservative) const
+{
+ auto block_policy_estimate = m_block_policy_estimator->EstimateFeeRate(target, conservative);
+ if (!block_policy_estimate) {
+ LogDebug(BCLog::ESTIMATEFEE, "%s", block_policy_estimate.error().reason);
+ return block_policy_estimate;
+ }
+ auto mempool_estimate = m_mempool_estimator->EstimateFeeRate(conservative);
+ if (!mempool_estimate) {
+ // A failed mempool estimate is surfaced as a warning rather than silently returning the
+ // block policy estimate, which callers can still request explicitly.
+ LogDebug(BCLog::ESTIMATEFEE, "%s", mempool_estimate.error().reason);
+ return mempool_estimate;
+ }
+ auto selected_estimate = std::min(*block_policy_estimate, *mempool_estimate);
+ LogDebug(BCLog::ESTIMATEFEE, "Fee rate estimated using %s: target=%s feerate=%s %s/kvB.",
+ FeeRateEstimatorTypeToString(selected_estimate.feerate_estimator),
+ selected_estimate.returned_target, CFeeRate(selected_estimate.feerate).GetFeePerK(), CURRENCY_ATOM);
+ return selected_estimate;
+}
+
+util::Expected<FeeRateEstimation, FeeRateEstimationError> FeeRateEstimatorManager::GetFeeRateEstimate(FeeRateEstimatorType type, int target, bool conservative) const
+{
+ switch (type) {
+ case FeeRateEstimatorType::NONE:
+ return GetFeeRateEstimate(target, conservative);
+ case FeeRateEstimatorType::BLOCK_POLICY:
+ return m_block_policy_estimator->EstimateFeeRate(target, conservative);
+ case FeeRateEstimatorType::MEMPOOL_POLICY:
+ return m_mempool_estimator->EstimateFeeRate(conservative);
+ } // no default case, so the compiler can warn about missing cases
+ assert(false);
+}
+
+void FeeRateEstimatorManager::IntervalFlush()
+{
+ m_block_policy_estimator->FlushFeeEstimates();
+ m_mempool_estimator->FlushMinedBlockStats();
+}
+
+void FeeRateEstimatorManager::ShutdownFlush()
+{
+ m_block_policy_estimator->Flush();
+ m_mempool_estimator->FlushMinedBlockStats();
+}
+
+std::vector<MinedBlockStats> FeeRateEstimatorManager::MempoolPolicyEstimatorBlocksStats() const
+{
+ return m_mempool_estimator->GetPrevBlockData();
+}
+
+void FeeRateEstimatorManager::TransactionAddedToMempool(const NewMempoolTransactionInfo& tx, uint64_t /*unused*/)
+{
+ m_block_policy_estimator->processTransaction(tx);
+}
+
+void FeeRateEstimatorManager::TransactionRemovedFromMempool(const CTransactionRef& tx, MemPoolRemovalReason /*unused*/, uint64_t /*unused*/)
+{
+ m_block_policy_estimator->removeTx(tx->GetHash());
+}
+
+void FeeRateEstimatorManager::MempoolTransactionsRemovedForBlock(const std::shared_ptr<const CBlock>& block, const std::vector<RemovedMempoolTransactionInfo>& txs_removed_for_block, unsigned int block_height)
+{
+ m_block_policy_estimator->processBlock(txs_removed_for_block, block_height);
+ m_mempool_estimator->MempoolTxsRemovedForBlock(block, txs_removed_for_block, block_height);
+}
+
+CFeeRate FeeRateEstimatorManager::BlockPolicyEstimateRawFee(unsigned int target, double threshold, FeeEstimateHorizon horizon, EstimationResult* buckets) const
+{
+ return m_block_policy_estimator->estimateRawFee(target, threshold, horizon, buckets);
+}
+
+unsigned int FeeRateEstimatorManager::BlockPolicyHighestTargetTracked(FeeEstimateHorizon horizon) const
+{
+ return m_block_policy_estimator->HighestTargetTracked(horizon);
+}
+
+unsigned int FeeRateEstimatorManager::MaximumTarget() const
+{
+ return std::max(m_block_policy_estimator->MaximumTarget(), m_mempool_estimator->MaximumTarget());
+}
### src/policy/fees/estimator_man.h
@@ -0,0 +1,102 @@
+// Copyright (c) The Bitcoin Core developers
+// Distributed under the MIT software license. See the accompanying
+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#ifndef BITCOIN_POLICY_FEES_ESTIMATOR_MAN_H
+#define BITCOIN_POLICY_FEES_ESTIMATOR_MAN_H
+
+#include <policy/fees/block_policy_estimator.h>
+#include <policy/fees/mempool_estimator.h>
+#include <primitives/transaction.h>
+#include <util/expected.h>
+#include <util/fees.h>
+#include <util/fs.h>
+#include <validationinterface.h>
+
+#include <chrono>
+#include <memory>
+
+class CFeeRate;
+class ChainstateManager;
+class CTxMemPool;
+class MemPoolFeeRateEstimator;
+
+// How often to flush data to disk
+inline constexpr std::chrono::hours FEE_FLUSH_INTERVAL{1};
+
+/** \class FeeRateEstimatorManager
+ * Manages fee rate estimators.
+ */
+class FeeRateEstimatorManager : public CValidationInterface
+{
+public:
+ /**
+ * @param[in] block_policy_path Path to the block policy fee estimates file.
+ * @param[in] read_stale_estimates Whether to load stale estimates from disk.
+ * @param[in] mempool_estimator_path Path to the mempool policy estimator data file.
+ * @param[in] mempool The mempool to use for the mempool fee rate estimator.
+ * @param[in] chainman The chainstate manager.
+ */
+ FeeRateEstimatorManager(const fs::path& block_policy_path,
+ bool read_stale_estimates,
+ const fs::path& mempool_estimator_path,
+ const CTxMemPool& mempool,
+ ChainstateManager& chainman);
+
+ virtual ~FeeRateEstimatorManager();
+
+ /**
+ * @brief Get a fee rate estimate from the available fee rate estimators.
+ * @param[in] target The target within which the transaction should be confirmed.
+ * @param[in] conservative Whether to select a more conservative, potentially higher, fee rate estimate.
+ * @return fee rate estimation, or an error on failure.
+ */
+ virtual util::Expected<FeeRateEstimation, FeeRateEstimationError> GetFeeRateEstimate(int target, bool conservative) const;
+
+ /**
+ * Like GetFeeRateEstimate, but only consults the specified estimator type.
+ * @param[in] type The estimator to query. NONE returns the manager-selected combined estimate.
+ * @param[in] target The target within which the transaction should be confirmed.
+ * @param[in] conservative Whether to select a more conservative, potentially higher, fee rate estimate.
+ * @return Fee rate estimation from the specified estimator, or an error on failure.
+ */
+ virtual util::Expected<FeeRateEstimation, FeeRateEstimationError> GetFeeRateEstimate(FeeRateEstimatorType type, int target, bool conservative) const;
+
+ /** Flush recorded data to disk. */
+ void IntervalFlush();
+
+ /** Flush recorded data to disk as part of shutdown sequence. */
+ void ShutdownFlush();
+
+ /**
+ * @brief Returns the maximum supported confirmation target from all fee rate estimators.
+ */
+ virtual unsigned int MaximumTarget() const;
+
+ /**
+ * @brief Delegate to the block policy estimator's estimateRawFee (used by the estimaterawfee RPC).
+ */
+ CFeeRate BlockPolicyEstimateRawFee(unsigned int target, double threshold, FeeEstimateHorizon horizon, EstimationResult* buckets) const;
+
+ /**
+ * @brief Returns the maximum supported confirmation target of block policy estimator.
+ */
+ unsigned int BlockPolicyHighestTargetTracked(FeeEstimateHorizon horizon) const;
+
+ /**
+ * Returns per-block weight statistics for the last MEMPOOL_HEALTH_WINDOW_BLOCKS mined blocks.
+ */
+ std::vector<MinedBlockStats> MempoolPolicyEstimatorBlocksStats() const;
+
+protected:
+ /** Overridden from CValidationInterface. */
+ void TransactionAddedToMempool(const NewMempoolTransactionInfo& tx, uint64_t /*unused*/) override;
+ void TransactionRemovedFromMempool(const CTransactionRef& tx, MemPoolRemovalReason /*unused*/, uint64_t /*unused*/) override;
+ void MempoolTransactionsRemovedForBlock(const std::shared_ptr<const CBlock>& block, const std::vector<RemovedMempoolTransactionInfo>& txs_removed_for_block, unsigned int block_height) override;
+
+private:
+ std::unique_ptr<CBlockPolicyEstimator> m_block_policy_estimator;
+ std::unique_ptr<MemPoolFeeRateEstimator> m_mempool_estimator;
+};
+
+#endif // BITCOIN_POLICY_FEES_ESTIMATOR_MAN_H
### src/policy/fees/mempool_estimator.cpp
@@ -0,0 +1,405 @@
+// Copyright (c) The Bitcoin Core developers
+// Distributed under the MIT software license. See the accompanying
+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#include <policy/fees/mempool_estimator.h>
+
+#include <logging.h>
+#include <node/miner.h>
+#include <policy/feerate.h>
+#include <policy/policy.h>
+#include <primitives/block.h>
+#include <serialize.h>
+#include <streams.h>
+#include <sync.h>
+#include <tinyformat.h>
+#include <txmempool.h>
+#include <util/check.h>
+#include <util/feefrac.h>
+#include <util/fees.h>
+#include <util/fs.h>
+#include <util/syserror.h>
+#include <validation.h>
+
+#include <algorithm>
+#include <iterator>
+#include <numeric>
+#include <optional>
+#include <string>
+#include <string_view>
+#include <system_error>
+#include <utility>
+
+constexpr int CURRENT_MEMPOOL_ESTIMATOR_VERSION{1};
+
+namespace {
+struct MinedBlockStatsFormatter {
+ template <typename Stream>
+ void Ser(Stream& s, const MinedBlockStats& v)
+ {
+ s << v.m_height << v.m_removed_block_txs_weight << v.m_block_weight;
+ }
+ template <typename Stream>
+ void Unser(Stream& s, MinedBlockStats& v)
+ {
+ s >> v.m_height >> v.m_removed_block_txs_weight >> v.m_block_weight;
+ }
+};
+
+void AddMinedBlockStats(std::vector<MinedBlockStats>& mined_blocks, MinedBlockStats stats)
+{
+ const auto stale_begin{std::find_if(mined_blocks.begin(), mined_blocks.end(), [&](const MinedBlockStats& block) {
+ return block.m_height >= stats.m_height;
+ })};
+ const auto stale_count{std::distance(stale_begin, mined_blocks.end())};
+ if (stale_count > 0) {
+ LogDebug(BCLog::ESTIMATEFEE,
+ "%s: connected block height=%s discards tracked mined-block stats "
+ "from height=%s to height=%s; stale_stats=%s",
+ FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY),
+ stats.m_height,
+ stale_begin->m_height,
+ mined_blocks.back().m_height,
+ stale_count);
+ }
+ mined_blocks.erase(stale_begin, mined_blocks.end());
+ if (!mined_blocks.empty() && mined_blocks.back().m_height + 1 != stats.m_height) {
+ LogDebug(BCLog::ESTIMATEFEE,
+ "%s: clearing mined-block stats after height gap; tracked_stats=%s "
+ "expected_height=%s received_height=%s",
+ FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY),
+ mined_blocks.size(),
+ mined_blocks.back().m_height + 1,
+ stats.m_height);
+ mined_blocks.clear();
+ }
+
+ if (mined_blocks.size() == MEMPOOL_HEALTH_WINDOW_BLOCKS) mined_blocks.erase(mined_blocks.begin());
+ mined_blocks.push_back(stats);
+}
+
+struct ActiveTip {
+ int height;
+ uint256 hash;
+};
+
+std::optional<ActiveTip> GetActiveTip(const ChainstateManager& chainman)
+{
+ LOCK(::cs_main);
+ const CBlockIndex* tip{chainman.ActiveTip()};
+ if (!tip) return std::nullopt;
+ return ActiveTip{tip->nHeight, tip->GetBlockHash()};
+}
+} // namespace
+
+MemPoolFeeRateEstimator::Percentiles MemPoolFeeRateEstimator::CalculateMaxWeightPercentiles(std::span<const FeePerVSize> chunk_feerates)
+{
+ Assume(std::is_sorted(chunk_feerates.begin(), chunk_feerates.end(), [](const auto& a, const auto& b) { return ByRatio{a} > ByRatio{b}; }));
+ constexpr int64_t total_weight{DEFAULT_BLOCK_MAX_WEIGHT};
+ const int64_t p50_weight{total_weight / 2};
+ const int64_t p75_weight{total_weight * 3 / 4};
+ Percentiles percentiles{};
+ int64_t accumulated_weight{0};
+ for (const auto& curr_feerate : chunk_feerates) {
+ accumulated_weight += int64_t{curr_feerate.size} * WITNESS_SCALE_FACTOR;
+ if (accumulated_weight >= p50_weight && percentiles.p50.IsEmpty()) {
+ percentiles.p50 = curr_feerate;
+ }
+ if (accumulated_weight >= p75_weight && percentiles.p75.IsEmpty()) {
+ percentiles.p75 = curr_feerate;
+ break;
+ }
+ }
+ return percentiles;
+}
+
+bool MemPoolFeeRateEstimatorCache::IsStale() const
+{
+ return !m_fee_rate_estimation || (m_last_updated + CACHE_LIFE) < NodeClock::now();
+}
+
+std::optional<MemPoolFeeRateEstimatorCache::FeeRateEstimate>
+MemPoolFeeRateEstimatorCache::GetCachedEstimate(const uint256& tip_hash) const
+{
+ if (IsStale() || tip_hash != m_tip_hash) return std::nullopt;
+ return m_fee_rate_estimation;
+}
+
+void MemPoolFeeRateEstimatorCache::Update(FeePerVSize conservative, FeePerVSize economical, const uint256& tip_hash)
+{
+ m_fee_rate_estimation = {conservative, economical};
+ m_tip_hash = tip_hash;
+ m_last_updated = NodeClock::now();
+}
+
+void MemPoolFeeRateEstimatorCache::Clear()
+{
+ m_fee_rate_estimation.reset();
+ m_tip_hash.SetNull();
+ m_last_updated = {};
+}
+
+//! Build the error result for a failed mempool fee rate estimation.
+static util::Unexpected<FeeRateEstimationError> EstimationError(std::string error)
+{
+ return EstimationError(FeeRateEstimatorType::MEMPOOL_POLICY, MEMPOOL_FEE_ESTIMATOR_MAX_TARGET, std::move(error));
+}
+
+static std::optional<std::string_view> MempoolHealthError(MemPoolFeeRateEstimator::MempoolHealth health)
+{
+ switch (health) {
+ case MemPoolFeeRateEstimator::MempoolHealth::INSUFFICIENT_DATA:
+ return "Not enough recent block data for fee rate estimation";
+ case MemPoolFeeRateEstimator::MempoolHealth::LOW_COVERAGE:
+ return "Mempool is unreliable for fee rate estimation";
+ case MemPoolFeeRateEstimator::MempoolHealth::HEALTHY:
+ return std::nullopt;
+ }
+ Assume(false);
+ return std::nullopt;
+}
+
+MemPoolFeeRateEstimator::MemPoolFeeRateEstimator(fs::path mempool_estimator_file_path,
+ const CTxMemPool& mempool,
+ ChainstateManager& chainman)
+ : m_mempool(mempool),
+ m_chainman(chainman),
+ m_mempool_estimator_file_path(std::move(mempool_estimator_file_path))
+{
+ ReadFromDisk();
+}
+
+void MemPoolFeeRateEstimator::ReadFromDisk()
+{
+ AutoFile file{fsbridge::fopen(m_mempool_estimator_file_path, "rb")};
+ if (file.IsNull()) {
+ LogDebug(BCLog::ESTIMATEFEE, "%s: %s does not exist. Continuing anyway",
+ FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY),
+ fs::PathToString(m_mempool_estimator_file_path));
+ return;
+ }
+ if (Read(file)) {
+ LogDebug(BCLog::ESTIMATEFEE, "%s: mined-block stats successfully read from %s.",
+ FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY),
+ fs::PathToString(m_mempool_estimator_file_path));
+ }
+}
+
+bool MemPoolFeeRateEstimator::Read(AutoFile& file)
+{
+ try {
+ int version_required;
+ file >> version_required;
+ if (version_required != CURRENT_MEMPOOL_ESTIMATOR_VERSION) {
+ LogWarning("%s: file version not supported; continuing anyway",
+ FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY));
+ return false;
+ }
+ // Stage into a local buffer and commit to the member only after validation passes.
+ std::vector<MinedBlockStats> blocks;
+ file >> Using<VectorFormatter<MinedBlockStatsFormatter>>(blocks);
+ uint256 tip_hash;
+ file >> tip_hash;
+ if (blocks.size() > MEMPOOL_HEALTH_WINDOW_BLOCKS) {
+ LogWarning("%s: Number of previously mined blocks read exceeds the maximum of %s; ignoring file",
+ FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY),
+ MEMPOOL_HEALTH_WINDOW_BLOCKS);
+ return false;
+ }
+ for (size_t i = 1; i < blocks.size(); ++i) {
+ if (blocks[i].m_height != blocks[i - 1].m_height + 1) {
+ LogWarning("%s: Non-consecutive block heights read, expected height %s but found %s; ignoring file",
+ FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY),
+ blocks[i - 1].m_height + 1, blocks[i].m_height);
+ return false;
+ }
+ }
+ if (!blocks.empty()) {
+ const auto& last_block{blocks.back()};
+ const std::optional<ActiveTip> active_tip{GetActiveTip(m_chainman)};
+ if (!active_tip) {
+ LogWarning("%s: Mined-block stats read end at height %s block %s, but there is no active chain tip; ignoring file",
+ FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY),
+ last_block.m_height, tip_hash.ToString());
+ return false;
+ }
+ if (last_block.m_height != static_cast<uint64_t>(active_tip->height) || tip_hash != active_tip->hash) {
+ LogWarning("%s: Mined-block stats read end at height %s block %s, but the active chain tip is height %s block %s; ignoring file",
+ FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY),
+ last_block.m_height, tip_hash.ToString(),
+ active_tip->height, active_tip->hash.ToString());
+ return false;
+ }
+ }
+ LOCK(cs);
+ m_prev_mined_blocks = std::move(blocks);
+ m_mined_blocks_tip_hash = tip_hash;
+ m_cache.Clear();
+ } catch (const std::exception&) {
+ LogWarning("%s: Unable to read mined-block stats from stream (non-fatal)",
+ FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY));
+ return false;
+ }
+ return true;
+}
+
+bool MemPoolFeeRateEstimator::Write(AutoFile& file) const
+{
+ try {
+ LOCK(cs);
+ file << CURRENT_MEMPOOL_ESTIMATOR_VERSION;
+ file << Using<VectorFormatter<MinedBlockStatsFormatter>>(m_prev_mined_blocks);
+ file << m_mined_blocks_tip_hash;
+ } catch (const std::exception&) {
+ return false;
+ }
+ return true;
+}
+
+void MemPoolFeeRateEstimator::FlushMinedBlockStats()
+{
+ if (!m_mempool_estimator_file_path.parent_path().empty()) {
+ std::error_code error;
+ fs::create_directories(m_mempool_estimator_file_path.parent_path(), error);
+ if (error) {
+ LogWarning("%s: failed to create mempool policy estimator directory %s: %s. Continuing anyway",
+ FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY),
+ fs::PathToString(m_mempool_estimator_file_path.parent_path()), error.message());
+ return;
+ }
+ }
+ AutoFile file{fsbridge::fopen(m_mempool_estimator_file_path, "wb")};
+ if (file.IsNull()) {
+ LogWarning("%s: unable to open %s for writing. Continuing anyway",
+ FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY),
+ fs::PathToString(m_mempool_estimator_file_path));
+ return;
+ }
+ if (!Write(file)) {
+ LogWarning("%s: Unable to write mined-block stats to %s (non-fatal)",
+ FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY),
+ fs::PathToString(m_mempool_estimator_file_path));
+ }
+ if (file.fclose() != 0) {
+ LogWarning("Failed to close mempool policy estimator file %s: %s. Continuing anyway.",
+ fs::PathToString(m_mempool_estimator_file_path), SysErrorString(errno));
+ return;
+ }
+ LogDebug(BCLog::ESTIMATEFEE, "%s: mined-block stats flushed to %s.",
+ FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY),
+ fs::PathToString(m_mempool_estimator_file_path));
+}
+
+
+void MemPoolFeeRateEstimator::MempoolTxsRemovedForBlock(const std::shared_ptr<const CBlock>& block,
+ const std::vector<RemovedMempoolTransactionInfo>& txs_removed_for_block,
+ unsigned int block_height)
+{
+ LOCK(cs);
+ Assert(!block->vtx.empty());
+ // Accumulate total block weight and removed mempool tx weight, both excluding the coinbase.
+ const auto get_tx_weight = [](const CTransactionRef& tx) {
+ return static_cast<uint64_t>(GetTransactionWeight(*tx));
+ };
+ // Skip vtx[0], which is the coinbase.
+ const uint64_t block_weight = std::accumulate(std::next(block->vtx.begin()), block->vtx.end(), uint64_t{0},
+ [&](uint64_t acc, const CTransactionRef& tx) {
+ return acc + get_tx_weight(tx);
+ });
+ const uint64_t removed_weight = std::accumulate(
+ txs_removed_for_block.begin(), txs_removed_for_block.end(), uint64_t{0},
+ [&](uint64_t acc, const RemovedMempoolTransactionInfo& tx) {
+ return acc + get_tx_weight(tx.info.m_tx);
+ });
+ AddMinedBlockStats(m_prev_mined_blocks, {block_height, removed_weight, block_weight});
+ m_mined_blocks_tip_hash = block->GetHash();
+ m_cache.Clear();
+}
+
+// Require at least one block worth of activity across the window before using
+// the coverage ratio as a representative mempool health signal.
+static constexpr uint64_t MIN_REPRESENTATIVE_WINDOW_WEIGHT{DEFAULT_BLOCK_MAX_WEIGHT};
+
+MemPoolFeeRateEstimator::MempoolHealth MemPoolFeeRateEstimator::GetMempoolHealth() const
+{
+ LOCK(cs);
+ const auto estimator_name{FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY)};
+ if (m_prev_mined_blocks.size() < MEMPOOL_HEALTH_WINDOW_BLOCKS) {
+ LogDebug(BCLog::ESTIMATEFEE, "%s: mempool health check failed; tracked_blocks=%s required_blocks=%s",
+ estimator_name, m_prev_mined_blocks.size(), MEMPOOL_HEALTH_WINDOW_BLOCKS);
+ return MempoolHealth::INSUFFICIENT_DATA;
+ }
+ uint64_t total_block_weight{0};
+ uint64_t total_removed_weight{0};
+ uint64_t expected_height{m_prev_mined_blocks.front().m_height};
+ for (const auto& block : m_prev_mined_blocks) {
+ Assume(block.m_height == expected_height);
+ ++expected_height;
+ total_block_weight += block.m_block_weight;
+ total_removed_weight += block.m_removed_block_txs_weight;
+ }
+ // Too little block activity for the coverage ratio to be meaningful; skip it.
+ if (total_block_weight < MIN_REPRESENTATIVE_WINDOW_WEIGHT) {
+ LogDebug(BCLog::ESTIMATEFEE, "%s: mempool health check passed; low activity, total_block_weight=%s minimum=%s",
+ estimator_name, total_block_weight, MIN_REPRESENTATIVE_WINDOW_WEIGHT);
+ return MempoolHealth::HEALTHY;
+ }
+ const double representation_ratio = static_cast<double>(total_removed_weight) / total_block_weight;
+ LogDebug(BCLog::ESTIMATEFEE,
+ "%s: mempool health check %s; removed_weight=%s total_block_weight=%s "
+ "coverage=%.2f required_coverage=%.2f",
+ estimator_name,
+ representation_ratio >= MEMPOOL_REPRESENTATION_THRESHOLD ? "passed" : "failed",
+ total_removed_weight,
+ total_block_weight,
+ representation_ratio,
+ MEMPOOL_REPRESENTATION_THRESHOLD);
+ return representation_ratio >= MEMPOOL_REPRESENTATION_THRESHOLD ? MempoolHealth::HEALTHY : MempoolHealth::LOW_COVERAGE;
+}
+
+util::Expected<FeeRateEstimation, FeeRateEstimationError> MemPoolFeeRateEstimator::EstimateFeeRate(bool conservative) const
+{
+ constexpr auto estimator_type{FeeRateEstimatorType::MEMPOOL_POLICY};
+ if (!m_mempool.GetLoadTried()) {
+ return EstimationError(strprintf("%s: Mempool not loaded yet, no fee rate estimate available", FeeRateEstimatorTypeToString(estimator_type)));
+ }
+ if (auto error{MempoolHealthError(GetMempoolHealth())}) {
+ return EstimationError(strprintf("%s: %s", FeeRateEstimatorTypeToString(estimator_type), *error));
+ }
+ // The estimator lock is not held while building a block template, so
+ // in a rare edge case concurrent callers may duplicate work.
+ //
+ // Cached fee rate estimates are tagged with the chain tip they were computed on
+ // and only served from the cache while that tip is current.
+ //
+ // The fee rate estimate returned directly below may still reflect a tip that went
+ // stale during the call; that is an accepted tradeoff of not holding
+ // locks across block assembly.
+ {
+ const uint256 tip_hash{WITH_LOCK(::cs_main, return Assume(m_chainman.CurrentChainstate().m_chain.Tip())->GetBlockHash())};
+ LOCK(cs);
+ const auto cached_estimate = m_cache.GetCachedEstimate(tip_hash);
+ if (cached_estimate) {
+ const auto cached_feerate{
+ conservative ? cached_estimate->m_conservative : cached_estimate->m_economical};
+ return FeeRateEstimation{estimator_type, cached_feerate, MEMPOOL_FEE_ESTIMATOR_MAX_TARGET};
+ }
+ }
+ node::BlockCreateOptions options;
+ options.test_block_validity = false;
+ const auto blocktemplate = WITH_LOCK(::cs_main, return (node::BlockAssembler{m_chainman.CurrentChainstate(), &m_mempool, options}).CreateNewBlock());
+ if (!blocktemplate) return EstimationError(strprintf("%s: Failed to create block template for fee rate estimation", FeeRateEstimatorTypeToString(estimator_type)));
+ // Sort again because the rounding up when converting from weight to vsize may cause slight misorder.
+ std::sort(blocktemplate->m_package_feerates.begin(), blocktemplate->m_package_feerates.end(), [](const auto& a, const auto& b) { return ByRatio{a} > ByRatio{b}; });
+ const auto percentiles = CalculateMaxWeightPercentiles(blocktemplate->m_package_feerates);
+ // Fall back to a relayable floor (the higher of the min relay fee and the current
+ // mempool min fee) for any percentile the mempool was too sparse to fill.
+ const FeePerVSize floor{std::max(m_mempool.m_opts.min_relay_feerate, m_mempool.GetMinFee()).GetFeePerVSize()};
+ const FeePerVSize p50{percentiles.p50.IsEmpty() ? floor : percentiles.p50};
+ const FeePerVSize p75{percentiles.p75.IsEmpty() ? floor : percentiles.p75};
+ WITH_LOCK(cs, m_cache.Update(p50, p75, blocktemplate->block.hashPrevBlock));
+ LogDebug(BCLog::ESTIMATEFEE, "%s: conservative/economical fee rate: %s/%s %s/kvB",
+ FeeRateEstimatorTypeToString(estimator_type), CFeeRate(p50).GetFeePerK(),
+ CFeeRate(p75).GetFeePerK(), CURRENCY_ATOM);
+ return FeeRateEstimation{estimator_type, conservative ? p50 : p75, MEMPOOL_FEE_ESTIMATOR_MAX_TARGET};
+}
### src/policy/fees/mempool_estimator.h
@@ -0,0 +1,157 @@
+// Copyright (c) The Bitcoin Core developers
+// Distributed under the MIT software license. See the accompanying
+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#ifndef BITCOIN_POLICY_FEES_MEMPOOL_ESTIMATOR_H
+#define BITCOIN_POLICY_FEES_MEMPOOL_ESTIMATOR_H
+
+#include <primitives/transaction.h>
+#include <sync.h>
+#include <threadsafety.h>
+#include <uint256.h>
+#include <util/expected.h>
+#include <util/feefrac.h>
+#include <util/fees.h>
+#include <util/fs.h>
+#include <util/time.h>
+
+#include <chrono>
+#include <memory>
+#include <optional>
+#include <span>
+#include <vector>
+
+class CBlock;
+class AutoFile;
+class ChainstateManager;
+class CTxMemPool;
+
+struct RemovedMempoolTransactionInfo;
+
+// Fee rate estimate for confirmation target above this is not reliable,
+// as mempool conditions are likely to change.
+constexpr int MEMPOOL_FEE_ESTIMATOR_MAX_TARGET{2};
+constexpr std::chrono::seconds CACHE_LIFE{7};
+
+// Constants for mempool sanity checks.
+constexpr size_t MEMPOOL_HEALTH_WINDOW_BLOCKS = 6;
+constexpr double MEMPOOL_REPRESENTATION_THRESHOLD = 0.75;
+
+//! Weight statistics for a recently mined block, used to assess mempool coverage.
+struct MinedBlockStats {
+ //! Block height.
+ uint64_t m_height{0};
+ //! Weight of mempool transactions removed for this block (excluding coinbase).
+ uint64_t m_removed_block_txs_weight{0};
+ //! Total non-coinbase transaction weight in the block.
+ uint64_t m_block_weight{0};
+};
+
+/**
+ * MemPoolFeeRateEstimatorCache holds a cache of recent fee rate estimates.
+ * A cached fee rate is only provided while it is not older than CACHE_LIFE
+ * and the chain tip has not changed.
+ */
+class MemPoolFeeRateEstimatorCache
+{
+public:
+ MemPoolFeeRateEstimatorCache() = default;
+ MemPoolFeeRateEstimatorCache(const MemPoolFeeRateEstimatorCache&) = delete;
+ MemPoolFeeRateEstimatorCache& operator=(const MemPoolFeeRateEstimatorCache&) = delete;
+ /** Returns true if the cache is empty or older than CACHE_LIFE. */
+ bool IsStale() const;
+ struct FeeRateEstimate {
+ FeePerVSize m_conservative;
+ FeePerVSize m_economical;
+ };
+ /** Returns cached estimates if not stale and computed on tip_hash, nullopt otherwise. */
+ std::optional<FeeRateEstimate> GetCachedEstimate(const uint256& tip_hash) const;
+ /** Update the cache with new estimates computed on tip_hash. */
+ void Update(FeePerVSize conservative, FeePerVSize economical, const uint256& tip_hash);
+ /** Clear cached fee rate estimates. */
+ void Clear();
+
+private:
+ std::optional<FeeRateEstimate> m_fee_rate_estimation;
+ uint256 m_tip_hash;
+ NodeClock::time_point m_last_updated{};
+};
+
+/**
+ * Estimate the fee rate required for a transaction to be included in the next block.
+ *
+ * Uses Bitcoin Core's block-building algorithm to generate a block template from the mempool,
+ * then calculates percentile fee rates from the selected chunks: the 75th percentile is returned
+ * as the economical estimate and the 50th percentile as the conservative estimate.
+ */
+class MemPoolFeeRateEstimator
+{
+public:
+ // Block percentiles fee rate (in sat/vB).
+ struct Percentiles {
+ FeePerVSize p50;
+ FeePerVSize p75;
+ };
+
+ MemPoolFeeRateEstimator(fs::path mempool_estimator_file_path,
+ const CTxMemPool& mempool,
+ ChainstateManager& chainman);
+ ~MemPoolFeeRateEstimator() = default;
+ /**
+ * Calculate the 50th and 75th percentile fee rates from block template chunks,
+ * sorted in descending mining-score order. A percentile is left empty when the
+ * chunks cannot cover the corresponding fraction of a block.
+ *
+ * @param[in] chunk_feerates Block template chunk fee rates sorted by descending mining score.
+ */
+ static Percentiles CalculateMaxWeightPercentiles(std::span<const FeePerVSize> chunk_feerates);
+ util::Expected<FeeRateEstimation, FeeRateEstimationError> EstimateFeeRate(bool conservative) const
+ EXCLUSIVE_LOCKS_REQUIRED(!cs);
+ unsigned int MaximumTarget() const
+ {
+ return MEMPOOL_FEE_ESTIMATOR_MAX_TARGET;
+ }
+
+ std::vector<MinedBlockStats> GetPrevBlockData() const EXCLUSIVE_LOCKS_REQUIRED(!cs)
+ {
+ LOCK(cs);
+ return m_prev_mined_blocks;
+ }
+
+ void MempoolTxsRemovedForBlock(const std::shared_ptr<const CBlock>& block,
+ const std::vector<RemovedMempoolTransactionInfo>& txs_removed_for_block,
+ unsigned int block_height)
+ EXCLUSIVE_LOCKS_REQUIRED(!cs);
+ //! Health of the recent mined-block window for fee rate estimation.
+ enum class MempoolHealth {
+ //! Recent blocks represent the mempool well enough to estimate a fee rate.
+ HEALTHY,
+ //! Too few recent mined blocks to estimate a fee rate.
+ INSUFFICIENT_DATA,
+ //! Recent blocks include too few mempool transactions to estimate a fee rate.
+ LOW_COVERAGE,
+ };
+ MempoolHealth GetMempoolHealth() const EXCLUSIVE_LOCKS_REQUIRED(!cs);
+ //! Checks if recent mined blocks indicate a healthy mempool state.
+ bool IsMempoolHealthy() const EXCLUSIVE_LOCKS_REQUIRED(!cs) { return GetMempoolHealth() == MempoolHealth::HEALTHY; }
+ void FlushMinedBlockStats() EXCLUSIVE_LOCKS_REQUIRED(!cs);
+ //! Deserialize mined-block stats without taking ownership of file.
+ bool Read(AutoFile& file) EXCLUSIVE_LOCKS_REQUIRED(!cs);
+ //! Serialize mined-block stats without taking ownership of file.
+ //! Callers must explicitly close file and check for errors after writing.
+ bool Write(AutoFile& file) const EXCLUSIVE_LOCKS_REQUIRED(!cs);
+
+private:
+ void ReadFromDisk() EXCLUSIVE_LOCKS_REQUIRED(!cs);
+ //! Tracks weight statistics for the last MEMPOOL_HEALTH_WINDOW_BLOCKS mined blocks.
+ std::vector<MinedBlockStats> m_prev_mined_blocks GUARDED_BY(cs);
+ uint256 m_mined_blocks_tip_hash GUARDED_BY(cs);
+
+ const CTxMemPool& m_mempool;
+ ChainstateManager& m_chainman;
+ mutable Mutex cs;
+ mutable MemPoolFeeRateEstimatorCache m_cache GUARDED_BY(cs);
+ const fs::path m_mempool_estimator_file_path;
+};
+
+#endif // BITCOIN_POLICY_FEES_MEMPOOL_ESTIMATOR_H
### src/qt/sendcoinsdialog.cpp
@@ -21,17 +21,18 @@
#include <key_io.h>
#include <node/interface_ui.h>
#include <node/types.h>
-#include <policy/fees/block_policy_estimator.h>
#include <txmempool.h>
#include <validation.h>
#include <wallet/coincontrol.h>
#include <wallet/fees.h>
+#include <wallet/types.h>
#include <wallet/wallet.h>
#include <array>
#include <chrono>
#include <fstream>
#include <memory>
+#include <optional>
#include <QFontMetrics>
#include <QScrollBar>
@@ -837,7 +838,7 @@ void SendCoinsDialog::updateSmartFeeLabel()
return;
updateCoinControlState();
m_coin_control->m_feerate.reset(); // Explicitly use only fee estimation rate for smart fee labels
- int returned_target;
+ std::optional<int> returned_target;
FeeReason reason;
CFeeRate feeRate = CFeeRate(model->wallet().getMinimumFee(1000, *m_coin_control, &returned_target, &reason));
@@ -855,7 +856,10 @@ void SendCoinsDialog::updateSmartFeeLabel()
else
{
ui->labelSmartFee2->hide();
- ui->labelFeeEstimation->setText(tr("Estimated to begin confirmation within %n block(s).", "", returned_target));
+ ui->labelFeeEstimation->setText("");
+ if (returned_target) {
+ ui->labelFeeEstimation->setText(tr("Estimated to begin confirmation within %n block(s).", "", *returned_target));
+ }
ui->fallbackFeeWarningLabel->setVisible(false);
}
### src/rpc/client.cpp
@@ -305,6 +305,7 @@ static const CRPCConvertParam vRPCConvertParams[] =
{ "getrawmempool", 1, "mempool_sequence" },
{ "getorphantxs", 0, "verbosity" },
{ "estimatesmartfee", 0, "conf_target" },
+ { "estimatesmartfee", 2, "options" },
{ "estimaterawfee", 0, "conf_target" },
{ "estimaterawfee", 1, "threshold" },
{ "prioritisetransaction", 1, "dummy" },
### src/rpc/fees.cpp
@@ -8,6 +8,7 @@
#include <node/context.h>
#include <policy/feerate.h>
#include <policy/fees/block_policy_estimator.h>
+#include <policy/fees/estimator_man.h>
#include <rpc/protocol.h>
#include <rpc/request.h>
#include <rpc/server.h>
@@ -41,54 +42,100 @@ static RPCMethod estimatesmartfee()
{"conf_target", RPCArg::Type::NUM, RPCArg::Optional::NO, "Confirmation target in blocks (1 - 1008)"},
{"estimate_mode", RPCArg::Type::STR, RPCArg::Default{"economical"}, "The fee estimate mode.\n"
+ FeeModesDetail(std::string("default mode will be used"))},
+ {"options", RPCArg::Type::OBJ, RPCArg::Optional::OMITTED, "",
+ {
+ {"fee_rate_estimator", RPCArg::Type::STR, RPCArg::Default{"none"},
+ "Selects which fee rate estimator to use.\n"
+ "\"none\" returns the lower of the block policy and mempool estimates. If the mempool\n"
+ "estimate is unavailable, it returns that error instead of falling back to the block\n"
+ "policy estimate; use \"block_policy\" in that case to get the block policy estimate.\n"
+ "\"block_policy\" uses only the block policy fee rate estimator.\n"
+ "\"mempool_policy\" uses only the mempool fee rate estimator.\n"
+ "Unknown values are treated as \"none\"."},
+ {"verbosity", RPCArg::Type::NUM, RPCArg::Default{1},
+ "1 returns feerate or errors. 2 also returns \"mempool_health_statistics\"."},
+ },
+ },
},
RPCResult{
RPCResult::Type::OBJ, "", "",
{
{RPCResult::Type::NUM, "feerate", /*optional=*/true, "estimate fee rate in " + CURRENCY_UNIT + "/kvB (only present if no errors were encountered)"},
+ {RPCResult::Type::STR, "estimator", /*optional=*/true, "the fee estimator used to produce the result (only present for successful estimates when fee_rate_estimator is \"none\")"},
{RPCResult::Type::ARR, "errors", /*optional=*/true, "Errors encountered during processing (if there are any)",
{
{RPCResult::Type::STR, "", "error"},
}},
- {RPCResult::Type::NUM, "blocks", "block number where estimate was found\n"
- "The request target will be clamped between 2 and the highest target\n"
- "fee estimation is able to return based on how long it has been running.\n"
- "An error is returned if not enough transactions and blocks\n"
- "have been observed to make an estimate for any number of blocks."},
+ {RPCResult::Type::NUM, "blocks", "the confirmation target in blocks for the returned fee rate estimate.\n"
+ "For the block policy fee rate estimator, this is the target the estimate was found at, clamped to at\n"
+ "least 2 and at most the estimator's maximum usable target. For the mempool fee rate\n"
+ "estimator, it is always 2."},
+ {RPCResult::Type::ARR, "mempool_health_statistics", /*optional=*/true, "Health statistics for the most recently mined blocks tracked by the mempool fee rate estimator (only present when verbosity >= 2)",
+ {
+ {RPCResult::Type::OBJ, "", "",
+ {
+ {RPCResult::Type::NUM, "block_height", "Block height"},
+ {RPCResult::Type::NUM, "block_weight", "Total weight of non-coinbase transactions in the block"},
+ {RPCResult::Type::NUM, "mempool_txs_weight", "Total weight of transactions removed from the mempool for this block"},
+ }},
+ }},
}},
RPCExamples{
HelpExampleCli("estimatesmartfee", "6") +
HelpExampleRpc("estimatesmartfee", "6")
},
[](const RPCMethod& self, const JSONRPCRequest& request) -> UniValue
{
- CBlockPolicyEstimator& fee_estimator = EnsureAnyFeeEstimator(request.context);
+ FeeRateEstimatorManager& fee_estimator_man = EnsureAnyFeeEstimatorMan(request.context);
const NodeContext& node = EnsureAnyNodeContext(request.context);
const CTxMemPool& mempool = EnsureMemPool(node);
CHECK_NONFATAL(mempool.m_opts.signals)->SyncWithValidationInterfaceQueue();
- unsigned int max_target = fee_estimator.HighestTargetTracked(FeeEstimateHorizon::LONG_HALFLIFE);
+ unsigned int max_target = fee_estimator_man.MaximumTarget();
unsigned int conf_target = ParseConfirmTarget(request.params[0], max_target);
FeeEstimateMode fee_mode;
if (!FeeModeFromString(self.Arg<std::string_view>("estimate_mode"), fee_mode)) {
throw JSONRPCError(RPC_INVALID_PARAMETER, InvalidEstimateModeErrorMessage());
}
-
+ const UniValue options{request.params[2].isNull() ? UniValue::VOBJ : request.params[2]};
+ RPCTypeCheckObj(options,
+ {
+ {"fee_rate_estimator", UniValueType(UniValue::VSTR)},
+ {"verbosity", UniValueType(UniValue::VNUM)},
+ }, /*fAllowNull=*/true, /*fStrict=*/true);
+ const auto fee_rate_estimator{FeeRateEstimatorTypeFromString(
+ options["fee_rate_estimator"].isNull() ? "none" : options["fee_rate_estimator"].get_str())};
+ bool conservative{fee_mode == FeeEstimateMode::CONSERVATIVE};
+ int verbosity{ParseVerbosity(options["verbosity"], /*default_verbosity=*/1, /*allow_bool=*/false)};
UniValue result(UniValue::VOBJ);
UniValue errors(UniValue::VARR);
- FeeCalculation feeCalc;
- bool conservative{fee_mode == FeeEstimateMode::CONSERVATIVE};
- CFeeRate feeRate{fee_estimator.estimateSmartFee(conf_target, &feeCalc, conservative)};
- if (feeRate != CFeeRate(0)) {
- CFeeRate min_mempool_feerate{mempool.GetMinFee()};
- CFeeRate min_relay_feerate{mempool.m_opts.min_relay_feerate};
- feeRate = std::max({feeRate, min_mempool_feerate, min_relay_feerate});
- result.pushKV("feerate", ValueFromAmount(feeRate.GetFeePerK()));
+ const auto estimate{fee_estimator_man.GetFeeRateEstimate(fee_rate_estimator, conf_target, conservative)};
+ if (estimate) {
+ const CFeeRate min_mempool_feerate{mempool.GetMinFee()};
+ const CFeeRate min_relay_feerate{mempool.m_opts.min_relay_feerate};
+ const auto fee_rate{std::max({CFeeRate(estimate->feerate), min_mempool_feerate, min_relay_feerate})};
+ result.pushKV("feerate", ValueFromAmount(fee_rate.GetFeePerK()));
} else {
- errors.push_back("Insufficient data or no feerate found");
+ errors.push_back(estimate.error().reason);
result.pushKV("errors", std::move(errors));
}
- result.pushKV("blocks", feeCalc.returnedTarget);
+ if (estimate && fee_rate_estimator == FeeRateEstimatorType::NONE) {
+ result.pushKV("estimator", FeeRateEstimatorTypeToString(estimate->feerate_estimator));
+ }
+ const FeeRateEstimation& estimation{FeeRateEstimationRef(estimate)};
+ result.pushKV("blocks", estimation.returned_target);
+ if (verbosity >= 2) {
+ UniValue mempool_health_stats(UniValue::VARR);
+ const auto blocks_data = fee_estimator_man.MempoolPolicyEstimatorBlocksStats();
+ for (auto it = blocks_data.rbegin(); it != blocks_data.rend(); ++it) {
+ UniValue entry(UniValue::VOBJ);
+ entry.pushKV("block_height", it->m_height);
+ entry.pushKV("block_weight", it->m_block_weight);
+ entry.pushKV("mempool_txs_weight", it->m_removed_block_txs_weight);
+ mempool_health_stats.push_back(std::move(entry));
+ }
+ result.pushKV("mempool_health_statistics", std::move(mempool_health_stats));
+ }
return result;
},
};
@@ -155,11 +202,11 @@ static RPCMethod estimaterawfee()
},
[](const RPCMethod& self, const JSONRPCRequest& request) -> UniValue
{
- CBlockPolicyEstimator& fee_estimator = EnsureAnyFeeEstimator(request.context);
+ FeeRateEstimatorManager& fee_estimator_man = EnsureAnyFeeEstimatorMan(request.context);
const NodeContext& node = EnsureAnyNodeContext(request.context);
CHECK_NONFATAL(node.validation_signals)->SyncWithValidationInterfaceQueue();
- unsigned int max_target = fee_estimator.HighestTargetTracked(FeeEstimateHorizon::LONG_HALFLIFE);
+ unsigned int max_target = fee_estimator_man.MaximumTarget();
unsigned int conf_target = ParseConfirmTarget(request.params[0], max_target);
double threshold = 0.95;
if (!request.params[1].isNull()) {
@@ -176,9 +223,9 @@ static RPCMethod estimaterawfee()
EstimationResult buckets;
// Only output results for horizons which track the target
- if (conf_target > fee_estimator.HighestTargetTracked(horizon)) continue;
+ if (conf_target > fee_estimator_man.BlockPolicyHighestTargetTracked(horizon)) continue;
- feeRate = fee_estimator.estimateRawFee(conf_target, threshold, horizon, &buckets);
+ feeRate = fee_estimator_man.BlockPolicyEstimateRawFee(conf_target, threshold, horizon, &buckets);
UniValue horizon_result(UniValue::VOBJ);
UniValue errors(UniValue::VARR);
UniValue passbucket(UniValue::VOBJ);
### src/rpc/server_util.cpp
@@ -9,7 +9,7 @@
#include <net_processing.h>
#include <node/context.h>
#include <node/miner.h>
-#include <policy/fees/block_policy_estimator.h>
+#include <policy/fees/estimator_man.h>
#include <pow.h>
#include <rpc/protocol.h>
#include <rpc/request.h>
@@ -84,17 +84,17 @@ ChainstateManager& EnsureAnyChainman(const std::any& context)
return EnsureChainman(EnsureAnyNodeContext(context));
}
-CBlockPolicyEstimator& EnsureFeeEstimator(const NodeContext& node)
+FeeRateEstimatorManager& EnsureFeeEstimatorMan(const NodeContext& node)
{
- if (!node.fee_estimator) {
+ if (!node.fee_estimator_man) {
throw JSONRPCError(RPC_INTERNAL_ERROR, "Fee estimation disabled");
}
- return *node.fee_estimator;
+ return *node.fee_estimator_man;
}
-CBlockPolicyEstimator& EnsureAnyFeeEstimator(const std::any& context)
+FeeRateEstimatorManager& EnsureAnyFeeEstimatorMan(const std::any& context)
{
- return EnsureFeeEstimator(EnsureAnyNodeContext(context));
+ return EnsureFeeEstimatorMan(EnsureAnyNodeContext(context));
}
CConnman& EnsureConnman(const NodeContext& node)
### src/rpc/server_util.h
@@ -12,7 +12,7 @@
class AddrMan;
class ArgsManager;
class CBlockIndex;
-class CBlockPolicyEstimator;
+class FeeRateEstimatorManager;
class CConnman;
class CTxMemPool;
class ChainstateManager;
@@ -34,8 +34,8 @@ ArgsManager& EnsureArgsman(const node::NodeContext& node);
ArgsManager& EnsureAnyArgsman(const std::any& context);
ChainstateManager& EnsureChainman(const node::NodeContext& node);
ChainstateManager& EnsureAnyChainman(const std::any& context);
-CBlockPolicyEstimator& EnsureFeeEstimator(const node::NodeContext& node);
-CBlockPolicyEstimator& EnsureAnyFeeEstimator(const std::any& context);
+FeeRateEstimatorManager& EnsureFeeEstimatorMan(const node::NodeContext& node);
+FeeRateEstimatorManager& EnsureAnyFeeEstimatorMan(const std::any& context);
CConnman& EnsureConnman(const node::NodeContext& node);
interfaces::Mining& EnsureMining(const node::NodeContext& node);
PeerManager& EnsurePeerman(const node::NodeContext& node);
### src/test/CMakeLists.txt
@@ -24,6 +24,7 @@ add_executable(test_bitcoin
blockfilter_index_tests.cpp
blockfilter_tests.cpp
blockmanager_tests.cpp
+ blockpolicyestimator_tests.cpp
bloom_tests.cpp
bswap_tests.cpp
btcsignals_tests.cpp
@@ -44,6 +45,7 @@ add_executable(test_bitcoin
denialofservice_tests.cpp
descriptor_tests.cpp
disconnected_transactions.cpp
+ fees_util_tests.cpp
feefrac_tests.cpp
feerounder_tests.cpp
flatfile_tests.cpp
@@ -58,6 +60,7 @@ add_executable(test_bitcoin
key_tests.cpp
logging_tests.cpp
mempool_tests.cpp
+ mempool_fee_estimator_tests.cpp
merkle_tests.cpp
merkleblock_tests.cpp
miner_tests.cpp
@@ -76,7 +79,6 @@ add_executable(test_bitcoin
pcp_tests.cpp
peerman_tests.cpp
pmt_tests.cpp
- policyestimator_tests.cpp
pool_tests.cpp
pow_tests.cpp
prevector_tests.cpp
### src/test/blockpolicyestimator_tests.cpp
@@ -3,25 +3,22 @@
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#include <policy/fees/block_policy_estimator.h>
-#include <policy/fees/block_policy_estimator_args.h>
+#include <policy/fees/estimator_args.h>
#include <policy/policy.h>
+#include <test/util/setup_common.h>
#include <test/util/txmempool.h>
#include <txmempool.h>
#include <uint256.h>
#include <util/time.h>
#include <validationinterface.h>
-#include <test/util/setup_common.h>
-
#include <boost/test/unit_test.hpp>
-BOOST_FIXTURE_TEST_SUITE(policyestimator_tests, ChainTestingSetup)
+BOOST_FIXTURE_TEST_SUITE(blockpolicyestimator_tests, ChainTestingSetup)
BOOST_AUTO_TEST_CASE(BlockPolicyEstimates)
{
- CBlockPolicyEstimator feeEst{FeeestPath(*m_node.args), DEFAULT_ACCEPT_STALE_FEE_ESTIMATES};
- CTxMemPool& mpool = *Assert(m_node.mempool);
- m_node.validation_signals->RegisterValidationInterface(&feeEst);
+ CBlockPolicyEstimator feeEst{BlockPolicyFeeEstPath(*m_node.args), DEFAULT_ACCEPT_STALE_FEE_ESTIMATES};
TestMemPoolEntryHelper entry;
CAmount basefee(2000);
CAmount deltaFee(100);
@@ -35,9 +32,9 @@ BOOST_AUTO_TEST_CASE(BlockPolicyEstimates)
// Store the hashes of transactions that have been
// added to the mempool by their associate fee
- // txHashes[j] is populated with transactions either of
+ // mempool_txs[j] is populated with transactions either of
// fee = basefee * (j+1)
- std::vector<Txid> txHashes[10];
+ std::list<CTxMemPoolEntry> mempool_txs[10];
// Create a transaction template
CScript garbage;
@@ -51,7 +48,7 @@ BOOST_AUTO_TEST_CASE(BlockPolicyEstimates)
CFeeRate baseRate(basefee, GetVirtualTransactionSize(CTransaction(tx)));
// Create a fake block
- std::vector<CTransactionRef> block;
+ std::vector<RemovedMempoolTransactionInfo> block_txs;
int blocknum = 0;
// Loop through 200 blocks
@@ -61,48 +58,36 @@ BOOST_AUTO_TEST_CASE(BlockPolicyEstimates)
for (int j = 0; j < 10; j++) { // For each fee
for (int k = 0; k < 4; k++) { // add 4 fee txs
tx.vin[0].prevout.n = 10000*blocknum+100*j+k; // make transaction unique
- {
- LOCK2(cs_main, mpool.cs);
- TryAddToMempool(mpool, entry.Fee(feeV[j]).Time(Now<NodeSeconds>()).Height(blocknum).FromTx(tx));
- // Since TransactionAddedToMempool callbacks are generated in ATMP,
- // not TryAddToMempool, we cheat and create one manually here
- const int64_t virtual_size = GetVirtualTransactionSize(*MakeTransactionRef(tx));
- const NewMempoolTransactionInfo tx_info{NewMempoolTransactionInfo(MakeTransactionRef(tx),
- feeV[j],
- virtual_size,
- entry.nHeight,
- /*mempool_limit_bypassed=*/false,
- /*submitted_in_package=*/false,
- /*chainstate_is_current=*/true,
- /*has_no_mempool_parents=*/true)};
- m_node.validation_signals->TransactionAddedToMempool(tx_info, mpool.GetAndIncrementSequence());
- }
- txHashes[j].push_back(tx.GetHash());
+ // Simulate the tx being added to the mempool by calling processTransaction(tx_info)
+ mempool_txs[j].emplace_back(entry.Fee(feeV[j]).Time(Now<NodeSeconds>()).Height(blocknum).FromTx(tx));
+ const int64_t virtual_size = GetVirtualTransactionSize(*MakeTransactionRef(tx));
+ const NewMempoolTransactionInfo tx_info{NewMempoolTransactionInfo(MakeTransactionRef(tx),
+ feeV[j],
+ virtual_size,
+ entry.nHeight,
+ /*mempool_limit_bypassed=*/false,
+ /*submitted_in_package=*/false,
+ /*chainstate_is_current=*/true,
+ /*has_no_mempool_parents=*/true)};
+ feeEst.processTransaction(tx_info);
}
}
//Create blocks where higher fee txs are included more often
for (int h = 0; h <= blocknum%10; h++) {
// 10/10 blocks add highest fee transactions
// 9/10 blocks add 2nd highest and so on until ...
// 1/10 blocks add lowest fee transactions
- while (txHashes[9-h].size()) {
- CTransactionRef ptx = mpool.get(txHashes[9-h].back());
- if (ptx)
- block.push_back(ptx);
- txHashes[9-h].pop_back();
+ while (mempool_txs[9 - h].size()) {
+ auto& tx_entry = mempool_txs[9 - h].back();
+ block_txs.emplace_back(tx_entry);
+ mempool_txs[9 - h].pop_back();
}
}
- {
- LOCK(mpool.cs);
- mpool.removeForBlock(block, ++blocknum);
- }
-
- block.clear();
+ feeEst.processBlock(block_txs, ++blocknum);
+ block_txs.clear();
// Check after just a few txs that combining buckets works as expected
if (blocknum == 3) {
- // Wait for fee estimator to catch up
- m_node.validation_signals->SyncWithValidationInterfaceQueue();
// At this point we should need to combine 3 buckets to get enough data points
// So estimateFee(1) should fail and estimateFee(2) should return somewhere around
// 9*baserate. estimateFee(2) %'s are 100,100,90 = average 97%
@@ -112,9 +97,6 @@ BOOST_AUTO_TEST_CASE(BlockPolicyEstimates)
}
}
- // Wait for fee estimator to catch up
- m_node.validation_signals->SyncWithValidationInterfaceQueue();
-
std::vector<CAmount> origFeeEst;
// Highest feerate is 10*baseRate and gets in all blocks,
// second highest feerate is 9*baseRate and gets in 9/10 blocks = 90%,
@@ -141,13 +123,9 @@ BOOST_AUTO_TEST_CASE(BlockPolicyEstimates)
// Mine 50 more blocks with no transactions happening, estimates shouldn't change
// We haven't decayed the moving average enough so we still have enough data points in every bucket
while (blocknum < 250) {
- LOCK(mpool.cs);
- mpool.removeForBlock(block, ++blocknum);
+ feeEst.processBlock(block_txs, ++blocknum);
}
- // Wait for fee estimator to catch up
- m_node.validation_signals->SyncWithValidationInterfaceQueue();
-
BOOST_CHECK(feeEst.estimateFee(1) == CFeeRate(0));
for (int i = 2; i < 10;i++) {
BOOST_CHECK(feeEst.estimateFee(i).GetFeePerK() < origFeeEst[i-1] + deltaFee);
@@ -161,57 +139,39 @@ BOOST_AUTO_TEST_CASE(BlockPolicyEstimates)
for (int j = 0; j < 10; j++) { // For each fee multiple
for (int k = 0; k < 4; k++) { // add 4 fee txs
tx.vin[0].prevout.n = 10000*blocknum+100*j+k;
- {
- LOCK2(cs_main, mpool.cs);
- TryAddToMempool(mpool, entry.Fee(feeV[j]).Time(Now<NodeSeconds>()).Height(blocknum).FromTx(tx));
- // Since TransactionAddedToMempool callbacks are generated in ATMP,
- // not TryAddToMempool, we cheat and create one manually here
- const int64_t virtual_size = GetVirtualTransactionSize(*MakeTransactionRef(tx));
- const NewMempoolTransactionInfo tx_info{NewMempoolTransactionInfo(MakeTransactionRef(tx),
- feeV[j],
- virtual_size,
- entry.nHeight,
- /*mempool_limit_bypassed=*/false,
- /*submitted_in_package=*/false,
- /*chainstate_is_current=*/true,
- /*has_no_mempool_parents=*/true)};
- m_node.validation_signals->TransactionAddedToMempool(tx_info, mpool.GetAndIncrementSequence());
- }
- txHashes[j].push_back(tx.GetHash());
+ // Simulate the tx being added to the mempool by calling processTransaction(tx_info)
+ mempool_txs[j].emplace_back(entry.Fee(feeV[j]).Time(Now<NodeSeconds>()).Height(blocknum).FromTx(tx));
+ const int64_t virtual_size = GetVirtualTransactionSize(*MakeTransactionRef(tx));
+ const NewMempoolTransactionInfo tx_info{NewMempoolTransactionInfo(MakeTransactionRef(tx),
+ feeV[j],
+ virtual_size,
+ entry.nHeight,
+ /*mempool_limit_bypassed=*/false,
+ /*submitted_in_package=*/false,
+ /*chainstate_is_current=*/true,
+ /*has_no_mempool_parents=*/true)};
+ feeEst.processTransaction(tx_info);
}
}
- {
- LOCK(mpool.cs);
- mpool.removeForBlock(block, ++blocknum);
- }
+ feeEst.processBlock(block_txs, ++blocknum);
}
- // Wait for fee estimator to catch up
- m_node.validation_signals->SyncWithValidationInterfaceQueue();
-
for (int i = 1; i < 10;i++) {
BOOST_CHECK(feeEst.estimateFee(i) == CFeeRate(0) || feeEst.estimateFee(i).GetFeePerK() > origFeeEst[i-1] - deltaFee);
}
// Mine all those transactions
// Estimates should still not be below original
for (int j = 0; j < 10; j++) {
- while(txHashes[j].size()) {
- CTransactionRef ptx = mpool.get(txHashes[j].back());
- if (ptx)
- block.push_back(ptx);
- txHashes[j].pop_back();
+ while (mempool_txs[j].size()) {
+ auto& tx_entry = mempool_txs[j].back();
+ block_txs.emplace_back(tx_entry);
+ mempool_txs[j].pop_back();
}
}
- {
- LOCK(mpool.cs);
- mpool.removeForBlock(block, 266);
- }
- block.clear();
-
- // Wait for fee estimator to catch up
- m_node.validation_signals->SyncWithValidationInterfaceQueue();
+ feeEst.processBlock(block_txs, ++blocknum);
+ block_txs.clear();
BOOST_CHECK(feeEst.estimateFee(1) == CFeeRate(0));
for (int i = 2; i < 10;i++) {
@@ -224,38 +184,27 @@ BOOST_AUTO_TEST_CASE(BlockPolicyEstimates)
for (int j = 0; j < 10; j++) { // For each fee multiple
for (int k = 0; k < 4; k++) { // add 4 fee txs
tx.vin[0].prevout.n = 10000*blocknum+100*j+k;
- {
- LOCK2(cs_main, mpool.cs);
- TryAddToMempool(mpool, entry.Fee(feeV[j]).Time(Now<NodeSeconds>()).Height(blocknum).FromTx(tx));
- // Since TransactionAddedToMempool callbacks are generated in ATMP,
- // not TryAddToMempool, we cheat and create one manually here
- const int64_t virtual_size = GetVirtualTransactionSize(*MakeTransactionRef(tx));
- const NewMempoolTransactionInfo tx_info{NewMempoolTransactionInfo(MakeTransactionRef(tx),
- feeV[j],
- virtual_size,
- entry.nHeight,
- /*mempool_limit_bypassed=*/false,
- /*submitted_in_package=*/false,
- /*chainstate_is_current=*/true,
- /*has_no_mempool_parents=*/true)};
- m_node.validation_signals->TransactionAddedToMempool(tx_info, mpool.GetAndIncrementSequence());
- }
- CTransactionRef ptx = mpool.get(tx.GetHash());
- if (ptx)
- block.push_back(ptx);
-
+ // These txs are mined in the same block, so there is no need to
+ // retain them in mempool_txs; use a local entry to build block_txs.
+ const CTxMemPoolEntry tx_entry{entry.Fee(feeV[j]).Time(Now<NodeSeconds>()).Height(blocknum).FromTx(tx)};
+ const int64_t virtual_size = GetVirtualTransactionSize(*MakeTransactionRef(tx));
+ const NewMempoolTransactionInfo tx_info{NewMempoolTransactionInfo(MakeTransactionRef(tx),
+ feeV[j],
+ virtual_size,
+ entry.nHeight,
+ /*mempool_limit_bypassed=*/false,
+ /*submitted_in_package=*/false,
+ /*chainstate_is_current=*/true,
+ /*has_no_mempool_parents=*/true)};
+
+ feeEst.processTransaction(tx_info);
+ block_txs.emplace_back(tx_entry);
}
}
- {
- LOCK(mpool.cs);
- mpool.removeForBlock(block, ++blocknum);
- }
-
- block.clear();
+ feeEst.processBlock(block_txs, ++blocknum);
+ block_txs.clear();
}
- // Wait for fee estimator to catch up
- m_node.validation_signals->SyncWithValidationInterfaceQueue();
BOOST_CHECK(feeEst.estimateFee(1) == CFeeRate(0));
for (int i = 2; i < 9; i++) { // At 9, the original estimate was already at the bottom (b/c scale = 2)
BOOST_CHECK(feeEst.estimateFee(i).GetFeePerK() < origFeeEst[i-1] - deltaFee);
### src/test/fees_util_tests.cpp
@@ -0,0 +1,26 @@
+// Copyright (c) The Bitcoin Core developers
+// Distributed under the MIT software license, see the accompanying
+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#include <util/fees.h>
+
+#include <boost/test/unit_test.hpp>
+
+BOOST_AUTO_TEST_SUITE(fees_util_tests)
+
+BOOST_AUTO_TEST_CASE(fee_rate_estimator_type_to_string)
+{
+ BOOST_CHECK_EQUAL(FeeRateEstimatorTypeToString(FeeRateEstimatorType::NONE), "none");
+ BOOST_CHECK_EQUAL(FeeRateEstimatorTypeToString(FeeRateEstimatorType::BLOCK_POLICY), "block_policy");
+ BOOST_CHECK_EQUAL(FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY), "mempool_policy");
+}
+
+BOOST_AUTO_TEST_CASE(fee_rate_estimator_type_from_string)
+{
+ BOOST_CHECK(FeeRateEstimatorTypeFromString("none") == FeeRateEstimatorType::NONE);
+ BOOST_CHECK(FeeRateEstimatorTypeFromString("block_policy") == FeeRateEstimatorType::BLOCK_POLICY);
+ BOOST_CHECK(FeeRateEstimatorTypeFromString("mempool_policy") == FeeRateEstimatorType::MEMPOOL_POLICY);
+ BOOST_CHECK(FeeRateEstimatorTypeFromString("unknown") == FeeRateEstimatorType::NONE);
+}
+
+BOOST_AUTO_TEST_SUITE_END()
### src/test/fuzz/CMakeLists.txt
@@ -20,6 +20,7 @@ add_executable(fuzz
block_header.cpp
block_index.cpp
block_index_tree.cpp
+ block_policy_estimator.cpp
blockfilter.cpp
bloom_filter.cpp
buffered_file.cpp
@@ -89,7 +90,6 @@ add_executable(fuzz
parse_script.cpp
parse_univalue.cpp
partially_downloaded_block.cpp
- policy_estimator.cpp
policy_estimator_io.cpp
poolresource.cpp
pow.cpp
### src/test/fuzz/block_policy_estimator.cpp
@@ -2,9 +2,10 @@
// Distributed under the MIT software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
-#include <kernel/mempool_entry.h>
#include <policy/fees/block_policy_estimator.h>
-#include <policy/fees/block_policy_estimator_args.h>
+
+#include <kernel/mempool_entry.h>
+#include <policy/fees/estimator_args.h>
#include <primitives/transaction.h>
#include <streams.h>
#include <test/fuzz/FuzzedDataProvider.h>
@@ -21,18 +22,18 @@ namespace {
const BasicTestingSetup* g_setup;
} // namespace
-void initialize_policy_estimator()
+void initialize_block_policy_estimator()
{
static const auto testing_setup = MakeNoLogFileContext<>();
g_setup = testing_setup.get();
}
-FUZZ_TARGET(policy_estimator, .init = initialize_policy_estimator)
+FUZZ_TARGET(block_policy_estimator, .init = initialize_block_policy_estimator)
{
FuzzedDataProvider fuzzed_data_provider(buffer.data(), buffer.size());
bool good_data{true};
- CBlockPolicyEstimator block_policy_estimator{FeeestPath(*g_setup->m_node.args), DEFAULT_ACCEPT_STALE_FEE_ESTIMATES};
+ CBlockPolicyEstimator block_policy_estimator{BlockPolicyFeeEstPath(*g_setup->m_node.args), DEFAULT_ACCEPT_STALE_FEE_ESTIMATES};
uint32_t current_height{0};
const auto advance_height{
### src/test/fuzz/fees.cpp
@@ -8,6 +8,7 @@
#include <test/fuzz/FuzzedDataProvider.h>
#include <test/fuzz/fuzz.h>
#include <test/fuzz/util.h>
+#include <util/fees.h>
#include <cstdint>
#include <string>
@@ -26,6 +27,11 @@ FUZZ_TARGET(fees)
const CAmount rounded_fee = fee_filter_rounder.round(current_minimum_fee);
assert(MoneyRange(rounded_fee));
}
- const FeeReason fee_reason = fuzzed_data_provider.PickValueInArray({FeeReason::NONE, FeeReason::HALF_ESTIMATE, FeeReason::FULL_ESTIMATE, FeeReason::DOUBLE_ESTIMATE, FeeReason::CONSERVATIVE, FeeReason::MEMPOOL_MIN, FeeReason::FALLBACK, FeeReason::REQUIRED});
+ const FeeReason fee_reason = fuzzed_data_provider.PickValueInArray({FeeReason::FEE_RATE_ESTIMATOR, FeeReason::MEMPOOL_MIN, FeeReason::USER_SPECIFIED, FeeReason::FALLBACK, FeeReason::REQUIRED});
(void)StringForFeeReason(fee_reason);
+ const BlockPolicyEstimateReason block_policy_fee_reason = fuzzed_data_provider.PickValueInArray({BlockPolicyEstimateReason::NONE, BlockPolicyEstimateReason::HALF_ESTIMATE, BlockPolicyEstimateReason::FULL_ESTIMATE, BlockPolicyEstimateReason::DOUBLE_ESTIMATE, BlockPolicyEstimateReason::CONSERVATIVE});
+ (void)StringForBlockPolicyEstimateReason(block_policy_fee_reason);
+ const FeeRateEstimatorType feerate_estimator_type = fuzzed_data_provider.PickValueInArray({FeeRateEstimatorType::NONE, FeeRateEstimatorType::BLOCK_POLICY, FeeRateEstimatorType::MEMPOOL_POLICY});
+ (void)FeeRateEstimatorTypeToString(feerate_estimator_type);
+ (void)FeeRateEstimatorTypeFromString(fuzzed_data_provider.ConsumeRandomLengthString());
}
### src/test/fuzz/policy_estimator_io.cpp
@@ -3,7 +3,8 @@
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#include <policy/fees/block_policy_estimator.h>
-#include <policy/fees/block_policy_estimator_args.h>
+#include <policy/fees/estimator_args.h>
+#include <policy/fees/mempool_estimator.h>
#include <streams.h>
#include <test/fuzz/FuzzedDataProvider.h>
#include <test/fuzz/fuzz.h>
@@ -13,24 +14,41 @@
#include <memory>
namespace {
-const BasicTestingSetup* g_setup;
+const TestingSetup* g_setup;
} // namespace
void initialize_policy_estimator_io()
{
- static const auto testing_setup = MakeNoLogFileContext<>();
+ static const auto testing_setup{
+ MakeNoLogFileContext<const TestingSetup>(ChainType::REGTEST, TestOpts{.setup_net = false})};
g_setup = testing_setup.get();
}
FUZZ_TARGET(policy_estimator_io, .init = initialize_policy_estimator_io)
{
FuzzedDataProvider fuzzed_data_provider(buffer.data(), buffer.size());
FuzzedFileProvider fuzzed_file_provider{fuzzed_data_provider};
- AutoFile fuzzed_auto_file{fuzzed_file_provider.open()};
- // Reusing block_policy_estimator across runs to avoid costly creation of CBlockPolicyEstimator object.
- static CBlockPolicyEstimator block_policy_estimator{FeeestPath(*g_setup->m_node.args), DEFAULT_ACCEPT_STALE_FEE_ESTIMATES};
- if (block_policy_estimator.Read(fuzzed_auto_file)) {
- block_policy_estimator.Write(fuzzed_auto_file);
+ // Reuse estimators across runs to avoid costly object creation.
+ static CBlockPolicyEstimator block_policy_estimator{
+ BlockPolicyFeeEstPath(*g_setup->m_node.args), DEFAULT_ACCEPT_STALE_FEE_ESTIMATES};
+ {
+ AutoFile fuzzed_auto_file_block_policy{fuzzed_file_provider.open()};
+ if (block_policy_estimator.Read(fuzzed_auto_file_block_policy)) {
+ block_policy_estimator.Write(fuzzed_auto_file_block_policy);
+ }
+ (void)fuzzed_auto_file_block_policy.fclose();
+ }
+ static MemPoolFeeRateEstimator mempool_feerate_estimator{
+ MempoolPolicyEstimatorPath(*g_setup->m_node.args),
+ *g_setup->m_node.mempool,
+ *g_setup->m_node.chainman};
+ {
+ AutoFile fuzzed_auto_file_mempool_policy{fuzzed_file_provider.open()};
+ if (mempool_feerate_estimator.Read(fuzzed_auto_file_mempool_policy)) {
+ mempool_feerate_estimator.Write(fuzzed_auto_file_mempool_policy);
+ }
+ // Write() does not take ownership; close explicitly because it may
+ // have written to the fuzzed file.
+ (void)fuzzed_auto_file_mempool_policy.fclose();
}
- (void)fuzzed_auto_file.fclose();
}
### src/test/fuzz/tx_pool.cpp
@@ -156,7 +156,7 @@ void Finish(FuzzedDataProvider& fuzzed_data_provider, MockedTxPool& tx_pool, Cha
// Try updating the mempool for this block, as though it were mined.
LOCK2(::cs_main, tx_pool.cs);
- tx_pool.removeForBlock(block_template->block.vtx, chainstate.m_chain.Height() + 1);
+ tx_pool.removeForBlock(block_template->block.vtx);
// Now try to add those transactions back, as though a reorg happened.
std::vector<Txid> hashes_to_update;
### src/test/mempool_fee_estimator_tests.cpp
@@ -0,0 +1,343 @@
+// Copyright (c) The Bitcoin Core developers
+// Distributed under the MIT software license. See the accompanying
+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#include <kernel/mempool_entry.h>
+#include <policy/fees/estimator_args.h>
+#include <policy/fees/mempool_estimator.h>
+#include <policy/policy.h>
+#include <primitives/block.h>
+#include <random.h>
+#include <test/util/setup_common.h>
+#include <test/util/txmempool.h>
+#include <txmempool.h>
+#include <uint256.h>
+#include <util/feefrac.h>
+#include <util/fees.h>
+#include <util/time.h>
+#include <validation.h>
+
+#include <boost/test/unit_test.hpp>
+
+#include <string>
+
+BOOST_FIXTURE_TEST_SUITE(mempool_fee_estimator_tests, TestingSetup)
+
+static inline CTransactionRef MakeRandomTx()
+{
+ auto rng = FastRandomContext();
+ auto tx = CMutableTransaction();
+ tx.vin.resize(1);
+ tx.vout.resize(1);
+ tx.vin[0].prevout.hash = Txid::FromUint256(rng.rand256());
+ tx.vin[0].prevout.n = 0;
+ tx.vin[0].scriptSig << OP_TRUE;
+ tx.vout[0].scriptPubKey = CScript() << OP_TRUE;
+ tx.vout[0].nValue = COIN;
+ return MakeTransactionRef(tx);
+}
+
+void AddRemovedBlock(MemPoolFeeRateEstimator& fee_est,
+ int32_t removed_txs_weight,
+ int32_t block_txs_weight,
+ unsigned int& height)
+{
+ auto block = std::make_shared<CBlock>();
+ std::vector<RemovedMempoolTransactionInfo> removed_txs;
+ TestMemPoolEntryHelper entry;
+ Assert(block_txs_weight >= removed_txs_weight);
+ block->vtx.emplace_back(MakeRandomTx()); // Add a coinbase tx
+ while (block_txs_weight > 0) {
+ auto tx = MakeRandomTx();
+ auto tx_weight = GetTransactionWeight(*tx);
+ if (block_txs_weight - tx_weight < 0) break;
+ block->vtx.emplace_back(tx);
+ block_txs_weight -= tx_weight;
+ if (removed_txs_weight - tx_weight >= 0) {
+ removed_txs.emplace_back(entry.FromTx(tx));
+ removed_txs_weight -= tx_weight;
+ }
+ }
+ fee_est.MempoolTxsRemovedForBlock(block, removed_txs, height);
+ height += 1;
+}
+
+BOOST_AUTO_TEST_CASE(calculate_max_weight_percentiles)
+{
+ // With no chunks neither percentile can be populated.
+ const auto empty = MemPoolFeeRateEstimator::CalculateMaxWeightPercentiles({});
+ BOOST_CHECK(empty.p50.IsEmpty());
+ BOOST_CHECK(empty.p75.IsEmpty());
+ const int32_t chunk_size{10};
+ const int32_t individual_tx_vsize = static_cast<int32_t>(DEFAULT_BLOCK_MAX_WEIGHT / WITNESS_SCALE_FACTOR) / chunk_size;
+ const FeePerVSize super_high_fee_rate{500 * individual_tx_vsize, individual_tx_vsize};
+ const FeePerVSize high_fee_rate{100 * individual_tx_vsize, individual_tx_vsize};
+ const FeePerVSize medium_fee_rate{50 * individual_tx_vsize, individual_tx_vsize};
+ const FeePerVSize low_fee_rate{10 * individual_tx_vsize, individual_tx_vsize};
+ std::vector<FeePerVSize> chunk_feerates;
+ chunk_feerates.reserve(chunk_size);
+ for (int i = 0; i < chunk_size; ++i) {
+ if (i < 3) {
+ chunk_feerates.emplace_back(super_high_fee_rate);
+ } else if (i < 5) {
+ chunk_feerates.emplace_back(high_fee_rate);
+ } else if (i < 8) {
+ chunk_feerates.emplace_back(medium_fee_rate);
+ // Once 50% coverage is reached but 75% is not, only the p50 (conservative)
+ // percentile is populated; p75 (economical) is left empty for the caller to floor.
+ if (i < 7) {
+ const auto partial = MemPoolFeeRateEstimator::CalculateMaxWeightPercentiles(chunk_feerates);
+ BOOST_CHECK_EQUAL(partial.p50.fee, high_fee_rate.fee);
+ BOOST_CHECK_EQUAL(partial.p50.size, high_fee_rate.size);
+ BOOST_CHECK(partial.p75.IsEmpty());
+ }
+ } else {
+ chunk_feerates.emplace_back(low_fee_rate);
+ }
+ }
+ const auto percentiles = MemPoolFeeRateEstimator::CalculateMaxWeightPercentiles(chunk_feerates);
+ BOOST_CHECK_EQUAL(percentiles.p50.fee, high_fee_rate.fee);
+ BOOST_CHECK_EQUAL(percentiles.p50.size, high_fee_rate.size);
+ BOOST_CHECK_EQUAL(percentiles.p75.fee, medium_fee_rate.fee);
+ BOOST_CHECK_EQUAL(percentiles.p75.size, medium_fee_rate.size);
+ BOOST_CHECK(ByRatio{percentiles.p50} > ByRatio{percentiles.p75});
+}
+
+BOOST_AUTO_TEST_CASE(mempool_fee_rate_estimator_cache)
+{
+ MemPoolFeeRateEstimatorCache cache;
+ const uint256 tip_hash{uint256::ONE};
+ const uint256 next_tip_hash{uint256{2}};
+ const FeePerVSize conservative{2, 1};
+ const FeePerVSize economical{1, 1};
+
+ BOOST_CHECK(cache.IsStale());
+ BOOST_CHECK(!cache.GetCachedEstimate(tip_hash));
+
+ cache.Update(conservative, economical, tip_hash);
+ BOOST_CHECK(!cache.IsStale());
+ const auto cached{cache.GetCachedEstimate(tip_hash)};
+ BOOST_REQUIRE(cached);
+ BOOST_CHECK(cached->m_conservative == conservative);
+ BOOST_CHECK(cached->m_economical == economical);
+ BOOST_CHECK(!cache.GetCachedEstimate(next_tip_hash));
+
+ SetMockTime(GetTime<std::chrono::seconds>() + CACHE_LIFE + std::chrono::seconds{1});
+ BOOST_CHECK(cache.IsStale());
+ BOOST_CHECK(!cache.GetCachedEstimate(tip_hash));
+ SetMockTime(0);
+}
+
+BOOST_AUTO_TEST_CASE(MempoolFeeRateEstimator)
+{
+ auto mempool_estimator = MemPoolFeeRateEstimator(MempoolPolicyEstimatorPath(*m_node.args), *m_node.mempool, *m_node.chainman);
+ BOOST_CHECK_EQUAL(mempool_estimator.MaximumTarget(), MEMPOOL_FEE_ESTIMATOR_MAX_TARGET);
+ // Before the mempool has finished loading, no estimate is available.
+ {
+ const std::string unloaded_err = strprintf("%s: Mempool not loaded yet, no fee rate estimate available",
+ FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY));
+ const auto result = mempool_estimator.EstimateFeeRate(/*conservative=*/true);
+ BOOST_CHECK(!result);
+ BOOST_CHECK_EQUAL(result.error().reason, unloaded_err);
+ }
+ m_node.mempool->SetLoadTried(true);
+
+ BOOST_CHECK(!mempool_estimator.IsMempoolHealthy());
+ BOOST_CHECK(mempool_estimator.GetMempoolHealth() == MemPoolFeeRateEstimator::MempoolHealth::INSUFFICIENT_DATA);
+ {
+ const auto result = mempool_estimator.EstimateFeeRate(/*conservative=*/true);
+ const std::string insufficient_err{strprintf("%s: Not enough recent block data for fee rate estimation",
+ FeeRateEstimatorTypeToString(FeeRateEstimatorType::MEMPOOL_POLICY))};
+ BOOST_CHECK(!result);
+ BOOST_CHECK_EQUAL(result.error().reason, insufficient_err);
+ }
+ {
+ MemPoolFeeRateEstimator custom_mempool_estimator{
+ MempoolPolicyEstimatorPath(*m_node.args), *m_node.mempool, *m_node.chainman};
+ unsigned int custom_height{100};
+ for (size_t block_count{1}; block_count < MEMPOOL_HEALTH_WINDOW_BLOCKS; ++block_count) {
+ AddRemovedBlock(custom_mempool_estimator,
+ /*removed_txs_weight=*/0,
+ /*block_txs_weight=*/0,
+ custom_height);
+ BOOST_CHECK(!custom_mempool_estimator.IsMempoolHealthy());
+ }
+ {
+ const int64_t low_activity_weight{1000};
+ AddRemovedBlock(custom_mempool_estimator, low_activity_weight / 2, low_activity_weight, custom_height);
+ }
+ // Below one block worth of total activity across the full window, even
+ // poor coverage in the only non-empty block is too noisy to reject the
+ // mempool as unhealthy.
+ BOOST_CHECK(custom_mempool_estimator.IsMempoolHealthy());
+ }
+ size_t block_count = 1;
+ const int64_t weight{DEFAULT_BLOCK_MAX_WEIGHT / 2};
+ unsigned int height = 100;
+ // Equal weight
+ while (block_count <= MEMPOOL_HEALTH_WINDOW_BLOCKS) {
+ AddRemovedBlock(mempool_estimator, weight, weight, height);
+ if (block_count < MEMPOOL_HEALTH_WINDOW_BLOCKS) {
+ BOOST_CHECK(!mempool_estimator.IsMempoolHealthy());
+ }
+ block_count += 1;
+ }
+ // Total txs weight ~11999k WU (~3.0 blocks), removed txs ~11999k WU (~3.0 blocks); coverage = 100%.
+ BOOST_CHECK(mempool_estimator.IsMempoolHealthy());
+ // Adding a single underrepresented block will not make the mempool unhealthy
+ // while the window coverage remains above the threshold.
+ AddRemovedBlock(mempool_estimator, weight / 2, weight, height);
+ // Total txs weight ~11999k WU (~3.0 blocks), removed txs ~10999k WU (~2.75 blocks); coverage = ~92%.
+ BOOST_CHECK(mempool_estimator.IsMempoolHealthy());
+ // Empty block
+ // Total txs weight ~9999k WU (~2.5 blocks), removed txs ~8999k WU (~2.25 blocks); coverage = 90%.
+ AddRemovedBlock(mempool_estimator, 0, 0, height);
+ BOOST_CHECK(mempool_estimator.IsMempoolHealthy());
+ // Total txs weight ~9999k WU (~2.5 blocks), removed txs ~7999k WU (~2.0 blocks); coverage = 80%.
+ AddRemovedBlock(mempool_estimator, weight / 2, weight, height);
+ BOOST_CHECK(mempool_estimator.IsMempoolHealthy());
+ // Total txs weight ~9999k WU (~2.5 blocks), removed txs ~7000k WU (~1.75 blocks); coverage = 70%.
+ AddRemovedBlock(mempool_estimator, weight / 2, weight, height);
+ BOOST_CHECK(!mempool_estimator.IsMempoolHealthy());
+ block_count = 1;
+ while (block_count <= 3) {
+ AddRemovedBlock(mempool_estimator, weight, weight, height);
+ if (block_count < 3) {
+ BOOST_CHECK(!mempool_estimator.IsMempoolHealthy());
+ }
+ block_count += 1;
+ }
+ // Total txs weight ~9999k WU (~2.5 blocks), removed txs ~7999k WU (~2.0 blocks); coverage = 80%.
+ BOOST_CHECK(mempool_estimator.IsMempoolHealthy());
+
+ // Reorg out and replace the last block. Replacing the tip block should keep a full
+ // healthy window when the replacement block has good mempool representation.
+ height -= 1;
+ AddRemovedBlock(mempool_estimator, weight, weight, height);
+ BOOST_CHECK(mempool_estimator.IsMempoolHealthy());
+
+ // Reorg out the last two blocks. The estimator should discard the stale suffix,
+ // become temporarily unhealthy due to having fewer than MEMPOOL_HEALTH_WINDOW_BLOCKS stats,
+ // then recover after the replacement chain catches up.
+ height -= 2;
+ AddRemovedBlock(mempool_estimator, weight, weight, height);
+ BOOST_CHECK(!mempool_estimator.IsMempoolHealthy());
+ AddRemovedBlock(mempool_estimator, weight, weight, height);
+ BOOST_CHECK(mempool_estimator.IsMempoolHealthy());
+
+ // A forward height gap (e.g. stale persisted stats after an unclean shutdown
+ // while the chain advanced) resets the tracked window entirely; the estimator
+ // stays unhealthy until a full window of contiguous blocks is seen again.
+ height += 3;
+ AddRemovedBlock(mempool_estimator, weight, weight, height);
+ BOOST_CHECK(!mempool_estimator.IsMempoolHealthy());
+ for (size_t i = 1; i < MEMPOOL_HEALTH_WINDOW_BLOCKS; ++i) {
+ AddRemovedBlock(mempool_estimator, weight, weight, height);
+ if (i < MEMPOOL_HEALTH_WINDOW_BLOCKS - 1) {
+ BOOST_CHECK(!mempool_estimator.IsMempoolHealthy());
+ }
+ }
+ BOOST_CHECK(mempool_estimator.IsMempoolHealthy());
+ {
+ LOCK(m_node.mempool->cs);
+ BOOST_CHECK_EQUAL(m_node.mempool->GetTotalTxSize(), 0);
+ }
+ // With an empty mempool there is nothing to build a feerate estimate from, so both
+ // estimates fall back to the floor fee rate: the higher of the minimum relay fee rate
+ // and the current mempool minimum fee rate.
+ const FeePerVSize floor{std::max(m_node.mempool->m_opts.min_relay_feerate, m_node.mempool->GetMinFee()).GetFeePerVSize()};
+ {
+ const auto result = mempool_estimator.EstimateFeeRate(/*conservative=*/true);
+ BOOST_REQUIRE(result.has_value());
+ BOOST_CHECK(result->feerate == floor);
+ BOOST_CHECK(result->feerate_estimator == FeeRateEstimatorType::MEMPOOL_POLICY);
+ BOOST_CHECK_EQUAL(result->returned_target, MEMPOOL_FEE_ESTIMATOR_MAX_TARGET);
+
+ // The floor estimate is cached like any other; a second call returns the same value.
+ const auto cached_result = mempool_estimator.EstimateFeeRate(/*conservative=*/true);
+ BOOST_REQUIRE(cached_result.has_value());
+ BOOST_CHECK(cached_result->feerate == floor);
+ }
+ TestMemPoolEntryHelper entry;
+ const auto tx_vsize = entry.FromTx(MakeRandomTx()).GetTxSize();
+ const CAmount low_fee{CENT / 3000};
+ const CAmount med_fee{CENT / 100};
+ const CAmount high_fee{CENT / 10};
+ const CAmount very_high_fee{CENT};
+ // A mempool that cannot fill 50% of a block leaves both percentiles empty,
+ // so both estimate still fall back to the floor.
+ {
+ // Add high_fee transactions until mempool weight exceeds 25% of DEFAULT_BLOCK_MAX_WEIGHT.
+ {
+ LOCK2(cs_main, m_node.mempool->cs);
+ while ((m_node.mempool->GetTotalTxSize() * WITNESS_SCALE_FACTOR) <= (DEFAULT_BLOCK_MAX_WEIGHT * 25 / 100)) {
+ TryAddToMempool(*m_node.mempool, entry.Fee(high_fee).FromTx(MakeRandomTx()));
+ }
+ }
+ // Expire the cached floor estimate so the denser mempool is observed.
+ SetMockTime(GetTime<std::chrono::seconds>() + CACHE_LIFE + std::chrono::seconds{1});
+ const auto result = mempool_estimator.EstimateFeeRate(/*conservative=*/true);
+ BOOST_REQUIRE(result.has_value());
+ BOOST_CHECK(result->feerate == floor);
+ }
+ // A mempool that fills 50% of a block but not 75% has a conservative (p50)
+ // estimate, while the economical (p75) estimate falls back to the floor.
+ {
+ // Add med_fee transactions until mempool weight exceeds 50% of DEFAULT_BLOCK_MAX_WEIGHT.
+ {
+ LOCK2(cs_main, m_node.mempool->cs);
+ while ((m_node.mempool->GetTotalTxSize() * WITNESS_SCALE_FACTOR) <= (DEFAULT_BLOCK_MAX_WEIGHT * 50 / 100)) {
+ TryAddToMempool(*m_node.mempool, entry.Fee(med_fee).FromTx(MakeRandomTx()));
+ }
+ }
+ SetMockTime(GetTime<std::chrono::seconds>() + CACHE_LIFE + std::chrono::seconds{1});
+ const auto conservative = mempool_estimator.EstimateFeeRate(/*conservative=*/true);
+ const auto economical = mempool_estimator.EstimateFeeRate(/*conservative=*/false);
+ BOOST_REQUIRE(conservative.has_value());
+ BOOST_REQUIRE(economical.has_value());
+ BOOST_CHECK(conservative->feerate == FeeFrac(med_fee, tx_vsize));
+ BOOST_CHECK(economical->feerate == floor);
+ }
+ // Mempool transactions are enough to provide both feerate estimates.
+ {
+ // Add low_fee transactions until mempool transactions weight
+ // is enough to reach the 75% coverage requirement
+ {
+ LOCK2(cs_main, m_node.mempool->cs);
+ while ((m_node.mempool->GetTotalTxSize() * WITNESS_SCALE_FACTOR) <= (DEFAULT_BLOCK_MAX_WEIGHT * 75 / 100)) {
+ TryAddToMempool(*m_node.mempool, entry.Fee(low_fee).FromTx(MakeRandomTx()));
+ }
+ }
+ // Expire the sparse-result cache before expecting the estimator to observe the denser mempool.
+ SetMockTime(GetTime<std::chrono::seconds>() + CACHE_LIFE + std::chrono::seconds{1});
+ const auto result_conservative = mempool_estimator.EstimateFeeRate(/*conservative=*/true);
+ const auto result_economical = mempool_estimator.EstimateFeeRate(/*conservative=*/false);
+ BOOST_CHECK(result_conservative.has_value());
+ BOOST_CHECK(result_economical.has_value());
+ BOOST_CHECK(result_economical->feerate == FeeFrac(low_fee, tx_vsize));
+ BOOST_CHECK(result_conservative->feerate == FeeFrac(med_fee, tx_vsize));
+ BOOST_CHECK(ByRatio{result_conservative->feerate} > ByRatio{result_economical->feerate});
+ BOOST_CHECK(result_conservative->feerate_estimator == FeeRateEstimatorType::MEMPOOL_POLICY);
+ BOOST_CHECK(result_economical->feerate_estimator == FeeRateEstimatorType::MEMPOOL_POLICY);
+ BOOST_CHECK_EQUAL(result_conservative->returned_target, MEMPOOL_FEE_ESTIMATOR_MAX_TARGET);
+ BOOST_CHECK_EQUAL(result_economical->returned_target, MEMPOOL_FEE_ESTIMATOR_MAX_TARGET);
+
+ // Adding another 30% of very-high-fee transactions should change the
+ // estimates after recomputation, but not while the cached estimate is fresh.
+ {
+ LOCK2(cs_main, m_node.mempool->cs);
+ while ((m_node.mempool->GetTotalTxSize() * WITNESS_SCALE_FACTOR) <=
+ (DEFAULT_BLOCK_MAX_WEIGHT * 105 / 100)) {
+ TryAddToMempool(*m_node.mempool, entry.Fee(very_high_fee).FromTx(MakeRandomTx()));
+ }
+ }
+ BOOST_CHECK(mempool_estimator.EstimateFeeRate(/*conservative=*/false).value().feerate == FeeFrac(low_fee, tx_vsize));
+ BOOST_CHECK(mempool_estimator.EstimateFeeRate(/*conservative=*/true).value().feerate == FeeFrac(med_fee, tx_vsize));
+ // Expire the cache by advancing mock time past CACHE_LIFE so the next call recomputes.
+ SetMockTime(GetTime<std::chrono::seconds>() + CACHE_LIFE + std::chrono::seconds{1});
+ BOOST_CHECK(mempool_estimator.EstimateFeeRate(/*conservative=*/false).value().feerate == FeeFrac(med_fee, tx_vsize));
+ BOOST_CHECK(mempool_estimator.EstimateFeeRate(/*conservative=*/true).value().feerate == FeeFrac(high_fee, tx_vsize));
+ }
+}
+
+BOOST_AUTO_TEST_SUITE_END()
### src/test/mempool_tests.cpp
@@ -283,7 +283,7 @@ BOOST_AUTO_TEST_CASE(MempoolSizeLimitTest)
clock += HALFLIFE;
BOOST_CHECK_EQUAL(pool.GetMinFee(1).GetFeePerK(), maxFeeRateRemoved.GetFeePerK() + DEFAULT_INCREMENTAL_RELAY_FEE);
// ... we should keep the same min fee until we get a block
- pool.removeForBlock(vtx, 1);
+ pool.removeForBlock(vtx);
clock += HALFLIFE;
BOOST_CHECK_EQUAL(pool.GetMinFee(1).GetFeePerK(), llround((maxFeeRateRemoved.GetFeePerK() + DEFAULT_INCREMENTAL_RELAY_FEE)/2.0));
// ... then feerate should drop 1/2 each halflife
### src/test/rbf_tests.cpp
@@ -216,7 +216,7 @@ BOOST_FIXTURE_TEST_CASE(rbf_conflicts_calculator, TestChain100Setup)
dummy.clear();
// If we mine the parent_tx's, then the clusters split (102 clusters).
- pool.removeForBlock({parent_tx_1, parent_tx_2}, /*nBlockHeight=*/ 1);
+ pool.removeForBlock({parent_tx_1, parent_tx_2});
// Add some descendants now to each of the direct children (we can do this now that the clusters have split).
for (const auto& child : direct_children) {
### src/test/util/setup_common.cpp
@@ -342,7 +342,7 @@ ChainTestingSetup::~ChainTestingSetup()
m_node.netgroupman.reset();
m_node.args = nullptr;
m_node.mempool.reset();
- Assert(!m_node.fee_estimator); // Each test must create a local object, if they wish to use the fee_estimator
+ Assert(!m_node.fee_estimator_man); // Each test must create a local object, if they wish to use the fee_estimator_man
m_node.chainman.reset();
m_node.validation_signals.reset();
m_node.scheduler.reset();
### src/txmempool.cpp
@@ -402,7 +402,7 @@ void CTxMemPool::removeConflicts(const CTransaction &tx)
}
}
-void CTxMemPool::removeForBlock(const std::vector<CTransactionRef>& vtx, unsigned int nBlockHeight)
+std::vector<RemovedMempoolTransactionInfo> CTxMemPool::removeForBlock(const std::vector<CTransactionRef>& vtx)
{
// Remove confirmed txs and conflicts when a new block is connected, updating the fee logic
AssertLockHeld(cs);
@@ -420,14 +420,12 @@ void CTxMemPool::removeForBlock(const std::vector<CTransactionRef>& vtx, unsigne
ClearPrioritisation(tx->GetHash());
}
}
- if (m_opts.signals) {
- m_opts.signals->MempoolTransactionsRemovedForBlock(txs_removed_for_block, nBlockHeight);
- }
lastRollingFeeUpdate = GetTime();
blockSinceLastRollingFeeBump = true;
if (!m_txgraph->DoWork(/*max_cost=*/POST_CHANGE_COST)) {
LogDebug(BCLog::MEMPOOL, "Mempool in non-optimal ordering after block.");
}
+ return txs_removed_for_block;
}
void CTxMemPool::check(const CCoinsViewCache& active_coins_tip, int64_t spendheight) const
### src/txmempool.h
@@ -329,7 +329,7 @@ class CTxMemPool
* and updates an entry's LockPoints.
* */
void removeForReorg(CChain& chain, std::function<bool(txiter)> filter_final_and_mature) EXCLUSIVE_LOCKS_REQUIRED(cs, cs_main);
- void removeForBlock(const std::vector<CTransactionRef>& vtx, unsigned int nBlockHeight) EXCLUSIVE_LOCKS_REQUIRED(cs);
+ std::vector<RemovedMempoolTransactionInfo> removeForBlock(const std::vector<CTransactionRef>& vtx) EXCLUSIVE_LOCKS_REQUIRED(cs);
/** Look up wtxids in the mempool and (partially) sort by mining score.
*
### src/util/CMakeLists.txt
@@ -11,6 +11,7 @@ add_library(bitcoin_util STATIC EXCLUDE_FROM_ALL
check.cpp
exec.cpp
exception.cpp
+ fees.cpp
feefrac.cpp
fs.cpp
fs_helpers.cpp
### src/util/fees.cpp
@@ -0,0 +1,32 @@
+// Copyright (c) The Bitcoin Core developers
+// Distributed under the MIT software license, see the accompanying
+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#include <util/fees.h>
+
+#include <util/strencodings.h>
+
+#include <cassert>
+#include <string_view>
+
+std::string_view FeeRateEstimatorTypeToString(FeeRateEstimatorType feerate_estimator_type)
+{
+ switch (feerate_estimator_type) {
+ case FeeRateEstimatorType::NONE:
+ return "none";
+ case FeeRateEstimatorType::BLOCK_POLICY:
+ return "block_policy";
+ case FeeRateEstimatorType::MEMPOOL_POLICY:
+ return "mempool_policy";
+ }
+ // no default case, so the compiler can warn about missing cases
+ assert(false);
+}
+
+FeeRateEstimatorType FeeRateEstimatorTypeFromString(std::string_view feerate_estimator_type)
+{
+ const auto normalized{ToLower(feerate_estimator_type)};
+ if (normalized == "block_policy") return FeeRateEstimatorType::BLOCK_POLICY;
+ if (normalized == "mempool_policy") return FeeRateEstimatorType::MEMPOOL_POLICY;
+ return FeeRateEstimatorType::NONE;
+}
### src/util/fees.h
@@ -5,11 +5,91 @@
#ifndef BITCOIN_UTIL_FEES_H
#define BITCOIN_UTIL_FEES_H
+#include <attributes.h>
+#include <util/expected.h>
+#include <util/feefrac.h>
+
+#include <string>
+#include <string_view>
+#include <utility>
+
/* Used to determine type of fee estimation requested */
enum class FeeEstimateMode {
UNSET, //!< Use default settings based on other criteria
- ECONOMICAL, //!< Force estimateSmartFee to use non-conservative estimates
- CONSERVATIVE, //!< Force estimateSmartFee to use conservative estimates
+ ECONOMICAL, //!< Force Fee rate estimator to return non-conservative estimates
+ CONSERVATIVE, //!< Force Fee rate estimator to return conservative estimates
+};
+
+/* Used to determine the reason a wallet selected a transaction fee rate */
+enum class FeeReason {
+ FEE_RATE_ESTIMATOR,
+ MEMPOOL_MIN,
+ USER_SPECIFIED,
+ FALLBACK,
+ REQUIRED,
};
+/**
+ * @enum FeeRateEstimatorType
+ * Identifier for fee rate estimator.
+ */
+enum class FeeRateEstimatorType {
+ NONE,
+ BLOCK_POLICY,
+ MEMPOOL_POLICY,
+};
+
+/**
+ * @struct FeeRateEstimation
+ * A successful fee rate estimate returned by a fee rate estimator.
+ */
+struct FeeRateEstimation {
+ //! This identifies which fee rate estimator is providing this feerate estimate
+ FeeRateEstimatorType feerate_estimator;
+ //! Fee rate sufficient for confirmation within target.
+ FeePerVSize feerate;
+ //! The returned confirmation target for the estimate.
+ int returned_target;
+ /**
+ * Compare two FeeRateEstimation objects based on fee rate
+ * @param other The other FeeRateEstimation object to compare with
+ * @return strong ordering of either less, greater or equal; based on feerate ratio comparison.
+ */
+ auto operator<=>(const FeeRateEstimation& other) const
+ {
+ return ByRatio{feerate} <=> ByRatio{other.feerate};
+ }
+};
+
+/**
+ * @struct FeeRateEstimationError
+ * A failed fee rate estimation, carrying the zero-value estimation that
+ * identifies the estimator and target alongside the error reason.
+ */
+struct FeeRateEstimationError {
+ FeeRateEstimation estimation;
+ std::string reason;
+};
+
+/**
+ * Build a fee rate estimation error result: a zero-value estimation
+ * identifying the estimator and target, alongside the error message.
+ */
+inline util::Unexpected<FeeRateEstimationError> EstimationError(FeeRateEstimatorType estimator, int returned_target, std::string error)
+{
+ return util::Unexpected{FeeRateEstimationError{{estimator, FeePerVSize{0, 0}, returned_target}, std::move(error)}};
+}
+
+/**
+ * Return the estimation carried by a fee rate estimate result: the
+ * successful estimation, or the error's zero-value estimation.
+ */
+inline const FeeRateEstimation& FeeRateEstimationRef(const util::Expected<FeeRateEstimation, FeeRateEstimationError>& result LIFETIMEBOUND)
+{
+ return result ? *result : result.error().estimation;
+}
+
+std::string_view FeeRateEstimatorTypeToString(FeeRateEstimatorType feerate_estimator_type);
+FeeRateEstimatorType FeeRateEstimatorTypeFromString(std::string_view feerate_estimator_type);
+
#endif // BITCOIN_UTIL_FEES_H
### src/validation.cpp
@@ -3087,14 +3087,19 @@ bool Chainstate::ConnectTip(
Ticks<MillisecondsDouble>(time_5 - time_4),
Ticks<SecondsDouble>(m_chainman.time_chainstate),
Ticks<MillisecondsDouble>(m_chainman.time_chainstate) / m_chainman.num_blocks_total);
- // Remove conflicting transactions from the mempool.;
+ // Remove conflicting transactions from the mempool.
+ std::vector<RemovedMempoolTransactionInfo> txs_removed_for_block;
if (m_mempool) {
- m_mempool->removeForBlock(block_to_connect->vtx, pindexNew->nHeight);
+ txs_removed_for_block = m_mempool->removeForBlock(block_to_connect->vtx);
disconnectpool.removeForBlock(block_to_connect->vtx);
}
// Update m_chain & related variables.
m_chain.SetTip(*pindexNew);
m_chainman.UpdateIBDStatus();
+ // Not fired while IBD is active. removeForBlock() above still runs.
+ if (m_mempool && m_chainman.m_options.signals && !m_chainman.IsInitialBlockDownload()) {
+ m_chainman.m_options.signals->MempoolTransactionsRemovedForBlock(block_to_connect, std::move(txs_removed_for_block), pindexNew->nHeight);
+ }
UpdateTip(pindexNew);
const auto time_6{SteadyClock::now()};
### src/validationinterface.cpp
@@ -230,13 +230,16 @@ void ValidationSignals::BlockConnected(const ChainstateRole& role, std::shared_p
ENQUEUE_AND_LOG_EVENT(std::move(event), std::move(log_msg));
}
-void ValidationSignals::MempoolTransactionsRemovedForBlock(const std::vector<RemovedMempoolTransactionInfo>& txs_removed_for_block, unsigned int nBlockHeight)
+void ValidationSignals::MempoolTransactionsRemovedForBlock(std::shared_ptr<const CBlock> block, std::vector<RemovedMempoolTransactionInfo> txs_removed_for_block, unsigned int block_height)
{
- auto log_msg = LOG_MSG("%s: block height=%s txs removed=%s", __func__,
- nBlockHeight,
- txs_removed_for_block.size());
- auto event = [txs_removed_for_block, nBlockHeight, this] {
- m_internals->Iterate([&](CValidationInterface& callbacks) { callbacks.MempoolTransactionsRemovedForBlock(txs_removed_for_block, nBlockHeight); });
+ Assume(block);
+ auto log_msg = LOG_MSG("%s: block hash=%s block height=%s txs removed=%s block txs=%s", __func__,
+ block->GetHash().ToString(),
+ block_height,
+ txs_removed_for_block.size(),
+ block->vtx.size());
+ auto event = [block = std::move(block), txs_removed_for_block = std::move(txs_removed_for_block), block_height, this] {
+ m_internals->Iterate([&](CValidationInterface& callbacks) { callbacks.MempoolTransactionsRemovedForBlock(block, txs_removed_for_block, block_height); });
};
ENQUEUE_AND_LOG_EVENT(std::move(event), std::move(log_msg));
}
### src/validationinterface.h
@@ -9,6 +9,7 @@
#include <kernel/cs_main.h>
#include <primitives/transaction.h>
#include <sync.h>
+#include <uint256.h>
#include <cstddef>
#include <cstdint>
@@ -112,9 +113,11 @@ class CValidationInterface {
* as a result of new block being connected.
* MempoolTransactionsRemovedForBlock will be fired before BlockConnected.
*
+ * Not fired while initial block download is active.
+ *
* Called on a background thread.
*/
- virtual void MempoolTransactionsRemovedForBlock(const std::vector<RemovedMempoolTransactionInfo>& txs_removed_for_block, unsigned int nBlockHeight) {}
+ virtual void MempoolTransactionsRemovedForBlock(const std::shared_ptr<const CBlock>& block, const std::vector<RemovedMempoolTransactionInfo>& txs_removed_for_block, unsigned int block_height) {}
/**
* Notifies listeners of a block being connected.
*
@@ -222,7 +225,7 @@ class ValidationSignals {
void ActiveTipChange(const CBlockIndex&, bool);
void TransactionAddedToMempool(const NewMempoolTransactionInfo&, uint64_t mempool_sequence);
void TransactionRemovedFromMempool(const CTransactionRef&, MemPoolRemovalReason, uint64_t mempool_sequence);
- void MempoolTransactionsRemovedForBlock(const std::vector<RemovedMempoolTransactionInfo>&, unsigned int nBlockHeight);
+ void MempoolTransactionsRemovedForBlock(std::shared_ptr<const CBlock>, std::vector<RemovedMempoolTransactionInfo>, unsigned int block_height);
void BlockConnected(const kernel::ChainstateRole&, std::shared_ptr<const CBlock>, const CBlockIndex* pindex);
void BlockDisconnected(std::shared_ptr<const CBlock>, const CBlockIndex* pindex);
void ChainStateFlushed(const kernel::ChainstateRole&, const CBlockLocator&);
### src/wallet/coincontrol.h
@@ -7,7 +7,6 @@
#include <outputtype.h>
#include <policy/feerate.h>
-#include <policy/fees/block_policy_estimator.h>
#include <primitives/transaction.h>
#include <script/keyorigin.h>
#include <script/signingprovider.h>
@@ -104,7 +103,7 @@ class CCoinControl
bool m_avoid_partial_spends = DEFAULT_AVOIDPARTIALSPENDS;
//! Forbids inclusion of dirty (previously used) addresses
bool m_avoid_address_reuse = false;
- //! Fee estimation mode to control arguments to estimateSmartFee
+ //! Fee estimation mode.
FeeEstimateMode m_fee_mode = FeeEstimateMode::UNSET;
//! Minimum chain depth value for coin availability
int m_min_depth = DEFAULT_MIN_DEPTH;
### src/wallet/feebumper.cpp
@@ -8,7 +8,6 @@
#include <common/system.h>
#include <consensus/validation.h>
#include <interfaces/chain.h>
-#include <policy/fees/block_policy_estimator.h>
#include <policy/policy.h>
#include <util/moneystr.h>
#include <util/rbf.h>
@@ -139,7 +138,7 @@ static CFeeRate EstimateFeeRate(const CWallet& wallet, const CWalletTx& wtx, con
feerate += std::max(node_incremental_relay_fee, wallet_incremental_relay_fee);
// Fee rate must also be at least the wallet's GetMinimumFeeRate
- CFeeRate min_feerate(GetMinimumFeeRate(wallet, coin_control, /*feeCalc=*/nullptr));
+ CFeeRate min_feerate(GetMinimumFeeRate(wallet, coin_control).fee_rate);
// Set the required fee rate for the replacement transaction in coin control.
return std::max(feerate, min_feerate);
### src/wallet/fees.cpp
@@ -5,9 +5,12 @@
#include <wallet/fees.h>
+#include <policy/feerate.h>
+#include <util/fees.h>
#include <wallet/coincontrol.h>
#include <wallet/wallet.h>
+#include <optional>
namespace wallet {
CAmount GetRequiredFee(const CWallet& wallet, unsigned int nTxBytes)
@@ -16,69 +19,79 @@ CAmount GetRequiredFee(const CWallet& wallet, unsigned int nTxBytes)
}
-CAmount GetMinimumFee(const CWallet& wallet, unsigned int nTxBytes, const CCoinControl& coin_control, FeeCalculation* feeCalc)
+CAmount GetMinimumFee(const MinimumFeeRateResult& min_fee_rate, unsigned int nTxBytes)
{
- return GetMinimumFeeRate(wallet, coin_control, feeCalc).GetFee(static_cast<int32_t>(nTxBytes));
+ return min_fee_rate.fee_rate.GetFee(static_cast<int32_t>(nTxBytes));
}
CFeeRate GetRequiredFeeRate(const CWallet& wallet)
{
return std::max(wallet.m_min_fee, wallet.chain().relayMinFee());
}
-CFeeRate GetMinimumFeeRate(const CWallet& wallet, const CCoinControl& coin_control, FeeCalculation* feeCalc)
+MinimumFeeRateResult GetMinimumFeeRate(const CWallet& wallet, const CCoinControl& coin_control)
{
/* User control of how to calculate fee uses the following parameter precedence:
1. coin_control.m_feerate
2. coin_control.m_confirm_target
3. m_confirm_target (user-set member variable of wallet)
The first parameter that is set is used.
*/
- CFeeRate feerate_needed;
if (coin_control.m_feerate) { // 1.
- feerate_needed = *(coin_control.m_feerate);
+ CFeeRate fee_rate{*coin_control.m_feerate};
// Allow to override automatic min/max check over coin control instance
- if (coin_control.fOverrideFeeRate) return feerate_needed;
+ if (coin_control.fOverrideFeeRate) return {fee_rate, FeeReason::USER_SPECIFIED, std::nullopt};
+
+ CFeeRate required_feerate = GetRequiredFeeRate(wallet);
+ if (required_feerate > fee_rate) return {required_feerate, FeeReason::REQUIRED, std::nullopt};
+
+ return {fee_rate, FeeReason::USER_SPECIFIED, std::nullopt};
+ }
+
+ // We will use smart fee estimation
+ unsigned int target = coin_control.m_confirm_target ? *coin_control.m_confirm_target : wallet.m_confirm_target;
+ // By default estimates are economical iff we are signaling opt-in-RBF
+ bool conservative_estimate = !coin_control.m_signal_bip125_rbf.value_or(wallet.m_signal_rbf);
+ // Allow to override the default fee estimate mode over the CoinControl instance
+ if (coin_control.m_fee_mode == FeeEstimateMode::CONSERVATIVE)
+ conservative_estimate = true;
+ else if (coin_control.m_fee_mode == FeeEstimateMode::ECONOMICAL)
+ conservative_estimate = false;
+
+ const auto fee_estimation_res = wallet.chain().getFeeRateEstimate(target, conservative_estimate);
+ const FeeRateEstimation& estimation{FeeRateEstimationRef(fee_estimation_res)};
+ CFeeRate fee_rate{estimation.feerate};
+ FeeReason fee_reason{FeeReason::FEE_RATE_ESTIMATOR};
+ // Only fee rate estimator results have a returned target.
+ std::optional<int> returned_target{estimation.returned_target};
+ if (fee_rate == CFeeRate(0)) {
+ // if we don't have enough data for getFeeRateEstimate, then use fallback fee
+ fee_rate = wallet.m_fallback_fee;
+ fee_reason = FeeReason::FALLBACK;
+ returned_target = std::nullopt;
+ // directly return if fallback fee is disabled (feerate 0 == disabled)
+ if (wallet.m_fallback_fee == CFeeRate(0)) return {fee_rate, FeeReason::FALLBACK, std::nullopt};
}
- else { // 2. or 3.
- // We will use smart fee estimation
- unsigned int target = coin_control.m_confirm_target ? *coin_control.m_confirm_target : wallet.m_confirm_target;
- // By default estimates are economical iff we are signaling opt-in-RBF
- bool conservative_estimate = !coin_control.m_signal_bip125_rbf.value_or(wallet.m_signal_rbf);
- // Allow to override the default fee estimate mode over the CoinControl instance
- if (coin_control.m_fee_mode == FeeEstimateMode::CONSERVATIVE) conservative_estimate = true;
- else if (coin_control.m_fee_mode == FeeEstimateMode::ECONOMICAL) conservative_estimate = false;
-
- feerate_needed = wallet.chain().estimateSmartFee(target, conservative_estimate, feeCalc);
- if (feerate_needed == CFeeRate(0)) {
- // if we don't have enough data for estimateSmartFee, then use fallback fee
- feerate_needed = wallet.m_fallback_fee;
- if (feeCalc) feeCalc->reason = FeeReason::FALLBACK;
-
- // directly return if fallback fee is disabled (feerate 0 == disabled)
- if (wallet.m_fallback_fee == CFeeRate(0)) return feerate_needed;
- }
- // Obey mempool min fee when using smart fee estimation
- CFeeRate min_mempool_feerate = wallet.chain().mempoolMinFee();
- if (feerate_needed < min_mempool_feerate) {
- feerate_needed = min_mempool_feerate;
- if (feeCalc) feeCalc->reason = FeeReason::MEMPOOL_MIN;
- }
+
+ // Obey mempool min fee when using smart fee estimation or fallback fee
+ CFeeRate min_mempool_feerate = wallet.chain().mempoolMinFee();
+ if (fee_rate < min_mempool_feerate) {
+ fee_rate = min_mempool_feerate;
+ fee_reason = FeeReason::MEMPOOL_MIN;
+ returned_target = std::nullopt;
}
- // prevent user from paying a fee below the required fee rate
CFeeRate required_feerate = GetRequiredFeeRate(wallet);
- if (required_feerate > feerate_needed) {
- feerate_needed = required_feerate;
- if (feeCalc) feeCalc->reason = FeeReason::REQUIRED;
- }
- return feerate_needed;
+ if (required_feerate > fee_rate) return {required_feerate, FeeReason::REQUIRED, std::nullopt};
+
+ return {fee_rate, fee_reason, returned_target};
}
CFeeRate GetDiscardRate(const CWallet& wallet)
{
- unsigned int highest_target = wallet.chain().estimateMaxBlocks();
- CFeeRate discard_rate = wallet.chain().estimateSmartFee(highest_target, /*conservative=*/false);
+ unsigned int highest_target = wallet.chain().maximumFeeEstimationTargetBlocks();
+ const auto res = wallet.chain().getFeeRateEstimate(highest_target, /*conservative=*/false);
+ auto discard_rate = res ? CFeeRate(res->feerate) : CFeeRate(0);
// Don't let discard_rate be greater than longest possible fee estimate if we get a valid fee estimate
discard_rate = (discard_rate == CFeeRate(0)) ? wallet.m_discard_rate : std::min(discard_rate, wallet.m_discard_rate);
// Discard rate must be at least dust relay feerate
### src/wallet/fees.h
@@ -7,9 +7,9 @@
#define BITCOIN_WALLET_FEES_H
#include <consensus/amount.h>
+#include <wallet/types.h>
class CFeeRate;
-struct FeeCalculation;
namespace wallet {
class CCoinControl;
@@ -22,10 +22,9 @@ class CWallet;
CAmount GetRequiredFee(const CWallet& wallet, unsigned int nTxBytes);
/**
- * Estimate the minimum fee considering user set parameters
- * and the required fee
+ * Return the minimum fee for this size given a fee rate result.
*/
-CAmount GetMinimumFee(const CWallet& wallet, unsigned int nTxBytes, const CCoinControl& coin_control, FeeCalculation* feeCalc);
+CAmount GetMinimumFee(const MinimumFeeRateResult& min_fee_rate, unsigned int nTxBytes);
/**
* Return the minimum required feerate taking into account the
@@ -37,7 +36,7 @@ CFeeRate GetRequiredFeeRate(const CWallet& wallet);
* Estimate the minimum fee rate considering user set parameters
* and the required fee
*/
-CFeeRate GetMinimumFeeRate(const CWallet& wallet, const CCoinControl& coin_control, FeeCalculation* feeCalc);
+MinimumFeeRateResult GetMinimumFeeRate(const CWallet& wallet, const CCoinControl& coin_control);
/**
* Return the maximum feerate for discarding change.
### src/wallet/interfaces.cpp
@@ -9,7 +9,6 @@
#include <interfaces/chain.h>
#include <interfaces/handler.h>
#include <node/types.h>
-#include <policy/fees/block_policy_estimator.h>
#include <primitives/transaction.h>
#include <rpc/server.h>
#include <scheduler.h>
@@ -469,14 +468,13 @@ class WalletImpl : public Wallet
CAmount getRequiredFee(unsigned int tx_bytes) override { return GetRequiredFee(*m_wallet, tx_bytes); }
CAmount getMinimumFee(unsigned int tx_bytes,
const CCoinControl& coin_control,
- int* returned_target,
+ std::optional<int>* returned_target,
FeeReason* reason) override
{
- FeeCalculation fee_calc;
- CAmount result;
- result = GetMinimumFee(*m_wallet, tx_bytes, coin_control, &fee_calc);
- if (returned_target) *returned_target = fee_calc.returnedTarget;
- if (reason) *reason = fee_calc.reason;
+ auto min_fee_rate{GetMinimumFeeRate(*m_wallet, coin_control)};
+ auto result = GetMinimumFee(min_fee_rate, tx_bytes);
+ if (returned_target) *returned_target = min_fee_rate.returned_target;
+ if (reason) *reason = min_fee_rate.fee_reason;
return result;
}
unsigned int getConfirmTarget() override { return m_wallet->m_confirm_target; }
### src/wallet/rpc/spend.cpp
@@ -195,7 +195,7 @@ UniValue SendMoney(CWallet& wallet, const CCoinControl &coin_control, std::vecto
if (verbose) {
UniValue entry(UniValue::VOBJ);
entry.pushKV("txid", tx->GetHash().GetHex());
- entry.pushKV("fee_reason", StringForFeeReason(res->fee_calc.reason));
+ entry.pushKV("fee_reason", StringForFeeReason(res->fee_reason));
return entry;
}
return tx->GetHash().GetHex();
@@ -235,7 +235,7 @@ static void SetFeeEstimateMode(const CWallet& wallet, CCoinControl& cc, const Un
throw JSONRPCError(RPC_INVALID_PARAMETER, InvalidEstimateModeErrorMessage());
}
if (!conf_target.isNull()) {
- cc.m_confirm_target = ParseConfirmTarget(conf_target, wallet.chain().estimateMaxBlocks());
+ cc.m_confirm_target = ParseConfirmTarget(conf_target, wallet.chain().maximumFeeEstimationTargetBlocks());
}
}
@@ -272,7 +272,7 @@ RPCMethod sendtoaddress()
RPCResult::Type::OBJ, "", "",
{
{RPCResult::Type::STR_HEX, "txid", "The transaction id."},
- {RPCResult::Type::STR, "fee_reason", "The transaction fee reason."}
+ {RPCResult::Type::STR, "fee_reason", "The reason the wallet selected this fee rate (e.g. fee rate estimator, mempool minimum, fallback, or minimum required)."}
},
},
},
@@ -382,7 +382,7 @@ RPCMethod sendmany()
{
{RPCResult::Type::STR_HEX, "txid", "The transaction id for the send. Only 1 transaction is created regardless of\n"
"the number of addresses."},
- {RPCResult::Type::STR, "fee_reason", "The transaction fee reason."}
+ {RPCResult::Type::STR, "fee_reason", "The reason the wallet selected this fee rate (e.g. fee rate estimator, mempool minimum, fallback, or minimum required)."}
},
},
},
@@ -1439,23 +1439,22 @@ RPCMethod sendall()
const bool rbf{options.exists("replaceable") ? options["replaceable"].get_bool() : pwallet->m_signal_rbf};
- FeeCalculation fee_calc_out;
- CFeeRate fee_rate{GetMinimumFeeRate(*pwallet, coin_control, &fee_calc_out)};
+ auto [fee_rate, fee_reason, returned_target] = GetMinimumFeeRate(*pwallet, coin_control);
// Do not, ever, assume that it's fine to change the fee rate if the user has explicitly
// provided one
if (coin_control.m_feerate && fee_rate > *coin_control.m_feerate) {
const auto feerate_format = FeeRateFormat::SAT_VB;
auto msg{strprintf("Fee rate (%s) is lower than the minimum fee rate setting (%s).",
coin_control.m_feerate->ToString(feerate_format),
fee_rate.ToString(feerate_format))};
- if (fee_calc_out.reason == FeeReason::REQUIRED) {
+ if (fee_reason == FeeReason::REQUIRED) {
msg += strprintf("\nConsider modifying -mintxfee (%s) or -minrelaytxfee (%s).",
pwallet->m_min_fee.ToString(feerate_format),
pwallet->chain().relayMinFee().ToString(feerate_format));
}
throw JSONRPCError(RPC_INVALID_PARAMETER, msg);
}
- if (fee_calc_out.reason == FeeReason::FALLBACK && !pwallet->m_allow_fallback_fee) {
+ if (fee_reason == FeeReason::FALLBACK && !pwallet->m_allow_fallback_fee) {
// eventually allow a fallback fee
throw JSONRPCError(RPC_WALLET_ERROR, "Fee estimation failed. Fallbackfee is disabled. Wait a few blocks or enable -fallbackfee.");
}
### src/wallet/spend.cpp
@@ -1151,16 +1151,16 @@ static util::Result<CreatedTransactionResult> CreateTransactionInternal(
coin_selection_params.m_discard_feerate = GetDiscardRate(wallet);
// Get the fee rate to use effective values in coin selection
- FeeCalculation feeCalc;
- coin_selection_params.m_effective_feerate = GetMinimumFeeRate(wallet, coin_control, &feeCalc);
+ auto min_fee_rate{GetMinimumFeeRate(wallet, coin_control)};
+ coin_selection_params.m_effective_feerate = min_fee_rate.fee_rate;
// Do not, ever, assume that it's fine to change the fee rate if the user has explicitly
// provided one
if (coin_control.m_feerate && coin_selection_params.m_effective_feerate > *coin_control.m_feerate) {
const auto feerate_format = FeeRateFormat::SAT_VB;
auto msg{strprintf(_("Fee rate (%s) is lower than the minimum fee rate setting (%s)."),
coin_control.m_feerate->ToString(feerate_format),
coin_selection_params.m_effective_feerate.ToString(feerate_format))};
- if (feeCalc.reason == FeeReason::REQUIRED) {
+ if (min_fee_rate.fee_reason == FeeReason::REQUIRED) {
msg += strprintf(_("\nConsider modifying %s (%s) or %s (%s)."),
"-mintxfee",
wallet.m_min_fee.ToString(feerate_format),
@@ -1169,7 +1169,7 @@ static util::Result<CreatedTransactionResult> CreateTransactionInternal(
}
return util::Error{msg};
}
- if (feeCalc.reason == FeeReason::FALLBACK && !wallet.m_allow_fallback_fee) {
+ if (min_fee_rate.fee_reason == FeeReason::FALLBACK && !wallet.m_allow_fallback_fee) {
// eventually allow a fallback fee
return util::Error{strprintf(_("Fee estimation failed. Fallbackfee is disabled. Wait a few blocks or enable %s."), "-fallbackfee")};
}
@@ -1434,15 +1434,9 @@ static util::Result<CreatedTransactionResult> CreateTransactionInternal(
reservedest.KeepDestination();
wallet.WalletLogPrintf("Coin Selection: Algorithm:%s, Waste Metric Score:%d\n", GetAlgorithmName(result.GetAlgo()), result.GetWaste());
- wallet.WalletLogPrintf("Fee Calculation: Fee:%d Bytes:%u Tgt:%d (requested %d) Reason:\"%s\" Decay %.5f: Estimation: (%g - %g) %.2f%% %.1f/(%.1f %d mem %.1f out) Fail: (%g - %g) %.2f%% %.1f/(%.1f %d mem %.1f out)\n",
- current_fee, nBytes, feeCalc.returnedTarget, feeCalc.desiredTarget, StringForFeeReason(feeCalc.reason), feeCalc.est.decay,
- feeCalc.est.pass.start, feeCalc.est.pass.end,
- (feeCalc.est.pass.totalConfirmed + feeCalc.est.pass.inMempool + feeCalc.est.pass.leftMempool) > 0.0 ? 100 * feeCalc.est.pass.withinTarget / (feeCalc.est.pass.totalConfirmed + feeCalc.est.pass.inMempool + feeCalc.est.pass.leftMempool) : 0.0,
- feeCalc.est.pass.withinTarget, feeCalc.est.pass.totalConfirmed, feeCalc.est.pass.inMempool, feeCalc.est.pass.leftMempool,
- feeCalc.est.fail.start, feeCalc.est.fail.end,
- (feeCalc.est.fail.totalConfirmed + feeCalc.est.fail.inMempool + feeCalc.est.fail.leftMempool) > 0.0 ? 100 * feeCalc.est.fail.withinTarget / (feeCalc.est.fail.totalConfirmed + feeCalc.est.fail.inMempool + feeCalc.est.fail.leftMempool) : 0.0,
- feeCalc.est.fail.withinTarget, feeCalc.est.fail.totalConfirmed, feeCalc.est.fail.inMempool, feeCalc.est.fail.leftMempool);
- return CreatedTransactionResult(tx, current_fee, change_pos, feeCalc);
+ wallet.WalletLogPrintf("Fee Calculation: Fee:%d Bytes:%u, Source: %s\n",
+ current_fee, nBytes, StringForFeeReason(min_fee_rate.fee_reason));
+ return CreatedTransactionResult(tx, current_fee, change_pos, min_fee_rate.fee_reason);
}
util::Result<CreatedTransactionResult> CreateTransaction(
### src/wallet/spend.h
@@ -6,7 +6,6 @@
#define BITCOIN_WALLET_SPEND_H
#include <consensus/amount.h>
-#include <policy/fees/block_policy_estimator.h>
#include <util/result.h>
#include <wallet/coinselection.h>
#include <wallet/transaction.h>
### src/wallet/test/fuzz/fees.cpp
@@ -2,60 +2,72 @@
// Distributed under the MIT software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+#include <policy/fees/estimator_man.h>
#include <test/fuzz/FuzzedDataProvider.h>
#include <test/fuzz/fuzz.h>
#include <test/fuzz/util.h>
#include <test/util/setup_common.h>
#include <test/util/time.h>
#include <test/util/txmempool.h>
+#include <util/expected.h>
+#include <util/fees.h>
#include <validation.h>
#include <wallet/coincontrol.h>
#include <wallet/fees.h>
#include <wallet/test/util.h>
#include <wallet/wallet.h>
+#include <optional>
+
namespace wallet {
namespace {
-struct FeeEstimatorTestingSetup : public TestingSetup {
- FeeEstimatorTestingSetup(const ChainType chain_type, TestOpts opts) : TestingSetup{chain_type, opts}
+struct FeeEstimatorManTestingSetup : public TestingSetup {
+ FeeEstimatorManTestingSetup(const ChainType chain_type, TestOpts opts) : TestingSetup{chain_type, opts}
{
}
- ~FeeEstimatorTestingSetup() {
- m_node.fee_estimator.reset();
+ ~FeeEstimatorManTestingSetup()
+ {
+ m_node.fee_estimator_man.reset();
}
- void SetFeeEstimator(std::unique_ptr<CBlockPolicyEstimator> fee_estimator)
+ void SetFeeEstimatorMan(std::unique_ptr<FeeRateEstimatorManager> fee_estimator_man)
{
- m_node.fee_estimator = std::move(fee_estimator);
+ m_node.fee_estimator_man = std::move(fee_estimator_man);
}
};
-FeeEstimatorTestingSetup* g_setup;
+FeeEstimatorManTestingSetup* g_setup;
-class FuzzedBlockPolicyEstimator : public CBlockPolicyEstimator
+class FuzzedFeeEstimatorMan : public FeeRateEstimatorManager
{
FuzzedDataProvider& fuzzed_data_provider;
public:
- FuzzedBlockPolicyEstimator(FuzzedDataProvider& provider)
- : CBlockPolicyEstimator(fs::path{}, false), fuzzed_data_provider(provider) {}
+ FuzzedFeeEstimatorMan(FuzzedDataProvider& provider, const CTxMemPool& mempool, ChainstateManager& chainman)
+ : FeeRateEstimatorManager(fs::path{}, false, fs::path{}, mempool, chainman), fuzzed_data_provider(provider) {}
- CFeeRate estimateSmartFee(int confTarget, FeeCalculation* feeCalc, bool conservative) const override
+ util::Expected<FeeRateEstimation, FeeRateEstimationError> GetFeeRateEstimate(int confTarget, bool conservative) const override
{
- return CFeeRate{ConsumeMoney(fuzzed_data_provider, /*max=*/1'000'000)};
+ FeePerVSize feerate(ConsumeMoney(fuzzed_data_provider, /*max=*/1'000'000), fuzzed_data_provider.ConsumeIntegralInRange<unsigned int>(1000, 1000000));
+ return FeeRateEstimation{FeeRateEstimatorType::BLOCK_POLICY, feerate, fuzzed_data_provider.ConsumeIntegralInRange<int>(2, 1004)};
}
-
- unsigned int HighestTargetTracked(FeeEstimateHorizon horizon) const override
+ util::Expected<FeeRateEstimation, FeeRateEstimationError> GetFeeRateEstimate(FeeRateEstimatorType type, int confTarget, bool conservative) const override
+ {
+ auto res = GetFeeRateEstimate(confTarget, conservative);
+ if (res) res->feerate_estimator = type;
+ return res;
+ }
+ unsigned int MaximumTarget() const override
{
- return fuzzed_data_provider.ConsumeIntegralInRange<unsigned int>(1, 1000);
+ return fuzzed_data_provider.ConsumeIntegralInRange<unsigned int>(1, 1004);
}
};
void initialize_setup()
{
- static const auto testing_setup = MakeNoLogFileContext<FeeEstimatorTestingSetup>();
+ static const auto testing_setup = MakeNoLogFileContext<FeeEstimatorManTestingSetup>();
g_setup = testing_setup.get();
}
@@ -74,8 +86,8 @@ FUZZ_TARGET(wallet_fees, .init = initialize_setup)
.dust_relay_feerate = CFeeRate{ConsumeMoney(fuzzed_data_provider, 1'000'000)}
};
node.mempool = std::make_unique<CTxMemPool>(mempool_opts, error);
- std::unique_ptr<CBlockPolicyEstimator> fee_estimator = std::make_unique<FuzzedBlockPolicyEstimator>(fuzzed_data_provider);
- g_setup->SetFeeEstimator(std::move(fee_estimator));
+ std::unique_ptr<FeeRateEstimatorManager> fee_estimator_man = std::make_unique<FuzzedFeeEstimatorMan>(fuzzed_data_provider, *node.mempool, *node.chainman);
+ g_setup->SetFeeEstimatorMan(std::move(fee_estimator_man));
auto target_feerate{CFeeRate{ConsumeMoney(fuzzed_data_provider, /*max=*/1'000'000)}};
if (target_feerate > node.mempool->m_opts.incremental_relay_feerate &&
target_feerate > node.mempool->m_opts.min_relay_feerate) {
@@ -115,11 +127,20 @@ FUZZ_TARGET(wallet_fees, .init = initialize_setup)
if (fuzzed_data_provider.ConsumeBool()) {
coin_control.m_fee_mode = fuzzed_data_provider.ConsumeBool() ? FeeEstimateMode::CONSERVATIVE : FeeEstimateMode::ECONOMICAL;
}
-
- FeeCalculation fee_calculation;
- FeeCalculation* maybe_fee_calculation{fuzzed_data_provider.ConsumeBool() ? nullptr : &fee_calculation};
- (void)GetMinimumFeeRate(wallet, coin_control, maybe_fee_calculation);
- (void)GetMinimumFee(wallet, tx_bytes, coin_control, maybe_fee_calculation);
+ FeeReason fee_reason{FeeReason::FEE_RATE_ESTIMATOR};
+ if (fuzzed_data_provider.ConsumeBool()) {
+ fee_reason = fuzzed_data_provider.PickValueInArray({FeeReason::FEE_RATE_ESTIMATOR, FeeReason::MEMPOOL_MIN, FeeReason::USER_SPECIFIED, FeeReason::FALLBACK, FeeReason::REQUIRED});
+ }
+ std::optional<int> returned_target;
+ if (fuzzed_data_provider.ConsumeBool()) {
+ returned_target = fuzzed_data_provider.ConsumeIntegralInRange<int>(1, 999'000);
+ }
+ MinimumFeeRateResult min_fee_rate{
+ CFeeRate{ConsumeMoney(fuzzed_data_provider, /*max=*/COIN)},
+ fee_reason,
+ returned_target};
+ (void)GetMinimumFeeRate(wallet, coin_control);
+ (void)GetMinimumFee(min_fee_rate, tx_bytes);
}
} // namespace
} // namespace wallet
### src/wallet/test/spend_tests.cpp
@@ -4,7 +4,6 @@
#include <consensus/amount.h>
#include <key.h>
-#include <policy/fees/block_policy_estimator.h>
#include <script/solver.h>
#include <validation.h>
#include <wallet/coincontrol.h>
### src/wallet/types.h
@@ -14,10 +14,25 @@
#ifndef BITCOIN_WALLET_TYPES_H
#define BITCOIN_WALLET_TYPES_H
-#include <policy/fees/block_policy_estimator.h>
+#include <consensus/amount.h>
+#include <policy/feerate.h>
+#include <primitives/transaction.h>
+#include <util/fees.h>
#include <util/translation.h>
+#include <optional>
+#include <utility>
+
namespace wallet {
+struct MinimumFeeRateResult {
+ CFeeRate fee_rate;
+ FeeReason fee_reason;
+ std::optional<int> returned_target;
+
+ MinimumFeeRateResult(CFeeRate fee_rate, FeeReason fee_reason, std::optional<int> returned_target)
+ : fee_rate{fee_rate}, fee_reason{fee_reason}, returned_target{std::move(returned_target)} {}
+};
+
/**
* Address purpose field that has been been stored with wallet sending and
* receiving addresses since BIP70 payment protocol support was added in
@@ -36,11 +51,11 @@ struct CreatedTransactionResult
{
CTransactionRef tx;
CAmount fee;
- FeeCalculation fee_calc;
+ FeeReason fee_reason;
std::optional<unsigned int> change_pos;
- CreatedTransactionResult(CTransactionRef _tx, CAmount _fee, std::optional<unsigned int> _change_pos, const FeeCalculation& _fee_calc)
- : tx(_tx), fee(_fee), fee_calc(_fee_calc), change_pos(_change_pos) {}
+ CreatedTransactionResult(CTransactionRef _tx, CAmount _fee, std::optional<unsigned int> _change_pos, FeeReason _fee_reason)
+ : tx(_tx), fee(_fee), fee_reason(_fee_reason), change_pos(_change_pos) {}
};
//! Machine-readable wallet error codes.
### test/functional/feature_fee_estimation.py
@@ -11,6 +11,9 @@
from test_framework.messages import (
COIN,
+ DEFAULT_BLOCK_RESERVED_WEIGHT,
+ MAX_BLOCK_WEIGHT,
+ WITNESS_SCALE_FACTOR,
)
from test_framework.test_framework import BitcoinTestFramework
from test_framework.util import (
@@ -27,6 +30,8 @@
SECONDS_PER_HOUR = 60 * 60
MIN_BUCKET_FEERATE = Decimal(100) / Decimal(COIN)
TXS_COUNT = 24
+BLOCK_POLICY_ESTIMATOR_ERROR = "Insufficient data or no feerate found"
+BLOCK_POLICY_ESTIMATOR_FILE_PATH = "fees/block_policy_estimates.dat"
def small_txpuzzle_randfee(
wallet, from_node, conflist, unconflist, amount, min_fee, fee_increment, batch_reqs
@@ -92,7 +97,7 @@ def check_smart_estimates(node, fees_seen):
"""Call estimatesmartfee and verify that the estimates meet certain invariants."""
delta = 1.0e-6 # account for rounding error
- all_smart_estimates = [node.estimatesmartfee(i) for i in range(1, 26)]
+ all_smart_estimates = [node.estimatesmartfee(i, "economical", {"fee_rate_estimator": "block_policy"}) for i in range(1, 26)]
mempoolMinFee = node.getmempoolinfo()["mempoolminfee"]
minRelaytxFee = node.getmempoolinfo()["minrelaytxfee"]
feerate_ceiling = max(max(fees_seen), float(mempoolMinFee), float(minRelaytxFee))
@@ -132,13 +137,22 @@ def make_tx(wallet, utxo, feerate):
)
def check_fee_estimates_btw_modes(node, expected_conservative, expected_economical):
- fee_est_conservative = node.estimatesmartfee(1, estimate_mode="conservative")['feerate']
- fee_est_economical = node.estimatesmartfee(1, estimate_mode="economical")['feerate']
- fee_est_default = node.estimatesmartfee(1)['feerate']
+ fee_est_conservative = node.estimatesmartfee(1, "conservative", {"fee_rate_estimator": "block_policy"})['feerate']
+ fee_est_economical = node.estimatesmartfee(1, "economical", {"fee_rate_estimator": "block_policy"})['feerate']
+ # Omit estimate_mode to check that the default mode is economical.
+ fee_est_default = node.estimatesmartfee(1, options={"fee_rate_estimator": "block_policy"})['feerate']
assert_equal(fee_est_conservative, expected_conservative)
assert_equal(fee_est_economical, expected_economical)
assert_equal(fee_est_default, expected_economical)
+def verify_estimate_response(estimate, feerate, errors):
+ if feerate is not None:
+ assert_equal(estimate["feerate"], feerate)
+ if errors:
+ assert all(err in estimate["errors"] for err in errors)
+ else:
+ assert "errors" not in estimate
+
class EstimateFeeTest(BitcoinTestFramework):
def set_test_params(self):
@@ -244,7 +258,7 @@ def sanity_check_estimates_range(self):
check_estimates(self.nodes[1], self.fees_per_kb)
def test_estimates_with_highminrelaytxfee(self):
- high_val = 3 * self.nodes[1].estimatesmartfee(2)["feerate"]
+ high_val = 3 * self.nodes[1].estimatesmartfee(2, "economical", {"fee_rate_estimator": "block_policy"})["feerate"]
self.restart_node(1, extra_args=[f"-minrelaytxfee={high_val}"])
check_smart_estimates(self.nodes[1], self.fees_per_kb)
self.restart_node(1)
@@ -309,138 +323,178 @@ def sanity_check_rbf_estimates(self):
# Only 10% of the transactions were really confirmed with a low feerate,
# the rest needed to be RBF'd. We must return the 90% conf rate feerate.
high_feerate_kvb = Decimal(high_feerate) / COIN * 10 ** 3
- est_feerate = node.estimatesmartfee(2)["feerate"]
+ est_feerate = node.estimatesmartfee(2, "economical", {"fee_rate_estimator": "block_policy"})["feerate"]
assert_equal(est_feerate, high_feerate_kvb)
def test_old_fee_estimate_file(self):
# Get the initial fee rate while node is running
- fee_rate = self.nodes[0].estimatesmartfee(1)["feerate"]
+ fee_rate = self.nodes[0].estimatesmartfee(1, "economical", {"fee_rate_estimator": "block_policy"})["feerate"]
- # Restart node to ensure fee_estimate.dat file is read
+ # Restart node to ensure block policy estimator file is read
self.restart_node(0)
- assert_equal(self.nodes[0].estimatesmartfee(1)["feerate"], fee_rate)
+ assert_equal(self.nodes[0].estimatesmartfee(1, "economical", {"fee_rate_estimator": "block_policy"})["feerate"], fee_rate)
- fee_dat = self.nodes[0].chain_path / "fee_estimates.dat"
+ block_policy_fee_dat = self.nodes[0].chain_path / BLOCK_POLICY_ESTIMATOR_FILE_PATH
+ legacy_fee_dat = self.nodes[0].chain_path / "fee_estimates.dat"
- # Stop the node and backdate the fee_estimates.dat file more than MAX_FILE_AGE
+ # If only the legacy fee_estimates.dat file exists, it is migrated to
+ # the new block policy estimator path.
self.stop_node(0)
- last_modified_time = time.time() - (MAX_FILE_AGE + 1) * SECONDS_PER_HOUR
- os.utime(fee_dat, (last_modified_time, last_modified_time))
+ os.rename(block_policy_fee_dat, legacy_fee_dat)
+ self.start_node(0)
+ assert_equal(self.nodes[0].estimatesmartfee(1, "economical", {"fee_rate_estimator": "block_policy"})["feerate"], fee_rate)
+ self.stop_node(0)
+ assert_equal(os.path.isfile(block_policy_fee_dat), True)
+ assert_equal(os.path.isfile(legacy_fee_dat), False)
- # Start node and ensure the fee_estimates.dat file was not read
+ # If both files exist, the new block policy estimator path is used and
+ # the obsolete legacy file is removed.
+ with open(legacy_fee_dat, "wb") as f:
+ f.write(b"ignored legacy fee estimates")
self.start_node(0)
- assert_equal(self.nodes[0].estimatesmartfee(1)["errors"], ["Insufficient data or no feerate found"])
+ assert_equal(self.nodes[0].estimatesmartfee(1, "economical", {"fee_rate_estimator": "block_policy"})["feerate"], fee_rate)
+ self.stop_node(0)
+ assert_equal(os.path.isfile(legacy_fee_dat), False)
+ # Stop the node and backdate the block policy estimator file more than MAX_FILE_AGE
+ last_modified_time = time.time() - (MAX_FILE_AGE + 1) * SECONDS_PER_HOUR
+ os.utime(block_policy_fee_dat, (last_modified_time, last_modified_time))
- def test_estimate_dat_is_flushed_periodically(self):
- fee_dat = self.nodes[0].chain_path / "fee_estimates.dat"
- os.remove(fee_dat) if os.path.exists(fee_dat) else None
+ # Start node and ensure the block policy estimator file was not read
+ self.start_node(0)
+ assert_equal(self.nodes[0].estimatesmartfee(1, "economical", {"fee_rate_estimator": "block_policy"})["errors"], [BLOCK_POLICY_ESTIMATOR_ERROR])
- # Verify that fee_estimates.dat does not exist
- assert_equal(os.path.isfile(fee_dat), False)
- # Verify if the string "Flushed fee estimates to fee_estimates.dat." is present in the debug log file.
- # If present, it indicates that fee estimates have been successfully flushed to disk.
- expected_messages = [f"Flushed fee estimates to {fee_dat}."]
+ def test_estimate_dat_is_flushed_periodically(self):
+ block_policy_fees_dat = self.nodes[0].chain_path / BLOCK_POLICY_ESTIMATOR_FILE_PATH
+ mempool_policy_dat = self.nodes[0].chain_path / "fees/mempool_policy_estimator.dat"
+ mempool_estimator_name_str = "mempool_policy"
+ os.remove(block_policy_fees_dat) if os.path.exists(block_policy_fees_dat) else None
+ os.remove(mempool_policy_dat) if os.path.exists(mempool_policy_dat) else None
+ if os.path.isdir(block_policy_fees_dat.parent):
+ os.rmdir(block_policy_fees_dat.parent)
+
+ # Verify that estimator data files and their parent directory do not exist
+ assert_equal(os.path.isfile(block_policy_fees_dat), False)
+ assert_equal(os.path.isfile(mempool_policy_dat), False)
+ assert_equal(os.path.isdir(block_policy_fees_dat.parent), False)
+ # Verify if the string "Flushed fee estimates to block_policy_estimates.dat." is present in the debug log file.
+ # If present, it indicates that fee estimator data has been successfully flushed to disk.
+ block_policy_estimator_message = f"Flushed fee estimates to {block_policy_fees_dat}."
+ mempool_policy_estimator_message = (
+ f"{mempool_estimator_name_str}: mined-block stats flushed to {mempool_policy_dat}."
+ )
+ expected_messages = [block_policy_estimator_message, mempool_policy_estimator_message]
with self.nodes[0].assert_debug_log(expected_msgs=expected_messages, timeout=1):
- # Mock the scheduler for an hour to flush fee estimates to fee_estimates.dat
+ # Mock the scheduler for an hour to flush estimator data.
self.nodes[0].mockscheduler(SECONDS_PER_HOUR)
- # Verify that fee estimates were flushed and fee_estimates.dat file is created
- assert_equal(os.path.isfile(fee_dat), True)
-
- # Verify that the estimates remain the same if there are no blocks in the flush interval
+ # Verify that estimator data was flushed and the estimator directory and files are created
+ assert_equal(os.path.isdir(block_policy_fees_dat.parent), True)
+ assert_equal(os.path.isfile(block_policy_fees_dat), True)
+ assert_equal(os.path.isfile(mempool_policy_dat), True)
+ # Verify that estimator data remains the same if there are no blocks in the flush interval
block_hash_before = self.nodes[0].getbestblockhash()
- fee_dat_initial_content = open(fee_dat, "rb").read()
+ block_policy_fees_dat_initial_content = open(block_policy_fees_dat, "rb").read()
+ mempool_policy_dat_initial_content = open(mempool_policy_dat, "rb").read()
with self.nodes[0].assert_debug_log(expected_msgs=expected_messages, timeout=1):
- # Mock the scheduler for an hour to flush fee estimates to fee_estimates.dat
+ # Mock the scheduler for an hour to flush estimator data.
self.nodes[0].mockscheduler(SECONDS_PER_HOUR)
-
# Verify that there were no blocks in between the flush interval
assert_equal(block_hash_before, self.nodes[0].getbestblockhash())
-
- fee_dat_current_content = open(fee_dat, "rb").read()
- assert_equal(fee_dat_current_content, fee_dat_initial_content)
-
- # Verify that the estimates remain the same after shutdown with no blocks before shutdown
+ block_policy_fees_dat_current_content = open(block_policy_fees_dat, "rb").read()
+ mempool_policy_dat_current_content = open(mempool_policy_dat, "rb").read()
+ assert_equal(block_policy_fees_dat_initial_content, block_policy_fees_dat_current_content)
+ assert_equal(mempool_policy_dat_initial_content, mempool_policy_dat_current_content)
+ # Verify that estimator data remains the same after shutdown with no blocks before shutdown
self.restart_node(0)
- fee_dat_current_content = open(fee_dat, "rb").read()
- assert_equal(fee_dat_current_content, fee_dat_initial_content)
-
- # Verify that the estimates are not the same if new blocks were produced in the flush interval
+ block_policy_fees_dat_current_content = open(block_policy_fees_dat, "rb").read()
+ mempool_policy_dat_current_content = open(mempool_policy_dat, "rb").read()
+ assert_equal(block_policy_fees_dat_initial_content, block_policy_fees_dat_current_content)
+ assert_equal(mempool_policy_dat_initial_content, mempool_policy_dat_current_content)
+ # Verify that estimator data changes if new blocks were produced in the flush interval
with self.nodes[0].assert_debug_log(expected_msgs=expected_messages, timeout=1):
- # Mock the scheduler for an hour to flush fee estimates to fee_estimates.dat
+ # Mock the scheduler for an hour to flush estimator data.
self.generate(self.nodes[0], 5, sync_fun=self.no_op)
self.nodes[0].mockscheduler(SECONDS_PER_HOUR)
-
- fee_dat_current_content = open(fee_dat, "rb").read()
- assert_not_equal(fee_dat_current_content, fee_dat_initial_content)
-
- fee_dat_initial_content = fee_dat_current_content
-
- # Generate blocks before shutdown and verify that the fee estimates are not the same
+ block_policy_fees_dat_current_content = open(block_policy_fees_dat, "rb").read()
+ assert_not_equal(block_policy_fees_dat_current_content, block_policy_fees_dat_initial_content)
+ block_policy_fees_dat_initial_content = block_policy_fees_dat_current_content
+ mempool_policy_dat_current_content = open(mempool_policy_dat, "rb").read()
+ assert_not_equal(mempool_policy_dat_current_content, mempool_policy_dat_initial_content)
+ mempool_policy_dat_initial_content = mempool_policy_dat_current_content
+ # Generate blocks before shutdown and verify that estimator data changes
self.generate(self.nodes[0], 5, sync_fun=self.no_op)
self.restart_node(0)
- fee_dat_current_content = open(fee_dat, "rb").read()
- assert_not_equal(fee_dat_current_content, fee_dat_initial_content)
+ block_policy_fees_dat_current_content = open(block_policy_fees_dat, "rb").read()
+ mempool_policy_dat_current_content = open(mempool_policy_dat, "rb").read()
+ assert_not_equal(block_policy_fees_dat_initial_content, block_policy_fees_dat_current_content)
+ assert_not_equal(mempool_policy_dat_initial_content, mempool_policy_dat_current_content)
def test_acceptstalefeeestimates_option(self):
# Get the initial fee rate while node is running
- fee_rate = self.nodes[0].estimatesmartfee(1)["feerate"]
+ fee_rate = self.nodes[0].estimatesmartfee(1, "economical", {"fee_rate_estimator": "block_policy"})["feerate"]
self.stop_node(0)
- fee_dat = self.nodes[0].chain_path / "fee_estimates.dat"
+ fee_dat = self.nodes[0].chain_path / BLOCK_POLICY_ESTIMATOR_FILE_PATH
- # Stop the node and backdate the fee_estimates.dat file more than MAX_FILE_AGE
+ # Stop the node and backdate the block policy estimator file more than MAX_FILE_AGE
last_modified_time = time.time() - (MAX_FILE_AGE + 1) * SECONDS_PER_HOUR
os.utime(fee_dat, (last_modified_time, last_modified_time))
- # Restart node with -acceptstalefeeestimates option to ensure fee_estimate.dat file is read
+ # Restart node with -acceptstalefeeestimates option to ensure block policy estimator file is read
self.start_node(0,extra_args=["-acceptstalefeeestimates"])
- assert_equal(self.nodes[0].estimatesmartfee(1)["feerate"], fee_rate)
+ assert_equal(self.nodes[0].estimatesmartfee(1, "economical", {"fee_rate_estimator": "block_policy"})["feerate"], fee_rate)
def clear_estimates(self):
self.log.info("Restarting node with fresh estimation")
self.stop_node(0)
- fee_dat = self.nodes[0].chain_path / "fee_estimates.dat"
+ fee_dat = self.nodes[0].chain_path / BLOCK_POLICY_ESTIMATOR_FILE_PATH
os.remove(fee_dat)
self.start_node(0)
self.connect_nodes(0, 1)
self.connect_nodes(0, 2)
self.sync_blocks()
- assert_equal(self.nodes[0].estimatesmartfee(1)["errors"], ["Insufficient data or no feerate found"])
+ assert_equal(self.nodes[0].estimatesmartfee(1, "economical", {"fee_rate_estimator": "block_policy"})["errors"], ["Insufficient data or no feerate found"])
- def broadcast_many(self, broadcaster, feerate, count, miner=None):
+ def broadcast_and_maybe_mine(self, broadcaster, feerate, txs, blocks=1, miner=None):
"""Broadcast and maybe mine some number of transactions with a specified fee rate."""
- tx_batch = []
- for _ in range(count):
- tx = self.wallet.create_self_transfer(fee_rate=feerate, utxo_to_spend=self.confutxo.pop(0))
- self.memutxo.append(tx["new_utxo"])
- tx_batch.append(tx)
- # To speed up the test, submit the transactions in batches to the nodes directly
- # avoiding having to wait for p2p to propagate them between the nodes.
- batch_send_tx = [broadcaster.sendrawtransaction.get_request(hexstring=tx["hex"]) for tx in tx_batch]
- for node in self.nodes:
- node.batch(batch_send_tx)
- self.sync_mempools(wait=0.1, nodes=[self.nodes[0], self.nodes[1], self.nodes[2]])
- if miner:
- mined = miner.getblock(self.generate(miner, 1)[0], True)["tx"]
- self.update_utxo(mined)
+ for _ in range(blocks):
+ tx_batch = []
+ for _ in range(txs):
+ tx = self.wallet.create_self_transfer(fee_rate=feerate, utxo_to_spend=self.confutxo.pop(0))
+ self.memutxo.append(tx["new_utxo"])
+ tx_batch.append(tx)
+ # To speed up the test, submit the transactions in batches to the nodes directly
+ # avoiding having to wait for p2p to propagate them between the nodes.
+ batch_send_tx = [broadcaster.sendrawtransaction.get_request(hexstring=tx["hex"]) for tx in tx_batch]
+ for node in self.nodes:
+ node.batch(batch_send_tx)
+ self.sync_mempools(wait=0.1, nodes=[self.nodes[0], self.nodes[1], self.nodes[2]])
+ if miner:
+ mined = miner.getblock(self.generate(miner, 1)[0], True)["tx"]
+ self.update_utxo(mined)
+
+ def send_transactions(self, utxos, fee_rate, target_vsize):
+ for utxo in utxos:
+ self.wallet.send_self_transfer(
+ from_node=self.nodes[0],
+ utxo_to_spend=utxo,
+ fee_rate=fee_rate,
+ target_vsize=target_vsize,
+ )
def test_estimation_modes(self):
low_feerate = Decimal("0.001")
high_feerate = Decimal("0.005")
# Broadcast and mine high fee transactions for the first 12 blocks.
- for _ in range(12):
- self.broadcast_many(self.nodes[1], high_feerate, TXS_COUNT, self.nodes[2])
+ self.broadcast_and_maybe_mine(self.nodes[1], high_feerate, TXS_COUNT, 12, self.nodes[2])
check_fee_estimates_btw_modes(self.nodes[0], high_feerate, high_feerate)
-
# We now track 12 blocks; short horizon stats will start decaying.
# Broadcast and mine low fee transactions for the next 4 blocks.
- for _ in range(4):
- self.broadcast_many(self.nodes[1], low_feerate, TXS_COUNT, self.nodes[2])
+ self.broadcast_and_maybe_mine(self.nodes[1], low_feerate, TXS_COUNT, 4, self.nodes[2])
# conservative mode will consider longer time horizons while economical mode does not
# Check the fee estimates for both modes after mining low fee transactions.
check_fee_estimates_btw_modes(self.nodes[0], high_feerate, low_feerate)
@@ -449,11 +503,117 @@ def test_sub_1s_per_vb_estimates(self):
feerate_0_5_s_per_vb = MIN_BUCKET_FEERATE * 5
feerate_1_s_per_vb = Decimal(1000) / Decimal(COIN)
for i in range(6):
- self.broadcast_many(self.nodes[1], feerate_0_5_s_per_vb, TXS_COUNT)
- self.broadcast_many(self.nodes[1], feerate_1_s_per_vb, TXS_COUNT, self.nodes[2])
- assert_equal(feerate_0_5_s_per_vb, self.nodes[0].estimatesmartfee(1)["feerate"])
+ self.broadcast_and_maybe_mine(self.nodes[1], feerate_0_5_s_per_vb, TXS_COUNT)
+ self.broadcast_and_maybe_mine(self.nodes[1], feerate_1_s_per_vb, TXS_COUNT, 1, self.nodes[2])
+ assert_equal(feerate_0_5_s_per_vb, self.nodes[0].estimatesmartfee(1, "economical", {"fee_rate_estimator": "block_policy"})["feerate"])
+ def test_estimatesmartfee_return_mempool_estimates(self):
+ node0 = self.nodes[0]
+ miner = self.nodes[1]
+ self.log.info("Ensure node0's mempool is empty at the start")
+ assert_equal(node0.getmempoolinfo()['size'], 0)
+ self.log.info("Test estimatesmartfee with empty mempool and no block policy estimator data")
+ estimate_after_restart = node0.estimatesmartfee(1, "economical", {"fee_rate_estimator": "none"})
+ verify_estimate_response(estimate_after_restart, None, [BLOCK_POLICY_ESTIMATOR_ERROR])
+ self.log.info("Populate block policy estimator with high-feerate history")
+ # Generate high-feerate transactions and mine them over 6 blocks to give block policy data.
+ high_feerate = Decimal("0.004")
+ self.broadcast_and_maybe_mine(node0, high_feerate, TXS_COUNT, 6, miner)
+ self.log.info("Test estimatesmartfee returns block policy estimator estimate when mempool is higher")
+ # Add 10 large insane-feerate transactions enough to generate a block template
+ num_txs = 10
+ target_vsize = int(((MAX_BLOCK_WEIGHT - DEFAULT_BLOCK_RESERVED_WEIGHT) / WITNESS_SCALE_FACTOR) / num_txs)
+ utxos = [self.wallet.get_utxo(confirmed_only=True) for _ in range(num_txs)]
+ insane_feerate = Decimal("0.01")
+ self.send_transactions(utxos, insane_feerate, target_vsize)
+ estimate_after_spike = node0.estimatesmartfee(1, "economical", {"verbosity": 2, "fee_rate_estimator": "none"})
+ assert_equal(len(estimate_after_spike["mempool_health_statistics"]), 6)
+ current_height = node0.getchaintips()[0]['height']
+ for block_stat in estimate_after_spike["mempool_health_statistics"]:
+ assert_equal(block_stat['block_height'], current_height)
+ current_height -= 1
+ assert block_stat['block_weight']
+ assert block_stat['mempool_txs_weight']
+ verify_estimate_response(estimate_after_spike, high_feerate, [])
+ assert_equal(estimate_after_spike["estimator"], "block_policy")
+ mempool_policy_estimate = node0.estimatesmartfee(1, "economical", {"fee_rate_estimator": "mempool_policy"})
+ verify_estimate_response(mempool_policy_estimate, insane_feerate, [])
+ # Confirm the spike transactions so they leave the mempool; the mined block
+ # keeps the mempool representation healthy. Then broadcast fresh low-feerate
+ # transactions so the mempool estimate is now the lower of the two.
+ self.generate(node0, 1, sync_fun=lambda: None)
+ assert_equal(node0.getmempoolinfo()['size'], 0)
+ low_feerate = Decimal("0.00004")
+ low_utxos = [self.wallet.get_utxo(confirmed_only=True) for _ in range(num_txs)]
+ self.send_transactions(low_utxos, low_feerate, target_vsize)
+ lower_estimate = node0.estimatesmartfee(1, "economical", {"fee_rate_estimator": "none"})
+ verify_estimate_response(lower_estimate, low_feerate, [])
+ # The mempool block stats are persisted across restarts, so the mempool
+ # stays healthy and the lower mempool estimate is still returned after a
+ # restart. Without persistence, the combined estimate would return a
+ # mempool-policy error until enough new blocks are observed.
+ self.restart_node(0)
+ estimate_post_restart = node0.estimatesmartfee(1, "economical", {"fee_rate_estimator": "none"})
+ verify_estimate_response(estimate_post_restart, low_feerate, [])
+
+ self.log.info("Test estimatesmartfee returns the fee rate floor when the mempool is empty but healthy")
+ self.generate(node0, 1, sync_fun=lambda: None)
+ assert_equal(node0.getmempoolinfo()['size'], 0)
+ block_policy_estimate = node0.estimatesmartfee(1, "economical", {"fee_rate_estimator": "block_policy"})
+ assert "feerate" in block_policy_estimate
+ # With an empty but healthy mempool the mempool estimator has no percentile data,
+ # so it falls back to the fee rate floor: the max of minrelaytxfee and mempoolminfee.
+ # That floor is lower than the block policy estimate, so the combined estimator returns it.
+ mempool_info = node0.getmempoolinfo()
+ floor = max(mempool_info["minrelaytxfee"], mempool_info["mempoolminfee"])
+ combined_estimate = node0.estimatesmartfee(1, "economical", {"fee_rate_estimator": "none"})
+ verify_estimate_response(combined_estimate, floor, [])
+ assert_equal(combined_estimate["estimator"], "mempool_policy")
+
+ def test_stale_mempool_block_stats_are_rejected_on_load(self):
+ # Persisted mempool block stats must be tied to the best block hash,
+ # not just height, because a reorg can replace the tip without
+ # changing the height.
+ node0 = self.nodes[0]
+ miner = self.nodes[1]
+ mempool_policy_dat = node0.chain_path / "fees/mempool_policy_estimator.dat"
+ healthy_feerate = Decimal("0.004")
+ self.connect_nodes(0, 1)
+ self.connect_nodes(0, 2)
+ self.sync_all()
+ # Build a full, healthy window whose tracked heights match the current tip.
+ self.broadcast_and_maybe_mine(node0, healthy_feerate, TXS_COUNT, 6, miner)
+ stale_stats = node0.estimatesmartfee(
+ 1,
+ "economical",
+ {"verbosity": 2, "fee_rate_estimator": "none"},
+ )["mempool_health_statistics"]
+ assert_equal(len(stale_stats), 6)
+ stale_height = node0.getblockcount()
+ assert_equal(stale_stats[0]["block_height"], stale_height)
+ stale_tip = node0.getbestblockhash()
+ self.stop_node(0)
+ stale_stats_snapshot = open(mempool_policy_dat, "rb").read()
+ self.start_node(0)
+ node0.invalidateblock(stale_tip)
+ assert_equal(node0.getblockcount(), stale_height - 1)
+ reorged_tip = self.generate(node0, 1, sync_fun=lambda: None)[0]
+ assert_equal(node0.getblockcount(), stale_height)
+ assert_not_equal(reorged_tip, stale_tip)
+ self.stop_node(0)
+ with open(mempool_policy_dat, "wb") as f:
+ f.write(stale_stats_snapshot)
+ self.start_node(0)
+ assert_equal(node0.getblockcount(), stale_height)
+ assert_equal(node0.getbestblockhash(), reorged_tip)
+ stats_after_restart = node0.estimatesmartfee(
+ 1,
+ "economical",
+ {"verbosity": 2, "fee_rate_estimator": "none"},
+ )["mempool_health_statistics"]
+ assert_equal(stats_after_restart, [])
+
def run_test(self):
self.log.info("This test is time consuming, please be patient")
self.log.info("Splitting inputs so we can generate tx's")
@@ -476,7 +636,7 @@ def run_test(self):
self.log.info("Testing estimates with single transactions.")
self.sanity_check_estimates_range()
- self.log.info("Test fee_estimates.dat is flushed periodically")
+ self.log.info("Test fees/block_policy_estimates.dat is flushed periodically")
self.test_estimate_dat_is_flushed_periodically()
# check that estimatesmartfee feerate is greater than or equal to maximum of mempoolminfee and minrelaytxfee
@@ -488,7 +648,7 @@ def run_test(self):
self.log.info("Test acceptstalefeeestimates option")
self.test_acceptstalefeeestimates_option()
- self.log.info("Test reading old fee_estimates.dat")
+ self.log.info("Test reading old block policy estimator file")
self.test_old_fee_estimate_file()
self.clear_estimates()
@@ -504,6 +664,13 @@ def run_test(self):
self.log.info("Test that estimatesmartfee returns a sub 1s/vb fee rate estimate")
self.test_sub_1s_per_vb_estimates()
+ self.log.info("Test that estimatesmartfee returns mempool estimates when lower")
+ self.clear_estimates()
+ self.test_estimatesmartfee_return_mempool_estimates()
+
+ self.log.info("Test that stale mempool block stats are rejected on load")
+ self.test_stale_mempool_block_stats_are_rejected_on_load()
+
self.log.info("Testing that fee estimation is disabled in blocksonly.")
self.restart_node(0, ["-blocksonly"])
assert_raises_rpc_error(
### test/functional/rpc_estimatefee.py
@@ -28,12 +28,19 @@ def run_test(self):
assert_raises_rpc_error(-3, "JSON value of type string is not of expected type number", self.nodes[0].estimaterawfee, 'foo')
# wrong type for estimatesmartfee(estimate_mode)
assert_raises_rpc_error(-3, "JSON value of type number is not of expected type string", self.nodes[0].estimatesmartfee, 1, 1)
+ # wrong type for estimatesmartfee(options.fee_rate_estimator)
+ assert_raises_rpc_error(-3, "JSON value of type number for field fee_rate_estimator is not of expected type string", self.nodes[0].estimatesmartfee, 1, 'ECONOMICAL', {'fee_rate_estimator': 1})
+ # wrong type for estimatesmartfee(options.verbosity)
+ assert_raises_rpc_error(-3, "JSON value of type string for field verbosity is not of expected type number", self.nodes[0].estimatesmartfee, 1, 'ECONOMICAL', {'verbosity': 'foo'})
# wrong type for estimaterawfee(threshold)
assert_raises_rpc_error(-3, "JSON value of type string is not of expected type number", self.nodes[0].estimaterawfee, 1, 'foo')
assert_raises_rpc_error(-8, 'Invalid estimate_mode parameter, must be one of: "unset", "economical", "conservative"', self.nodes[0].estimatesmartfee, 1, 'foo')
+ assert_raises_rpc_error(-8, "Unknown named parameter fee_rate_estimator", self.nodes[0].estimatesmartfee, 1, fee_rate_estimator=True)
+ assert_raises_rpc_error(-3, "Unexpected key block_policy_only", self.nodes[0].estimatesmartfee, 1, 'ECONOMICAL', {'block_policy_only': True})
# extra params
- assert_raises_rpc_error(-1, "estimatesmartfee", self.nodes[0].estimatesmartfee, 1, 'ECONOMICAL', 1)
+ assert_raises_rpc_error(-1, "estimatesmartfee", self.nodes[0].estimatesmartfee, 1, 'ECONOMICAL', {}, 1)
+ assert_raises_rpc_error(-1, "estimatesmartfee", self.nodes[0].estimatesmartfee, 1, 'ECONOMICAL', {'verbosity': 1}, 1)
assert_raises_rpc_error(-1, "estimaterawfee", self.nodes[0].estimaterawfee, 1, 1, 1)
# max value of 1008 per src/policy/fees/block_policy_estimator.h
@@ -45,6 +52,10 @@ def run_test(self):
self.nodes[0].estimatesmartfee(1, 'ECONOMICAL')
self.nodes[0].estimatesmartfee(1, 'unset')
self.nodes[0].estimatesmartfee(1, 'conservative')
+ self.nodes[0].estimatesmartfee(1, 'ECONOMICAL', {"fee_rate_estimator": "block_policy"})
+ self.nodes[0].estimatesmartfee(1, 'ECONOMICAL', {"fee_rate_estimator": "mempool_policy"})
+ self.nodes[0].estimatesmartfee(1, 'ECONOMICAL', {"fee_rate_estimator": "foo"})
+ self.nodes[0].estimatesmartfee(1, 'ECONOMICAL', {'verbosity': 1, 'fee_rate_estimator': "none"})
self.nodes[0].estimaterawfee(1)
self.nodes[0].estimaterawfee(1, None)
### test/functional/test_framework/test_framework.py
@@ -965,6 +965,7 @@ def cache_path(*paths):
return os.path.join(cache_node_dir, self.chain, *paths)
os.rmdir(cache_path('wallets')) # Remove empty wallets dir
+ shutil.rmtree(cache_path('fees'), ignore_errors=True)
for entry in os.listdir(cache_path()):
if entry not in ['chainstate', 'blocks', 'indexes']: # Only indexes, chainstate and blocks folders
os.remove(cache_path(entry))
### test/functional/wallet_bumpfee.py
@@ -511,24 +511,25 @@ def test_small_output_with_feerate_succeeds(self, rbf_node, dest_address):
def test_dust_to_fee(self, rbf_node, dest_address):
self.log.info('Test that bumped output that is dust is dropped to fee')
- rbfid = spend_one_input(rbf_node, dest_address)
+ rbfid = spend_one_input(rbf_node, dest_address, change_size=Decimal("0.00030000"), dest_amount=Decimal("0.00064730"))
fulltx = rbf_node.getrawtransaction(rbfid, 1)
# The DER formatting used by Bitcoin to serialize ECDSA signatures means that signatures can have a
# variable size of 70-72 bytes (or possibly even less), with most being 71 or 72 bytes. The signature
# in the witness is divided by 4 for the vsize, so this variance can take the weight across a 4-byte
# boundary. Thus expected transaction size (p2wpkh, 1 input, 2 outputs) is 140-141 vbytes, usually 141.
if not 140 <= fulltx["vsize"] <= 141:
raise AssertionError("Invalid tx vsize of {} (140-141 expected), full tx: {}".format(fulltx["vsize"], fulltx))
- # Bump with fee_rate of 350.25 sat/vB vbytes to create dust.
- # Expected fee is 141 vbytes * fee_rate 0.00350250 BTC / 1000 vbytes = 0.00049385 BTC.
- # or occasionally 140 vbytes * fee_rate 0.00350250 BTC / 1000 vbytes = 0.00049035 BTC.
- # Dust should be dropped to the fee, so actual bump fee is 0.00050000 BTC.
- bumped_tx = rbf_node.bumpfee(rbfid, fee_rate=350.25)
+ # Bump with fee_rate of 250 sat/vB. The leftover change is below the dust
+ # threshold, so it is dropped and folded into the fee.
+ # Target fee is 141 vbytes * 0.00250000 BTC / 1000 vbytes = 0.00035250 BTC (20 sat change left),
+ # or occasionally 140 vbytes * 0.00250000 BTC / 1000 vbytes = 0.00035000 BTC (270 sat change left).
+ # Either way the sub-dust change is added to the fee, giving 0.00035270 BTC.
+ bumped_tx = rbf_node.bumpfee(rbfid, fee_rate=250)
full_bumped_tx = rbf_node.getrawtransaction(bumped_tx["txid"], 1)
- assert_equal(bumped_tx["fee"], Decimal("0.00050000"))
+ assert_equal(bumped_tx["fee"], Decimal("0.00035270"))
assert_equal(len(fulltx["vout"]), 2)
assert_equal(len(full_bumped_tx["vout"]), 1) # change output is eliminated
- assert_equal(full_bumped_tx["vout"][0]['value'], Decimal("0.00050000"))
+ assert_equal(full_bumped_tx["vout"][0]['value'], Decimal("0.00064730"))
self.clear_mempool()
def test_maxtxfee_fails(self, rbf_node, dest_address):
@@ -755,10 +756,10 @@ def test_change_script_match(self, rbf_node, dest_address):
self.clear_mempool()
-def spend_one_input(node, dest_address, change_size=Decimal("0.00049000"), data=None):
+def spend_one_input(node, dest_address, change_size=Decimal("0.00049000"), data=None, dest_amount=Decimal("0.00050000")):
tx_input = dict(
sequence=MAX_BIP125_RBF_SEQUENCE, **next(u for u in node.listunspent() if u["amount"] == Decimal("0.00100000")))
- destinations = {dest_address: Decimal("0.00050000")}
+ destinations = {dest_address: dest_amount}
if change_size > 0:
destinations[node.getrawchangeaddress()] = change_size
if data:
### test/functional/wallet_taproot.py
@@ -165,8 +165,9 @@ def do_test_sendtoaddress(self, comment, pattern, privmap, treefn, keys_pay, key
self.generatetoaddress(self.nodes[0], 1, self.boring.getnewaddress(), sync_fun=self.no_op)
assert rpc_online.gettransaction(res)["confirmations"] > 0
- # Cleanup
- txid = rpc_online.sendall(recipients=[self.boring.getnewaddress()])["txid"]
+ # Match the framework fallbackfee; otherwise the underestimated taproot
+ # script-path spend size can produce an effective feerate below min relay.
+ txid = rpc_online.sendall(recipients=[self.boring.getnewaddress()], fee_rate=20)["txid"]
self.generatetoaddress(self.nodes[0], 1, self.boring.getnewaddress(), sync_fun=self.no_op)
assert rpc_online.gettransaction(txid)["confirmations"] > 0
rpc_online.unloadwallet()
@@ -238,8 +239,9 @@ def do_test_psbt(self, comment, pattern, privmap, treefn, keys_pay, keys_change)
self.generatetoaddress(self.nodes[0], 1, self.boring.getnewaddress(), sync_fun=self.no_op)
assert psbt_online.gettransaction(txid)['confirmations'] > 0
- # Cleanup
- psbt = psbt_online.sendall(recipients=[self.boring.getnewaddress()], psbt=True)["psbt"]
+ # Match the framework fallbackfee; otherwise the underestimated taproot
+ # script-path spend size can produce an effective feerate below min relay.
+ psbt = psbt_online.sendall(recipients=[self.boring.getnewaddress()], psbt=True, fee_rate=20)["psbt"]
res = psbt_offline.walletprocesspsbt(psbt=psbt, finalize=False)
rawtx = self.nodes[0].finalizepsbt(res['psbt'])['hex']
txid = self.nodes[0].sendrawtransaction(rawtx)Why this scored 40/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.