cryptonote_core: improve block template backlog handling
What changed, and why it matters
This Monero commit fixes a bug in how mining software picks transactions for the next block. The old code could accidentally include two transactions that spend the same 'key image' (like the same coin being spent twice), which would create an invalid block. It also cleans up and speeds up the transaction selection logic. The commit message explicitly calls this a 'key image conflict issue' and says it is fixed.
Treat this as a security-relevant correctness fix and include it in the next release. Mining pools, solo miners, and RPC consumers of get_miner_data should upgrade. Review the new get_block_template_backlog() behavior with overpick ratios to ensure RPC clients do not misinterpret non-profitable backlog entries as guaranteed inclusions.
Security signals we found
Fixes key image conflict in block template construction (commit message)
Replaces lazy transaction-prefix parsing with direct TXID-key_image index lookup for conflict detection
Changes key_images_container to boost::bimap for consistent bidirectional key-image/TXID mapping
Hardens insert_key_images and remove_transaction_keyimages against duplicate or mismatched entries
Adds overflow checks in block weight/fee accumulation
Removes deprecated DEBUG_CREATE_BLOCK_TEMPLATE validation block
Evidence from the diff
The patch refactors tx_memory_pool::fill_block_template() and get_block_template_backlog() into a single get_block_template_backlog() implementation. The key security change is that key-image conflict detection now uses the in-memory m_spent_key_images bimap keyed by TXID, instead of relying on lazy deserialization of the transaction prefix. Previously, the candidate transaction might never be parsed, so its key images could be missed and a conflicting transaction could be added to the block template. The patch also changes m_spent_key_images from unordered_map
Changed components
src/cryptonote_core/tx_pool.cppsrc/cryptonote_core/tx_pool.hsrc/cryptonote_core/blockchain.cppsrc/cryptonote_config.htests/functional_tests/txpool.pyInspect captured patch +277 / −355
### src/cryptonote_config.h
@@ -1,4 +1,4 @@
-// Copyright (c) 2014-2024, The Monero Project
+// Copyright (c) 2014-2026, The Monero Project
//
// All rights reserved.
//
### src/cryptonote_core/blockchain.cpp
@@ -83,6 +83,9 @@ DISABLE_VS_WARNINGS(4267)
// used to overestimate the block reward when estimating a per kB to use
#define BLOCK_REWARD_OVERESTIMATE (10 * 1000000000000)
+/// @brief Parameter for block template backlog fetching: targets total weight 112.5% of the block's limit
+#define DEFAULT_RPC_TX_BACKLOG_OVERPICK 0.125f
+
//------------------------------------------------------------------
Blockchain::Blockchain(tx_memory_pool& tx_pool) :
m_db(), m_tx_pool(tx_pool), m_hardfork(NULL), m_timestamps_and_difficulties_height(0), m_reset_timestamps_and_difficulties_height(true), m_current_block_cumul_weight_limit(0), m_current_block_cumul_weight_median(0),
@@ -1513,11 +1516,6 @@ uint64_t Blockchain::get_current_cumulative_block_weight_median() const
// unchanged for the time being.
//
// This function makes a new block for a miner to mine the hash for
-//
-// FIXME: this codebase references #if defined(DEBUG_CREATE_BLOCK_TEMPLATE)
-// in a lot of places. That flag is not referenced in any of the code
-// nor any of the makefiles, however. Need to look into whether or not it's
-// necessary at all.
bool Blockchain::create_block_template(block& b, const crypto::hash *from_block, const account_public_address& miner_address, difficulty_type& diffic, uint64_t& height, uint64_t& expected_reward, uint64_t& cumulative_weight, const blobdata& ex_nonce, uint64_t &seed_height, crypto::hash &seed_hash, bool include_sensitive)
{
LOG_PRINT_L3("Blockchain::" << __func__);
@@ -1663,65 +1661,28 @@ bool Blockchain::create_block_template(block& b, const crypto::hash *from_block,
CHECK_AND_ASSERT_MES(diffic, false, "difficulty overhead.");
- size_t txs_weight;
- uint64_t fee;
- if (!m_tx_pool.fill_block_template(b, median_weight, already_generated_coins, txs_weight, fee, expected_reward, b.major_version, include_sensitive))
+ std::vector<tx_block_template_backlog_entry> selected_backlog;
+ if (!m_tx_pool.get_block_template_backlog(median_weight, already_generated_coins,
+ b.major_version, /*overpick=*/0, include_sensitive, selected_backlog))
{
return false;
}
pool_cookie = m_tx_pool.cookie();
-#if defined(DEBUG_CREATE_BLOCK_TEMPLATE)
- size_t real_txs_weight = 0;
- uint64_t real_fee = 0;
- for(crypto::hash &cur_hash: b.tx_hashes)
- {
- auto cur_res = m_tx_pool.m_transactions.find(cur_hash);
- if (cur_res == m_tx_pool.m_transactions.end())
- {
- LOG_ERROR("Creating block template: error: transaction not found");
- continue;
- }
- tx_memory_pool::tx_details &cur_tx = cur_res->second;
- real_txs_weight += cur_tx.weight;
- real_fee += cur_tx.fee;
- if (cur_tx.weight != get_transaction_weight(cur_tx.tx))
- {
- LOG_ERROR("Creating block template: error: invalid transaction weight");
- }
- if (cur_tx.tx.version == 1)
- {
- uint64_t inputs_amount;
- if (!get_inputs_money_amount(cur_tx.tx, inputs_amount))
- {
- LOG_ERROR("Creating block template: error: cannot get inputs amount");
- }
- else if (cur_tx.fee != inputs_amount - get_outs_money_amount(cur_tx.tx))
- {
- LOG_ERROR("Creating block template: error: invalid fee");
- }
- }
- else
- {
- if (cur_tx.fee != cur_tx.tx.rct_signatures.txnFee)
- {
- LOG_ERROR("Creating block template: error: invalid fee");
- }
- }
- }
- if (txs_weight != real_txs_weight)
- {
- LOG_ERROR("Creating block template: error: wrongly calculated transaction weight");
- }
- if (fee != real_fee)
+
+ // collect TXID list and total stats
+ uint64_t txs_weight = 0;
+ uint64_t fee = 0;
+ b.tx_hashes.reserve(selected_backlog.size());
+ for (const tx_block_template_backlog_entry &e : selected_backlog)
{
- LOG_ERROR("Creating block template: error: wrongly calculated fee");
+ CHECK_AND_ASSERT_MES(std::numeric_limits<uint64_t>::max() - e.weight >= txs_weight,
+ false, "Overflow in block weight calculation");
+ CHECK_AND_ASSERT_MES(std::numeric_limits<uint64_t>::max() - e.fee >= fee,
+ false, "Overflow in block total fee calculation");
+ txs_weight += e.weight;
+ fee += e.fee;
+ b.tx_hashes.push_back(e.id);
}
- MDEBUG("Creating block template: height " << height <<
- ", median weight " << median_weight <<
- ", already generated coins " << already_generated_coins <<
- ", transaction weight " << txs_weight <<
- ", fee " << fee);
-#endif
/*
two-phase miner transaction generation: we don't know exact block weight until we prepare block, but we don't know reward until we know
@@ -1730,37 +1691,39 @@ bool Blockchain::create_block_template(block& b, const crypto::hash *from_block,
//make blocks coin-base tx looks close to real coinbase tx to get truthful blob weight
uint8_t hf_version = b.major_version;
size_t max_outs = hf_version >= 4 ? 1 : 11;
- bool r = construct_miner_tx(height, median_weight, already_generated_coins, txs_weight, fee, miner_address, b.miner_tx, ex_nonce, max_outs, hf_version);
- CHECK_AND_ASSERT_MES(r, false, "Failed to construct miner tx, first chance");
- cumulative_weight = txs_weight + get_transaction_weight(b.miner_tx);
-#if defined(DEBUG_CREATE_BLOCK_TEMPLATE)
- MDEBUG("Creating block template: miner tx weight " << get_transaction_weight(b.miner_tx) <<
- ", cumulative weight " << cumulative_weight);
-#endif
+ cumulative_weight = txs_weight;
for (size_t try_count = 0; try_count != 10; ++try_count)
{
- r = construct_miner_tx(height, median_weight, already_generated_coins, cumulative_weight, fee, miner_address, b.miner_tx, ex_nonce, max_outs, hf_version);
+ const bool miner_tx_made = construct_miner_tx(height, median_weight, already_generated_coins,
+ cumulative_weight, fee, miner_address, b.miner_tx, ex_nonce, max_outs, hf_version);
+
+ // On miner tx creation failure, if we have txs to pop, try that...
+ if (!miner_tx_made)
+ {
+ CHECK_AND_ASSERT_MES(!b.tx_hashes.empty(),
+ false, "Failed to create miner transaction, but have no non-miner txs to pop");
+ assert(selected_backlog.size() == b.tx_hashes.size());
+ b.tx_hashes.pop_back();
+ const tx_block_template_backlog_entry &backlog_entry = selected_backlog.back();
+ assert(txs_weight >= backlog_entry.weight);
+ txs_weight -= backlog_entry.weight;
+ assert(fee >= backlog_entry.fee);
+ fee -= backlog_entry.fee;
+ selected_backlog.pop_back();
+ cumulative_weight = txs_weight;
+ continue;
+ }
- CHECK_AND_ASSERT_MES(r, false, "Failed to construct miner tx, second chance");
size_t coinbase_weight = get_transaction_weight(b.miner_tx);
if (coinbase_weight > cumulative_weight - txs_weight)
{
cumulative_weight = txs_weight + coinbase_weight;
-#if defined(DEBUG_CREATE_BLOCK_TEMPLATE)
- MDEBUG("Creating block template: miner tx weight " << coinbase_weight <<
- ", cumulative weight " << cumulative_weight << " is greater than before");
-#endif
continue;
}
if (coinbase_weight < cumulative_weight - txs_weight)
{
size_t delta = cumulative_weight - txs_weight - coinbase_weight;
-#if defined(DEBUG_CREATE_BLOCK_TEMPLATE)
- MDEBUG("Creating block template: miner tx weight " << coinbase_weight <<
- ", cumulative weight " << txs_weight + coinbase_weight <<
- " is less than before, adding " << delta << " zero bytes");
-#endif
b.miner_tx.extra.insert(b.miner_tx.extra.end(), delta, 0);
//here could be 1 byte difference, because of extra field counter is varint, and it can become from 1-byte len to 2-bytes len.
if (cumulative_weight != txs_weight + get_transaction_weight(b.miner_tx))
@@ -1778,11 +1741,8 @@ bool Blockchain::create_block_template(block& b, const crypto::hash *from_block,
}
}
CHECK_AND_ASSERT_MES(cumulative_weight == txs_weight + get_transaction_weight(b.miner_tx), false, "unexpected case: cumulative_weight=" << cumulative_weight << " is not equal txs_cumulative_weight=" << txs_weight << " + get_transaction_weight(b.miner_tx)=" << get_transaction_weight(b.miner_tx));
-#if defined(DEBUG_CREATE_BLOCK_TEMPLATE)
- MDEBUG("Creating block template: miner tx weight " << coinbase_weight <<
- ", cumulative weight " << cumulative_weight << " is now good");
-#endif
+ expected_reward = get_outs_money_amount(b.miner_tx);
if (!from_block)
cache_block_template(b, miner_address, ex_nonce, diffic, height, expected_reward, cumulative_weight, seed_height, seed_hash, pool_cookie, include_sensitive);
return true;
@@ -1815,9 +1775,8 @@ bool Blockchain::get_miner_data(uint8_t& major_version, uint64_t& height, crypto
median_weight = m_current_block_cumul_weight_median;
already_generated_coins = m_db->get_block_already_generated_coins(height - 1);
- m_tx_pool.get_block_template_backlog(tx_backlog);
-
- return true;
+ return m_tx_pool.get_block_template_backlog(median_weight, already_generated_coins, major_version,
+ DEFAULT_RPC_TX_BACKLOG_OVERPICK, /*include_sensitive=*/false, tx_backlog);
}
//------------------------------------------------------------------
// for an alternate chain, get the timestamps from the main chain to complete
@@ -5606,8 +5565,8 @@ bool Blockchain::for_all_outputs(uint64_t amount, std::function<bool(uint64_t he
void Blockchain::invalidate_block_template_cache()
{
- MDEBUG("Invalidating block template cache");
- m_btc_valid = false;
+ if (std::exchange(m_btc_valid, false))
+ MDEBUG("Invalidating block template cache");
}
void Blockchain::cache_block_template(const block &b, const cryptonote::account_public_address &address, const blobdata &nonce, const difficulty_type &diff, uint64_t height, uint64_t expected_reward, uint64_t cumulative_weight, uint64_t seed_height, const crypto::hash &seed_hash, uint64_t pool_cookie, bool include_sensitive)
@@ -5635,9 +5594,18 @@ void Blockchain::send_miner_notifications(uint64_t height, const crypto::hash &s
const uint8_t major_version = m_hardfork->get_ideal_version(height);
const difficulty_type diff = get_difficulty_for_next_block();
const uint64_t median_weight = m_current_block_cumul_weight_median;
+ const float overpick = DEFAULT_RPC_TX_BACKLOG_OVERPICK;
std::vector<tx_block_template_backlog_entry> tx_backlog;
- m_tx_pool.get_block_template_backlog(tx_backlog);
+ if (!m_tx_pool.get_block_template_backlog(median_weight, already_generated_coins, major_version,
+ overpick, /*include_sensitive=*/false, tx_backlog))
+ {
+ MERROR("Could not retrieve mempool's block template backlog for block " << height
+ << ": median weight " << median_weight << " already generated coins "
+ << already_generated_coins << " major version " << major_version
+ << " overpick ratio " << overpick << ". Miner notifications will not be sent.");
+ return;
+ }
for (const auto& notifier : m_miner_notifiers)
{
### src/cryptonote_core/tx_pool.cpp
@@ -1,4 +1,4 @@
-// Copyright (c) 2014-2024, The Monero Project
+// Copyright (c) 2014-2026, The Monero Project
//
// All rights reserved.
//
@@ -36,12 +36,10 @@
#include "tx_pool.h"
#include "cryptonote_tx_utils.h"
-#include "cryptonote_basic/events.h"
#include "cryptonote_config.h"
#include "blockchain.h"
#include "blockchain_db/locked_txn.h"
#include "blockchain_db/blockchain_db.h"
-#include "int-util.h"
#include "misc_language.h"
#include "misc_log_ex.h"
#include "tx_verification_utils.h"
@@ -107,11 +105,6 @@ namespace cryptonote
return d;
}
- uint64_t template_accept_threshold(uint64_t amount)
- {
- return amount * ACCEPT_THRESHOLD;
- }
-
// external lock must be held for the comparison+set to work properly
void set_if_less(std::atomic<time_t>& next_check, const time_t candidate) noexcept
{
@@ -497,27 +490,43 @@ namespace cryptonote
//---------------------------------------------------------------------------------
bool tx_memory_pool::insert_key_images(const transaction_prefix &tx, const crypto::hash &id, relay_method tx_relay)
{
+ bool previously_private = false;
+
+ // collect key images for existing TXID, *should* be empty or matches set in `tx`
+ std::unordered_set<crypto::key_image> existing_tx_key_images;
+ existing_tx_key_images.reserve(tx.vin.size());
+ const auto existing_tx_range = m_spent_key_images.right.equal_range(id);
+ for (auto it = existing_tx_range.first; it != existing_tx_range.second; ++it) {
+ existing_tx_key_images.insert(it->second);
+ }
+
for(const auto& in: tx.vin)
{
CHECKED_GET_SPECIFIC_VARIANT(in, const txin_to_key, txin, false);
- std::unordered_set<crypto::hash>& kei_image_set = m_spent_key_images[txin.k_image];
// Only allow multiple txes per key-image if kept-by-block. Only allow
// the same txid if going from local/stem->fluff.
if (tx_relay != relay_method::block)
{
- const bool one_txid =
- (kei_image_set.empty() || (kei_image_set.size() == 1 && *(kei_image_set.cbegin()) == id));
- CHECK_AND_ASSERT_MES(one_txid, false, "internal error: tx_relay=" << unsigned(tx_relay)
- << ", kei_image_set.size()=" << kei_image_set.size() << ENDL << "txin.k_image=" << txin.k_image << ENDL
- << "tx_id=" << id);
+ const auto range = m_spent_key_images.left.equal_range(txin.k_image);
+ const auto it = range.first;
+ const bool no_other_tx = (it == range.second) || (it->second == id && std::next(it) == range.second);
+ CHECK_AND_ASSERT_MES(no_other_tx, false,
+ "insert_key_images: trying to insert multiple non-kept-by-block tx for existing key image:" << ENDL
+ << " tx_relay=" << unsigned(tx_relay) << ENDL
+ << " txin.k_image=" << txin.k_image << ENDL
+ << " tx_id=" << id);
}
- const bool new_or_previously_private =
- kei_image_set.insert(id).second ||
- !m_blockchain.txpool_tx_matches_category(id, relay_category::legacy);
- CHECK_AND_ASSERT_MES(new_or_previously_private, false, "internal error: try to insert duplicate iterator in key_image set");
+ if (!existing_tx_key_images.insert(txin.k_image).second)
+ {
+ if (!previously_private)
+ previously_private = !m_blockchain.txpool_tx_matches_category(id, relay_category::legacy);
+ CHECK_AND_ASSERT_MES(previously_private, false, "internal error: try to insert duplicate iterator in key_image set");
+ }
+ else
+ m_spent_key_images.insert(key_images_container::value_type(txin.k_image, id));
}
++m_cookie;
return true;
@@ -530,28 +539,12 @@ namespace cryptonote
{
CRITICAL_REGION_LOCAL(m_transactions_lock);
CRITICAL_REGION_LOCAL1(m_blockchain);
- // ND: Speedup
- for(const txin_v& vi: tx.vin)
- {
- CHECKED_GET_SPECIFIC_VARIANT(vi, const txin_to_key, txin, false);
- auto it = m_spent_key_images.find(txin.k_image);
- CHECK_AND_ASSERT_MES(it != m_spent_key_images.end(), false, "failed to find transaction input in key images. img=" << txin.k_image << ENDL
- << "transaction id = " << actual_hash);
- std::unordered_set<crypto::hash>& key_image_set = it->second;
- CHECK_AND_ASSERT_MES(key_image_set.size(), false, "empty key_image set, img=" << txin.k_image << ENDL
- << "transaction id = " << actual_hash);
-
- auto it_in_set = key_image_set.find(actual_hash);
- CHECK_AND_ASSERT_MES(it_in_set != key_image_set.end(), false, "transaction id not found in key_image set, img=" << txin.k_image << ENDL
- << "transaction id = " << actual_hash);
- key_image_set.erase(it_in_set);
- if(!key_image_set.size())
- {
- //it is now empty hash container for this key_image
- m_spent_key_images.erase(it);
- }
- }
+ const size_t n_kis_erased = m_spent_key_images.right.erase(actual_hash);
+ CHECK_AND_ASSERT_MES(n_kis_erased == tx.vin.size(), false,
+ "Erased wrong number of TXID<->KI entries in in-memory cache for TXID" << actual_hash
+ << ". Expected " << tx.vin.size() << " vs actual " << n_kis_erased);
+
++m_cookie;
return true;
}
@@ -1106,67 +1099,6 @@ namespace cryptonote
}, false, category);
}
//------------------------------------------------------------------
- void tx_memory_pool::get_block_template_backlog(std::vector<tx_block_template_backlog_entry>& backlog, bool include_sensitive) const
- {
- CRITICAL_REGION_LOCAL(m_transactions_lock);
- CRITICAL_REGION_LOCAL1(m_blockchain);
-
- std::vector<tx_block_template_backlog_entry> tmp;
- uint64_t total_weight = 0;
-
- // First get everything from the mempool, filter it later
- m_blockchain.for_all_txpool_txes([&tmp, &total_weight](const crypto::hash &txid, const txpool_tx_meta_t &meta, const cryptonote::blobdata_ref*){
- tmp.emplace_back(tx_block_template_backlog_entry{txid, meta.weight, meta.fee});
- total_weight += meta.weight;
- return true;
- }, false, include_sensitive ? relay_category::all : relay_category::broadcasted);
-
- // Limit backlog to 112.5% of current median weight. This is enough to mine a full block with the optimal block reward
- const uint64_t median_weight = m_blockchain.get_current_cumulative_block_weight_median();
- const uint64_t max_backlog_weight = median_weight + (median_weight / 8);
-
- // If the total weight is too high, choose the best paying transactions
- if (total_weight > max_backlog_weight)
- std::stable_sort(tmp.begin(), tmp.end(), [](const auto& a, const auto& b){ return a.fee * b.weight > b.fee * a.weight; });
-
- backlog.clear();
- uint64_t w = 0;
-
- std::unordered_set<crypto::key_image> k_images;
-
- for (const tx_block_template_backlog_entry& e : tmp)
- {
- try
- {
- txpool_tx_meta_t meta;
- if (!m_blockchain.get_txpool_tx_meta(e.id, meta))
- continue;
-
- cryptonote::blobdata txblob;
- if (!m_blockchain.get_txpool_tx_blob(e.id, txblob, relay_category::all))
- continue;
-
- cryptonote::transaction tx;
- if (is_transaction_ready_to_go(meta, e.id, txblob, tx))
- {
- if (have_key_images(k_images, tx))
- continue;
- append_key_images(k_images, tx);
-
- backlog.push_back(e);
- w += e.weight;
- if (w > max_backlog_weight)
- break;
- }
- }
- catch (const std::exception &e)
- {
- MERROR("Failed to check transaction readiness: " << e.what());
- // continue, not fatal
- }
- }
- }
- //------------------------------------------------------------------
void tx_memory_pool::get_transaction_stats(struct txpool_stats& stats, bool include_sensitive) const
{
CRITICAL_REGION_LOCAL(m_transactions_lock);
@@ -1292,13 +1224,16 @@ namespace cryptonote
return true;
}, true, relay_category::all);
- for (const key_images_container::value_type& kee : m_spent_key_images) {
- const crypto::key_image& k_image = kee.first;
- const std::unordered_set<crypto::hash>& kei_image_set = kee.second;
+ for (auto ki_it = m_spent_key_images.left.begin(); ki_it != m_spent_key_images.left.end();) {
+ const crypto::key_image& k_image = ki_it->first;
+ const auto ki_range = m_spent_key_images.left.equal_range(k_image);
+ assert(ki_range.first == ki_it);
+ assert(ki_range.first != ki_range.second);
spent_key_image_info ki;
ki.id_hash = epee::string_tools::pod_to_hex(k_image);
- for (const crypto::hash& tx_id_hash : kei_image_set)
+ for (; ki_it != ki_range.second; ++ki_it)
{
+ const crypto::hash &tx_id_hash = ki_it->second;
if (m_blockchain.txpool_tx_matches_category(tx_id_hash, relay_category::all))
ki.txs_hashes.push_back(epee::string_tools::pod_to_hex(tx_id_hash));
}
@@ -1343,17 +1278,22 @@ namespace cryptonote
return true;
}, true, relay_category::broadcasted);
- for (const key_images_container::value_type& kee : m_spent_key_images) {
+ for (auto ki_it = m_spent_key_images.left.begin(); ki_it != m_spent_key_images.left.end();) {
+ const crypto::key_image& k_image = ki_it->first;
+ const auto ki_range = m_spent_key_images.left.equal_range(k_image);
+ assert(ki_range.first == ki_it);
+ assert(ki_range.first != ki_range.second);
std::vector<crypto::hash> tx_hashes;
- const std::unordered_set<crypto::hash>& kei_image_set = kee.second;
- for (const crypto::hash& tx_id_hash : kei_image_set)
+ for (; ki_it != ki_range.second; ++ki_it)
{
+ const crypto::hash &tx_id_hash = ki_it->second;
if (m_blockchain.txpool_tx_matches_category(tx_id_hash, relay_category::broadcasted))
tx_hashes.push_back(tx_id_hash);
}
if (!tx_hashes.empty())
- key_image_infos[kee.first] = std::move(tx_hashes);
+ key_image_infos[k_image] = std::move(tx_hashes);
+
}
return true;
}
@@ -1368,12 +1308,9 @@ namespace cryptonote
for (const auto& image : key_images)
{
bool is_spent = false;
- const auto found = m_spent_key_images.find(image);
- if (found != m_spent_key_images.end())
- {
- for (const crypto::hash& tx_hash : found->second)
- is_spent |= m_blockchain.txpool_tx_matches_category(tx_hash, relay_category::broadcasted);
- }
+ const auto ki_range = m_spent_key_images.left.equal_range(image);
+ for (auto ki_it = ki_range.first; ki_it != ki_range.second; ++ki_it)
+ is_spent |= m_blockchain.txpool_tx_matches_category(ki_it->second, relay_category::broadcasted);
spent.push_back(is_spent);
}
@@ -1435,14 +1372,13 @@ namespace cryptonote
bool tx_memory_pool::have_tx_keyimg_as_spent(const crypto::key_image& key_im, const crypto::hash& txid) const
{
CRITICAL_REGION_LOCAL(m_transactions_lock);
- const auto found = m_spent_key_images.find(key_im);
- if (found != m_spent_key_images.end() && !found->second.empty())
+ const auto ki_range = m_spent_key_images.left.equal_range(key_im);
+ for (auto ki_it = ki_range.first; ki_it != ki_range.second; ++ki_it)
{
// If another tx is using the key image, always return as spent.
// See `insert_key_images`.
- if (1 < found->second.size() || *(found->second.cbegin()) != txid)
+ if (ki_it->second != txid || m_blockchain.txpool_tx_matches_category(txid, relay_category::legacy))
return true;
- return m_blockchain.txpool_tx_matches_category(txid, relay_category::legacy);
}
return false;
}
@@ -1482,27 +1418,23 @@ namespace cryptonote
return ret;
}
//---------------------------------------------------------------------------------
- bool tx_memory_pool::is_transaction_ready_to_go(txpool_tx_meta_t& txd, const crypto::hash &txid, const cryptonote::blobdata_ref& txblob, transaction &tx) const
+ bool tx_memory_pool::is_transaction_ready_to_go(txpool_tx_meta_t& txd, const crypto::hash &txid) const
{
- struct transaction_parser
+ // NOTE: needs blockchain lock to be held
+
+ auto lazy_tx = [this, &txid, tx = transaction(), parsed = false]() mutable
+ -> transaction&
{
- transaction_parser(const cryptonote::blobdata_ref &txblob, const crypto::hash &txid, transaction &tx): txblob(txblob), txid(txid), tx(tx), parsed(false) {}
- cryptonote::transaction &operator()()
+ if (!parsed)
{
- if (!parsed)
- {
- if (!parse_and_validate_tx_from_blob(txblob, tx))
- throw std::runtime_error("failed to parse transaction blob");
- tx.set_hash(txid);
- parsed = true;
- }
- return tx;
+ const cryptonote::blobdata txblob = this->m_blockchain.get_txpool_tx_blob(txid, relay_category::all);
+ if (!parse_and_validate_tx_from_blob(txblob, tx))
+ throw std::runtime_error("failed to parse transaction blob");
+ tx.set_hash(txid);
+ parsed = true;
}
- const cryptonote::blobdata_ref &txblob;
- const crypto::hash &txid;
- transaction &tx;
- bool parsed;
- } lazy_tx(txblob, txid, tx);
+ return tx;
+ };
const std::uint64_t top_block_height{m_blockchain.get_current_blockchain_height() - 1};
const crypto::hash top_block_hash{m_blockchain.get_block_id_by_height(top_block_height)};
@@ -1527,33 +1459,6 @@ namespace cryptonote
return true;
}
//---------------------------------------------------------------------------------
- bool tx_memory_pool::is_transaction_ready_to_go(txpool_tx_meta_t& txd, const crypto::hash &txid, const cryptonote::blobdata& txblob, transaction &tx) const
- {
- return is_transaction_ready_to_go(txd, txid, cryptonote::blobdata_ref{txblob.data(), txblob.size()}, tx);
- }
- //---------------------------------------------------------------------------------
- bool tx_memory_pool::have_key_images(const std::unordered_set<crypto::key_image>& k_images, const transaction_prefix& tx)
- {
- for(size_t i = 0; i!= tx.vin.size(); i++)
- {
- CHECKED_GET_SPECIFIC_VARIANT(tx.vin[i], const txin_to_key, itk, false);
- if(k_images.count(itk.k_image))
- return true;
- }
- return false;
- }
- //---------------------------------------------------------------------------------
- bool tx_memory_pool::append_key_images(std::unordered_set<crypto::key_image>& k_images, const transaction_prefix& tx)
- {
- for(size_t i = 0; i!= tx.vin.size(); i++)
- {
- CHECKED_GET_SPECIFIC_VARIANT(tx.vin[i], const txin_to_key, itk, false);
- auto i_res = k_images.insert(itk.k_image);
- CHECK_AND_ASSERT_MES(i_res.second, false, "internal error: key images pool cache - inserted duplicate image in set: " << itk.k_image);
- }
- return true;
- }
- //---------------------------------------------------------------------------------
void tx_memory_pool::mark_double_spend(const transaction &tx)
{
CRITICAL_REGION_LOCAL(m_transactions_lock);
@@ -1563,11 +1468,10 @@ namespace cryptonote
for(size_t i = 0; i!= tx.vin.size(); i++)
{
CHECKED_GET_SPECIFIC_VARIANT(tx.vin[i], const txin_to_key, itk, void());
- const key_images_container::const_iterator it = m_spent_key_images.find(itk.k_image);
- if (it != m_spent_key_images.end())
+ const auto ki_range = m_spent_key_images.left.equal_range(itk.k_image);
+ for (auto ki_it = ki_range.first; ki_it != ki_range.second; ++ki_it)
{
- for (const crypto::hash &txid: it->second)
- {
+ const crypto::hash &txid = ki_it->second;
txpool_tx_meta_t meta;
if (!m_blockchain.get_txpool_tx_meta(txid, meta))
{
@@ -1590,7 +1494,6 @@ namespace cryptonote
// continue, not fatal
}
}
- }
}
}
lock.commit();
@@ -1631,14 +1534,24 @@ namespace cryptonote
}
//---------------------------------------------------------------------------------
//TODO: investigate whether boolean return is appropriate
- bool tx_memory_pool::fill_block_template(block &bl, size_t median_weight, uint64_t already_generated_coins, size_t &total_weight, uint64_t &fee, uint64_t &expected_reward, uint8_t version, bool include_sensitive)
+ bool tx_memory_pool::get_block_template_backlog(const size_t median_weight,
+ const uint64_t already_generated_coins,
+ const uint8_t version/*AKA hf_version*/,
+ const float overpick,
+ const bool include_sensitive,
+ std::vector<tx_block_template_backlog_entry> &selected_backlog)
{
+ selected_backlog.clear();
+
+ CHECK_AND_ASSERT_MES(overpick >= 0, false, "Block template blocklog overpick cannot be negative");
+ CHECK_AND_ASSERT_MES(overpick <= 100, false, "Block template blocklog overpick should be less than 10000%");
+
CRITICAL_REGION_LOCAL(m_transactions_lock);
CRITICAL_REGION_LOCAL1(m_blockchain);
- uint64_t best_coinbase = 0, coinbase = 0;
- total_weight = 0;
- fee = 0;
+ uint64_t best_coinbase = 0;
+ uint64_t total_weight = 0;
+ uint64_t fee = 0;
//baseline empty block
if (!get_block_reward(median_weight, total_weight, already_generated_coins, best_coinbase, version))
@@ -1647,31 +1560,49 @@ namespace cryptonote
return false;
}
+ static_assert(MAX_HF_VERSION == 16, "Check below weights and limits");
+ const relay_category category = include_sensitive ? relay_category::legacy : relay_category::broadcasted;
+ const size_t max_total_weight_pre_v5 = (130 * median_weight) / 100;
+ const size_t max_total_weight_v5 = 2 * median_weight;
+ const size_t consensus_max_total_weight = version >= 5 ? max_total_weight_v5 : max_total_weight_pre_v5;
+ size_t max_total_weight = consensus_max_total_weight
+ + static_cast<size_t>(overpick * consensus_max_total_weight);
+ const uint64_t reference_tx_weight = DYNAMIC_FEE_REFERENCE_TRANSACTION_WEIGHT;
+ constexpr size_t min_tx_size_approx = 1200;
+
+ selected_backlog.reserve(std::min<size_t>(2 * median_weight / min_tx_size_approx, CRYPTONOTE_MAX_TX_PER_BLOCK));
- size_t max_total_weight_pre_v5 = (130 * median_weight) / 100 - CRYPTONOTE_COINBASE_BLOB_RESERVED_SIZE;
- size_t max_total_weight_v5 = 2 * median_weight - CRYPTONOTE_COINBASE_BLOB_RESERVED_SIZE;
- size_t max_total_weight = version >= 5 ? max_total_weight_v5 : max_total_weight_pre_v5;
std::unordered_set<crypto::key_image> k_images;
LOG_PRINT_L2("Filling block template, median weight " << median_weight << ", " << m_txs_by_fee_and_receive_time.size() << " txes in the pool");
LockedTXN lock(m_blockchain.get_db());
auto sorted_it = m_txs_by_fee_and_receive_time.begin();
+ bool tentative_break = false;
for (; sorted_it != m_txs_by_fee_and_receive_time.end(); ++sorted_it)
{
+ if (tentative_break)
+ {
+ if (selected_backlog.size() && total_weight + reference_tx_weight > max_total_weight)
+ break; // there is not much room left, and last tx breaches the limit, so next txs will probably too
+ tentative_break = false;
+ }
+
+ const crypto::hash &txid = sorted_it->get_right();
txpool_tx_meta_t meta;
- if (!m_blockchain.get_txpool_tx_meta(sorted_it->get_right(), meta))
+ if (!m_blockchain.get_txpool_tx_meta(txid, meta))
{
static bool warned = false;
if (!warned)
- MERROR(" failed to find tx meta: " << sorted_it->get_right() << " (will only print once)");
+ MERROR(" failed to find tx meta: " << txid << " (will only print once)");
warned = true;
continue;
}
- LOG_PRINT_L2("Considering " << sorted_it->get_right() << ", weight " << meta.weight << ", current block weight " << total_weight << "/" << max_total_weight << ", current coinbase " << print_money(best_coinbase) << ", relay method " << (unsigned)meta.get_relay_method());
+ LOG_PRINT_L2("Considering " << txid << ", weight " << meta.weight << ", current block weight "
+ << total_weight << "/" << max_total_weight << ", current coinbase "
+ << print_money(best_coinbase) << ", relay method " << (unsigned)meta.get_relay_method());
- const relay_category category = include_sensitive ? relay_category::legacy : relay_category::broadcasted;
if (!meta.matches(category) && !(m_mine_stem_txes && meta.get_relay_method() == relay_method::stem))
{
LOG_PRINT_L2(" tx relay method is " << (unsigned)meta.get_relay_method());
@@ -1683,55 +1614,53 @@ namespace cryptonote
continue;
}
- // Can not exceed maximum block weight
+ // Cannot exceed max weight limit we set (may be greater than the block weight limit, depending on `overpick`)
if (max_total_weight < total_weight + meta.weight)
{
LOG_PRINT_L2(" would exceed maximum block weight");
+ tentative_break = true;
continue;
}
- // start using the optimal filling algorithm from v5
- if (version >= 5)
+ uint64_t coinbase = 0;
{
- // If we're getting lower coinbase tx,
- // stop including more tx
+ // If we're getting lower coinbase tx, and not overpicking,
+ // stop including more tx. If overpicking, set `tentative_break`, which
+ // will trigger a re-calculation of `max_total_weight` next loop.
uint64_t block_reward;
if(!get_block_reward(median_weight, total_weight + meta.weight, already_generated_coins, block_reward, version))
{
- LOG_PRINT_L2(" would exceed maximum block weight");
- continue;
- }
- coinbase = block_reward + fee + meta.fee;
- if (coinbase < template_accept_threshold(best_coinbase))
- {
- LOG_PRINT_L2(" would decrease coinbase to " << print_money(coinbase));
- continue;
+ block_reward = 0;
+ tentative_break = true;
+ if (0 == overpick)
+ {
+ LOG_PRINT_L2(" would exceed maximum block weight");
+ continue;
+ }
}
- }
- else
- {
- // If we've exceeded the penalty free weight,
- // stop including more tx
- if (total_weight > median_weight)
+ else
{
- LOG_PRINT_L2(" would exceed median block weight");
- break;
+ coinbase = block_reward + fee + meta.fee;
+ if (coinbase < best_coinbase)
+ {
+ tentative_break = true;
+ if (0 == overpick)
+ {
+ LOG_PRINT_L2(" would decrease coinbase to " << print_money(coinbase));
+ continue;
+ }
+ }
}
}
- // "local" and "stem" txes are filtered above
- cryptonote::blobdata txblob = m_blockchain.get_txpool_tx_blob(sorted_it->get_right(), relay_category::all);
-
- cryptonote::transaction tx;
-
// Skip transactions that are not ready to be
// included into the blockchain or that are
// missing key images
const cryptonote::txpool_tx_meta_t original_meta = meta;
bool ready = false;
try
{
- ready = is_transaction_ready_to_go(meta, sorted_it->get_right(), txblob, tx);
+ ready = is_transaction_ready_to_go(meta, txid);
}
catch (const std::exception &e)
{
@@ -1741,37 +1670,59 @@ namespace cryptonote
if (memcmp(&original_meta, &meta, sizeof(meta)))
{
try
- {
- m_blockchain.update_txpool_tx(sorted_it->get_right(), meta);
- }
+ {
+ m_blockchain.update_txpool_tx(txid, meta);
+ }
catch (const std::exception &e)
- {
- MERROR("Failed to update tx meta: " << e.what());
- // continue, not fatal
- }
+ {
+ MERROR("Failed to update tx meta: " << e.what());
+ // continue, not fatal
+ }
}
if (!ready)
{
LOG_PRINT_L2(" not ready to go");
continue;
}
- if (have_key_images(k_images, tx))
+
+ // Test key images do not conflict w/ other txs' key images in the template
+ std::unordered_set<crypto::key_image> tx_k_images;
+ const auto tx_ki_range = m_spent_key_images.right.equal_range(txid);
+ bool has_ki_conflict = false;
+ for (auto ki_it = tx_ki_range.first; ki_it != tx_ki_range.second; ++ki_it)
+ {
+ const crypto::key_image &ki = ki_it->second;
+ if (k_images.count(ki))
+ {
+ has_ki_conflict = true;
+ break;
+ }
+ tx_k_images.insert(ki_it->second);
+ }
+
+ if (has_ki_conflict)
{
LOG_PRINT_L2(" key images already seen");
continue;
}
- bl.tx_hashes.push_back(sorted_it->get_right());
+ if (tentative_break && overpick > 0)
+ {
+ // If overpicking, set weight limit as overpick ratio against optimal, which is the lowest
+ // `total_weight` value when `tentative_break` is tripped
+ max_total_weight = std::min<size_t>(max_total_weight, total_weight * (1 + overpick));
+ }
+
+ selected_backlog.push_back(tx_block_template_backlog_entry{txid, meta.weight, meta.fee});
total_weight += meta.weight;
fee += meta.fee;
- best_coinbase = coinbase;
- append_key_images(k_images, tx);
+ best_coinbase = std::max(best_coinbase, coinbase);
+ k_images.merge(tx_k_images);
LOG_PRINT_L2(" added, new block weight " << total_weight << "/" << max_total_weight << ", coinbase " << print_money(best_coinbase));
}
lock.commit();
- expected_reward = best_coinbase;
- LOG_PRINT_L2("Block template filled with " << bl.tx_hashes.size() << " txes, weight "
+ LOG_PRINT_L2("Block template backlog filled with " << selected_backlog.size() << " txes, weight "
<< total_weight << "/" << max_total_weight << ", coinbase " << print_money(best_coinbase)
<< " (including " << print_money(fee) << " in fees)");
return true;
### src/cryptonote_core/tx_pool.h
@@ -1,4 +1,4 @@
-// Copyright (c) 2014-2024, The Monero Project
+// Copyright (c) 2014-2026, The Monero Project
//
// All rights reserved.
//
@@ -41,6 +41,7 @@
#include <boost/bimap.hpp>
#include <boost/bimap/set_of.hpp>
#include <boost/bimap/multiset_of.hpp>
+#include <boost/bimap/unordered_multiset_of.hpp>
#include "span.h"
#include "string_tools.h"
@@ -247,17 +248,38 @@ namespace cryptonote
/**
* @brief Chooses transactions for a block to include
*
- * @param bl return-by-reference the block to fill in with transactions
- * @param median_weight the current median block weight
+ * @param median_weight the current median block weight used for penalty calculations
* @param already_generated_coins the current total number of coins "minted"
- * @param total_weight return-by-reference the total weight of the new block
- * @param fee return-by-reference the total of fees from the included transactions
- * @param expected_reward return-by-reference the total reward awarded to the miner finding this block, including transaction fees
- * @param version hard fork version to use for consensus rules
- *
- * @return true
- */
- bool fill_block_template(block &bl, size_t median_weight, uint64_t already_generated_coins, size_t &total_weight, uint64_t &fee, uint64_t &expected_reward, uint8_t version, bool include_sensitive = true);
+ * @param hf_version the current hard fork version
+ * @param overpick if >0, the ratio over the optimal weight for which to limit total weight of `selected_backlog`
+ * @param include_sensitive return stempool (only if m_mine_stem_txes), anonymity-pool, and unrelayed txes
+ * @param[out] selected_backlog potential (TXID, weight, fee) tuples, in descending preference order
+ *
+ * Not all transactions in the pool will be returned for performance, consensus, and/or
+ * profitability reasons. If there are too many transactions in the pool, only the
+ * highest-paying / oldest transactions will be returned - but enough for the miner to create a
+ * full block. Transactions which do not pass validation rules for the next block are also
+ * filtered out. However, @warning, `selected_backlog` may also contain *too many entries*, even
+ * when `overpick_ratio` == 0, such that block weight limit is surpassed after including the
+ * coinbase transaction. In this case, it is a good idea to pop candidates from the back of the
+ * list until the weight is within the limit.
+ *
+ * The target limit is calculated as (1 + `overpick`) * optimal weight. So for example, an
+ * `overpick` value of 0.1 will try to fill `selected_backlog` so that it's total weight is 110%
+ * of the total weight if `overpick` were 0. When `overpick` > 0, entries in `selected_backlog`
+ * may not be net profitable to include in the block, even if they fit within the block.
+ *
+ * This function *may* do some upkeep on the mempool, including updating transaction
+ * verification metadata, and removing stale unverifiable transactions.
+ *
+ * @return true on success
+ */
+ bool get_block_template_backlog(size_t median_weight,
+ uint64_t already_generated_coins,
+ uint8_t hf_version,
+ float overpick,
+ bool include_sensitive,
+ std::vector<tx_block_template_backlog_entry> &selected_backlog);
/**
* @brief get a list of all transactions in the pool
@@ -286,19 +308,6 @@ namespace cryptonote
*/
void get_transaction_backlog(std::vector<tx_backlog_entry>& backlog, bool include_sensitive = false) const;
- /**
- * @brief get (hash, weight, fee) for transactions in the pool - the minimum required information to create a block template
- *
- * Not all transactions in the pool will be returned for performance reasons
- * If there are too many transactions in the pool, only the highest-paying transactions
- * will be returned - but enough for the miner to create a full block
- *
- * @param backlog return-by-reference that data
- * @param include_sensitive return stempool, anonymity-pool, and unrelayed txes
- *
- */
- void get_block_template_backlog(std::vector<tx_block_template_backlog_entry>& backlog, bool include_sensitive = false) const;
-
/**
* @brief get a summary statistics of all transaction hashes in the pool
*
@@ -553,38 +562,15 @@ namespace cryptonote
*/
bool remove_transaction_keyimages(const transaction_prefix& tx, const crypto::hash &txid);
- /**
- * @brief check if any of a transaction's spent key images are present in a given set
- *
- * @param kic the set of key images to check against
- * @param tx the transaction to check
- *
- * @return true if any key images present in the set, otherwise false
- */
- static bool have_key_images(const std::unordered_set<crypto::key_image>& kic, const transaction_prefix& tx);
-
- /**
- * @brief append the key images from a transaction to the given set
- *
- * @param kic the set of key images to append to
- * @param tx the transaction
- *
- * @return false if any append fails, otherwise true
- */
- static bool append_key_images(std::unordered_set<crypto::key_image>& kic, const transaction_prefix& tx);
-
/**
* @brief check if a transaction is a valid candidate for inclusion in a block
*
* @param txd the transaction to check (and info about it)
* @param txid the txid of the transaction to check
- * @param txblob the transaction blob to check
- * @param tx the parsed transaction, if successful
*
* @return true if the transaction is good to go, otherwise false
*/
- bool is_transaction_ready_to_go(txpool_tx_meta_t& txd, const crypto::hash &txid, const cryptonote::blobdata_ref &txblob, transaction&tx) const;
- bool is_transaction_ready_to_go(txpool_tx_meta_t& txd, const crypto::hash &txid, const cryptonote::blobdata &txblob, transaction&tx) const;
+ bool is_transaction_ready_to_go(txpool_tx_meta_t& txd, const crypto::hash &txid) const;
/**
* @brief mark all transactions double spending the one passed
@@ -604,14 +590,17 @@ namespace cryptonote
//TODO: confirm the below comments and investigate whether or not this
// is the desired behavior
- //! map key images to transactions which spent them
+ //! map key images to transactions which spent them, and vice versa
/*! this seems odd, but it seems that multiple transactions can exist
* in the pool which both have the same spent key. This would happen
* in the event of a reorg where someone creates a new/different
* transaction on the assumption that the original will not be in a
* block again.
*/
- typedef std::unordered_map<crypto::key_image, std::unordered_set<crypto::hash>> key_images_container;
+ typedef boost::bimap<
+ boost::bimaps::unordered_multiset_of<crypto::key_image>,
+ boost::bimaps::unordered_multiset_of<crypto::hash>
+ > key_images_container;
#if defined(DEBUG_CREATE_BLOCK_TEMPLATE)
public:
### tests/functional_tests/txpool.py
@@ -35,6 +35,8 @@
from framework.wallet import Wallet
from framework.zmq import Zmq
+DEFAULT_TXPOOL_MAX_WEIGHT = 648000000 # See cryptonote_config.h
+
class TransferTest():
def run_test(self):
self.reset()
@@ -170,6 +172,18 @@ def check_txpool(self):
min_bytes = min(min_bytes, x.blob_size)
max_bytes = max(max_bytes, x.blob_size)
+ print('Checking miner data: when pool size less than median block weight')
+ assert total_bytes < DEFAULT_TXPOOL_MAX_WEIGHT, total_bytes # test needs updating for this case
+ pool_res = res
+ miner_data_res = daemon.get_miner_data()
+ assert total_bytes < miner_data_res.median_weight, total_bytes # test needs updating for this case
+ pool_txids = set(x.id_hash for x in pool_res.transactions)
+ miner_data_txids = set(x.id for x in miner_data_res.tx_backlog)
+ assert len(pool_txids) == len(pool_res.transactions) # dups
+ assert len(miner_data_txids) == len(miner_data_res.tx_backlog) # dups
+ assert pool_txids == miner_data_txids
+
+
print('Checking all txs received via zmq')
for i in range(len(txes.keys())):
zmq_event = zmq.recv(zmq_topic)Why this scored 59/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.