What changed, and why it matters
This is a large feature merge for Monero that adds database support for FCMP++ (Full Chain Membership Proof++) curve trees. It changes how outputs are indexed, adds new database tables, and includes a migration from database version 5 to 7. The changes are primarily structural and preparatory for a future privacy upgrade, not a finished activated protocol change. The code is gated behind a compile-time flag ENABLE_FCMP_INTEGRATION that is off by default.
Treat this as a significant but gated infrastructure change. Reviewers should focus on: (1) correctness of the DB migration and rollback behavior, (2) endianness handling in locked_outputs/timelocked_outputs DUPSORT keys, (3) whether the new txout_to_carrot_v1 variant tag reuse (0x1) for previously txout_to_scripthash could cause deserialization issues with old data, (4) validation that tree growth/trim invariants hold during reorgs, and (5) that ENABLE_FCMP_INTEGRATION remains off in release builds until the full protocol is ready. No immediate emergency patch is indicated.
Security signals we found
Large database schema change with new tables and DB version bump (5 -> 7)
Compile-time feature flag ENABLE_FCMP_INTEGRATION keeps new code disabled by default
New migration path migrate_5_7 with resumable batching and temporary cache table
Replacement of txout_to_scripthash with txout_to_carrot_v1 changes output variant serialization tags
Addition of custom unlock-time handling with deprecation block and mainnet-specific constants
New curve tree grow/trim logic with CHECK_AND_ASSERT_THROW_MES validation checks
Locked outputs table uses little-endian uint64 comparison due to binary serializer byte order
Evidence from the diff
The commit integrates FCMP++ tree building into the blockchain DB layer. Key changes: (1) adds ENABLE_FCMP_INTEGRATION CMake option and CI build variant; (2) renames output_id to unified_id across output tables and renames get_output_tx_and_index_from_global to unified; (3) adds new LMDB tables (locked_outputs, timelocked_outputs, leaves, layers, tree_edges, tree_meta) and associated virtual DB methods; (4) implements tree growth/trim logic in BlockchainDB and BlockchainLMDB; (5) adds a migration from DB version 5 to 7 that populates locked outputs and grows the curve tree for existing chain data; (6) introduces txout_to_carrot_v1 output type replacing txout_to_scripthash; (7) adds helper functions for computing last-locked-block indices and transparent amount commitments. The feature is not active in production builds unless explicitly compiled with ENABLE_FCMP_INTEGRATION=ON.
Changed components
src/blockchain_db/blockchain_db.cpp/hsrc/blockchain_db/lmdb/db_lmdb.cpp/hsrc/blockchain_db/testdb.hsrc/fcmp_pp/curve_trees.cpp/hsrc/fcmp_pp/fcmp_pp_rust/*src/fcmp_pp/fcmp_pp_types.hsrc/fcmp_pp/fcmp_pp_serialization.hsrc/fcmp_pp/tower_cycle.cpp/hsrc/cryptonote_basic/cryptonote_basic.hsrc/cryptonote_basic/cryptonote_format_utils.cpp/hsrc/cryptonote_core/blockchain.cppsrc/cryptonote_core/tx_verification_utils.cpp/hsrc/serialization/*CMakeLists.txt.github/workflows/build.ymlInspect captured patch +2494 / −118
### .github/workflows/build.yml
@@ -15,6 +15,7 @@ env:
REMOVE_BUNDLED_PACKAGES : sudo rm -rf /usr/local
# ARCH="default" (not "native") ensures, that a different execution host can execute binaries compiled elsewhere.
BUILD_DEFAULT: 'cmake -S . -B build -D ARCH="default" -D BUILD_TESTS=ON -D BUILD_GUI_DEPS=ON -D ENABLE_FUZZ_TEST=ON -D CMAKE_BUILD_TYPE=Release && cmake --build build --target all'
+ BUILD_DEFAULT_WITH_FCMP_INTEGRATION_ENABLED: 'cmake -S . -B build -D ARCH="default" -D BUILD_TESTS=ON -D BUILD_GUI_DEPS=ON -D ENABLE_FUZZ_TEST=ON -D CMAKE_BUILD_TYPE=Release -D ENABLE_FCMP_INTEGRATION=ON && cmake --build build --target all'
APT_INSTALL_LINUX: 'apt -y install build-essential cmake libboost-all-dev libunbound-dev graphviz doxygen libunwind8-dev pkg-config libssl-dev libzmq3-dev libsodium-dev libhidapi-dev libusb-1.0-0-dev libprotobuf-dev protobuf-compiler ccache curl git'
APT_SET_CONF: |
tee -a /etc/apt/apt.conf.d/80-custom << EOF
@@ -245,7 +246,7 @@ jobs:
CMAKE_BUILD_PARALLEL_LEVEL: ${{env.MAKE_JOB_COUNT}}
run: |
${{env.CCACHE_SETTINGS}}
- ${{env.BUILD_DEFAULT}}
+ ${{env.BUILD_DEFAULT_WITH_FCMP_INTEGRATION_ENABLED}}
- name: run tests except core_tests
env:
DNS_PUBLIC: tcp
@@ -256,7 +257,7 @@ jobs:
CFLAGS: -DMONERO_CRYPTO_SLOW_HASH_ITER=20
CMAKE_BUILD_PARALLEL_LEVEL: ${{env.MAKE_JOB_COUNT}}
run: |
- cmake -S . -B build --fresh -D ARCH="default" -D BUILD_TESTS=ON
+ cmake -S . -B build --fresh -D ARCH="default" -D BUILD_TESTS=ON -D ENABLE_FCMP_INTEGRATION=ON
cmake --build build --target core_tests
- name: run core_tests
run: ctest --test-dir build --output-on-failure -R core_tests
### CMakeLists.txt
@@ -1037,6 +1037,11 @@ find_package(Sodium REQUIRED)
find_package(PkgConfig REQUIRED)
pkg_check_modules(libzmq REQUIRED IMPORTED_TARGET libzmq)
+option(ENABLE_FCMP_INTEGRATION "Enable the FCMP++ integration" OFF)
+if (ENABLE_FCMP_INTEGRATION)
+ add_definitions(-DENABLE_FCMP_INTEGRATION)
+endif()
+
include(external/supercop/functions.cmake) # place after setting flags and before src directory inclusion
add_subdirectory(contrib)
add_subdirectory(src)
### src/blockchain_db/CMakeLists.txt
@@ -46,6 +46,7 @@ target_link_libraries(blockchain_db
common
cncrypto
cryptonote_basic
+ fcmp_pp
ringct_basic
${LMDB_LIBRARY}
${Boost_FILESYSTEM_LIBRARY}
### src/blockchain_db/blockchain_db.cpp
@@ -31,6 +31,7 @@
#include "string_tools.h"
#include "blockchain_db.h"
#include "cryptonote_basic/cryptonote_format_utils.h"
+#include "fcmp_pp/curve_trees.h"
#include "profile_tools.h"
#include "ringct/rctOps.h"
@@ -172,7 +173,7 @@ void BlockchainDB::init_options(boost::program_options::options_description& des
command_line::add_arg(desc, arg_db_salvage);
}
-void BlockchainDB::add_transaction(const crypto::hash& blk_hash, const transaction& tx, const epee::span<const std::uint8_t> blob, const crypto::hash* tx_hash_ptr, const crypto::hash* tx_prunable_hash_ptr)
+void BlockchainDB::add_transaction(const crypto::hash& blk_hash, const transaction& tx, epee::span<const std::uint8_t> blob, const std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments, const crypto::hash* tx_hash_ptr, const crypto::hash* tx_prunable_hash_ptr)
{
bool miner_tx = false;
crypto::hash tx_hash, tx_prunable_hash;
@@ -226,10 +227,12 @@ void BlockchainDB::add_transaction(const crypto::hash& blk_hash, const transacti
if (miner_tx && tx.version == 2)
{
cryptonote::tx_out vout = tx.vout[i];
- rct::key commitment = rct::zeroCommitVartime(vout.amount);
+ const auto commitment_it = transparent_amount_commitments.find(vout.amount);
+ if (commitment_it == transparent_amount_commitments.end())
+ throw std::runtime_error("Failed to get miner tx commitment, aborting");
vout.amount = 0;
amount_output_indices[i] = add_output(tx_hash, vout, i, tx.unlock_time,
- &commitment);
+ &commitment_it->second);
}
else
{
@@ -246,6 +249,7 @@ uint64_t BlockchainDB::add_block( const std::pair<block, blobdata>& blck
, const difficulty_type& cumulative_difficulty
, const uint64_t& coins_generated
, const std::vector<std::pair<transaction, blobdata>>& txs
+ , const std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments
)
{
const block &blk = blck.first;
@@ -259,23 +263,24 @@ uint64_t BlockchainDB::add_block( const std::pair<block, blobdata>& blck
TIME_MEASURE_FINISH(time1);
time_blk_hash += time1;
- uint64_t prev_height = height();
+ const uint64_t blk_idx = height();
+ const uint64_t first_unified_id = total_outputs(); // Call this before adding this block's outputs
// call out to add the transactions
time1 = epee::misc_utils::get_tick_count();
uint64_t num_rct_outs = 0;
blobdata miner_bd = tx_to_blob(blk.miner_tx);
- add_transaction(blk_hash, blk.miner_tx, epee::strspan<std::uint8_t>(miner_bd));
+ add_transaction(blk_hash, blk.miner_tx, epee::strspan<std::uint8_t>(miner_bd), transparent_amount_commitments);
if (blk.miner_tx.version == 2)
num_rct_outs += blk.miner_tx.vout.size();
int tx_i = 0;
crypto::hash tx_hash = crypto::null_hash;
for (const std::pair<transaction, blobdata>& tx : txs)
{
tx_hash = blk.tx_hashes[tx_i];
- add_transaction(blk_hash, tx.first, epee::strspan<std::uint8_t>(tx.second), &tx_hash);
+ add_transaction(blk_hash, tx.first, epee::strspan<std::uint8_t>(tx.second), transparent_amount_commitments, &tx_hash);
for (const auto &vout: tx.first.vout)
{
if (vout.amount == 0)
@@ -292,11 +297,219 @@ uint64_t BlockchainDB::add_block( const std::pair<block, blobdata>& blck
TIME_MEASURE_FINISH(time1);
time_add_block1 += time1;
- m_hardfork->add(blk, prev_height);
+#ifdef ENABLE_FCMP_INTEGRATION
+ // call out to handle growing the FCMP tree
+ time1 = epee::misc_utils::get_tick_count();
+ handle_fcmp_tree(blk_idx, first_unified_id, blk.miner_tx, txs, transparent_amount_commitments);
+ TIME_MEASURE_FINISH(time1);
+ time_grow_tree += time1;
+#endif
+
+ m_hardfork->add(blk, blk_idx);
++num_calls;
- return prev_height;
+ return blk_idx;
+}
+
+void BlockchainDB::handle_fcmp_tree(const uint64_t block_idx,
+ const uint64_t first_unified_id,
+ const transaction &miner_tx,
+ const std::vector<std::pair<transaction, blobdata>> &txs,
+ const std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments)
+{
+ // Collect outs by last locked block to add to the db
+ OutsByLastLockedBlockMeta new_locked_outs = cryptonote::get_outs_by_last_locked_block(
+ miner_tx,
+ txs,
+ transparent_amount_commitments,
+ first_unified_id,
+ block_idx
+ );
+
+ // Get the outputs with default last locked block
+ const uint64_t default_last_locked_block = cryptonote::get_default_last_locked_block_index(block_idx);
+ auto new_default_locked_outs_it = new_locked_outs.outs_by_last_locked_block.find(default_last_locked_block);
+ const auto new_default_locked_outs = new_default_locked_outs_it != new_locked_outs.outs_by_last_locked_block.end()
+ ? std::move(new_default_locked_outs_it->second)
+ : std::vector<fcmp_pp::UnifiedOutput>{};
+
+ // Insert the new locked outputs into the db, excluding outputs created in
+ // this block with default last locked block. Outputs with default last locked
+ // block will be added to the tree immediately below. Outputs with last
+ // locked block higher than the default will be added to the locked outputs
+ // tables, staged for insertion to the tree later.
+ new_locked_outs.outs_by_last_locked_block.erase(default_last_locked_block);
+ this->add_locked_outs(new_locked_outs.outs_by_last_locked_block, new_locked_outs.timelocked_outputs);
+
+ // Assume we just added block n. The soonest that outputs from block n can be
+ // included in the chain is in block n + CRYPTNOTE_DEFAULT_SPENDABLE_AGE. So
+ // we grow the tree with these outputs (and any others with last locked block
+ // n + CRYPTNOTE_DEFAULT_SPENDABLE_AGE - 1). We then expect this tree root
+ // be included in block header n+1. This way miners will build on top of the
+ // tree root usable in FCMP++'s in a future block. After block
+ // n + (CRYPTNOTE_DEFAULT_SPENDABLE_AGE - 1) is added to the chain, SPV
+ // clients syncing just block headers will have a solid assurance that the
+ // root usable to construct FCMP++ proofs is the correct root, since it will
+ // have 9 blocks of PoW on top of it.
+ // To be clear, block header n+1 includes the tree root usable to spend
+ // outputs with last locked block n + CRYPTNOTE_DEFAULT_SPENDABLE_AGE - 1.
+ static_assert(CRYPTONOTE_DEFAULT_TX_SPENDABLE_AGE > 0, "Expect a non-0 spendable age");
+ this->advance_tree(block_idx, new_default_locked_outs);
+}
+
+void BlockchainDB::advance_tree(const uint64_t blk_idx, const std::vector<fcmp_pp::UnifiedOutput> &known_new_outputs)
+{
+ LOG_PRINT_L3("BlockchainDB::" << __func__);
+
+ // Get the earliest possible last locked block of outputs created in blk_idx
+ const uint64_t earliest_last_locked_block = cryptonote::get_default_last_locked_block_index(blk_idx);
+
+ // If we're advancing the genesis block, make sure to initialize the tree
+ if (blk_idx == 0)
+ {
+ // Expected: tree meta table is currently empty
+
+ // We grow the first blocks with empty outputs, since no outputs in this range should be spendable yet
+ for (uint64_t new_blk_idx = blk_idx; new_blk_idx < earliest_last_locked_block; ++new_blk_idx)
+ {
+ this->grow_tree(new_blk_idx, {});
+ }
+ }
+ // Expected: earliest_last_locked_block == last block idx + 1 in tree meta
+
+ // Now we can advance the tree 1 block
+ auto unlocked_outputs = this->get_outs_at_last_locked_block_idx(earliest_last_locked_block);
+
+ // Include known new outputs if provided
+ unlocked_outputs.insert(unlocked_outputs.end(), known_new_outputs.begin(), known_new_outputs.end());
+
+ // Grow the tree with outputs that are spendable once the earliest_last_locked_block is in the chain
+ this->grow_tree(earliest_last_locked_block, std::move(unlocked_outputs));
+
+ // Now that we've used the unlocked leaves to grow the tree, we delete them from the locked outputs table
+ this->del_locked_outs_at_block_idx(earliest_last_locked_block);
+}
+
+void BlockchainDB::grow_tree(const uint64_t blk_idx, std::vector<fcmp_pp::UnifiedOutput> &&new_outputs)
+{
+ LOG_PRINT_L3("BlockchainDB::" << __func__);
+
+ MDEBUG("Growing tree usable once block " << blk_idx << " is in the chain");
+
+ // Get the number of leaf tuples that exist in the current tree
+ const uint64_t old_n_leaf_tuples = this->get_n_leaf_tuples();
+
+ if (blk_idx == 0)
+ CHECK_AND_ASSERT_THROW_MES(old_n_leaf_tuples == 0, "Tree is not empty at blk idx 0");
+
+ // Get the prev block's tree edge (i.e. the current tree edge before growing)
+ std::vector<crypto::ec_point> prev_tree_edge;
+ uint64_t prev_blk_idx = 0;
+ if (blk_idx > 0)
+ {
+ prev_blk_idx = blk_idx - 1;
+
+ // Make sure tree tip lines up to expected block
+ const uint64_t tree_block_idx = this->get_tree_block_idx();
+
+ CHECK_AND_ASSERT_THROW_MES(tree_block_idx == prev_blk_idx,
+ "Unexpected tree block idx mismatch to prev block (" + std::to_string(tree_block_idx) + " vs " + std::to_string(prev_blk_idx) + ")");
+
+ prev_tree_edge = this->get_tree_edge(prev_blk_idx);
+ }
+
+ // We re-save the prev tree edge at this next block if the tree doesn't grow
+ const auto save_prev_tree_edge = [&, this]() { this->save_tree_meta(blk_idx, old_n_leaf_tuples, prev_tree_edge); };
+ if (new_outputs.empty())
+ {
+ save_prev_tree_edge();
+ return;
+ }
+
+ // Set the tree's existing last hashes from the existing edge
+ const auto last_hashes = m_curve_trees->tree_edge_to_last_hashes(prev_tree_edge);
+
+ // Use the number of leaf tuples and the existing last hashes to get a struct we can use to extend the tree
+ auto tree_extension = m_curve_trees->get_tree_extension(old_n_leaf_tuples, last_hashes, {std::move(new_outputs)});
+ if (tree_extension.leaves.tuples.empty())
+ {
+ save_prev_tree_edge();
+ return;
+ }
+
+ const auto compressed_tree_extension = m_curve_trees->compress_tree_extension(std::move(tree_extension));
+ const auto tree_edge = this->grow_with_tree_extension(compressed_tree_extension);
+
+ const uint64_t new_n_leaf_tuples = compressed_tree_extension.leaves.tuples.size() + old_n_leaf_tuples;
+ this->save_tree_meta(blk_idx, new_n_leaf_tuples, tree_edge);
+}
+
+void BlockchainDB::trim_block()
+{
+ LOG_PRINT_L3("BlockchainDB::" << __func__);
+
+ const uint64_t n_blocks = this->height();
+ if (n_blocks == 0)
+ return;
+
+ const uint64_t removing_block_idx = n_blocks - 1;
+
+ // Get the earliest possible last locked block of outputs created in removing_block_idx
+ const uint64_t default_last_locked_block = cryptonote::get_default_last_locked_block_index(removing_block_idx);
+ const uint64_t tree_block_idx = this->get_tree_block_idx();
+
+ CHECK_AND_ASSERT_THROW_MES(tree_block_idx > 0, "tree block idx must be >0");
+ CHECK_AND_ASSERT_THROW_MES(tree_block_idx == default_last_locked_block,
+ "Unexpected tree block idx mismatch (" + std::to_string(tree_block_idx) + " vs " + std::to_string(default_last_locked_block) + ")");
+
+ const uint64_t prev_tree_block_idx = tree_block_idx - 1;
+
+ MDEBUG("Trimming tree to block " << prev_tree_block_idx << " (removing block " << removing_block_idx << ")");
+
+ // Read n leaf tuples from the prev tree block to see how how many leaves
+ // should remain in the tree after trimming a block from the tree.
+ const uint64_t new_n_leaf_tuples = this->get_block_n_leaf_tuples(prev_tree_block_idx);
+
+ // Trim the tree to the new n leaf tuples
+ this->trim_tree(new_n_leaf_tuples, tree_block_idx);
+
+ // Remove block from tree meta
+ this->del_tree_meta(tree_block_idx);
+
+ // If we're removing the genesis block, we need to reverse the initially added in advance_tree.
+ // Since we know they're all empty, we can just delete the tree meta.
+ if (removing_block_idx == 0)
+ {
+ for (uint64_t rm_blk_idx = 0; rm_blk_idx < default_last_locked_block; ++rm_blk_idx)
+ {
+ this->del_tree_meta(rm_blk_idx);
+ }
+ }
+}
+
+void BlockchainDB::trim_tree(const uint64_t new_n_leaf_tuples, const uint64_t trim_block_idx)
+{
+ LOG_PRINT_L3("BlockchainDB::" << __func__);
+
+ const uint64_t old_n_leaf_tuples = this->trim_leaves(new_n_leaf_tuples, trim_block_idx);
+
+ // If nothing to trim, return
+ if (old_n_leaf_tuples == new_n_leaf_tuples)
+ return;
+
+ if (new_n_leaf_tuples == 0)
+ {
+ // Empty the tree
+ this->trim_layers(new_n_leaf_tuples, {}/*new_n_elems_per_layer*/, {}/*new_tree_edge*/, 0/*new_root_layer_idx*/);
+ return;
+ }
+
+ // Trim the expected layers
+ const auto new_n_elems_per_layer = m_curve_trees->n_elems_per_layer(new_n_leaf_tuples);
+ const auto new_tree_edge = this->get_tree_edge(trim_block_idx - 1);
+ const uint64_t new_root_layer_idx = m_curve_trees->n_layers(new_n_leaf_tuples) - 1;
+ this->trim_layers(new_n_leaf_tuples, new_n_elems_per_layer, new_tree_edge, new_root_layer_idx);
}
void BlockchainDB::set_hard_fork(HardFork* hf)
@@ -421,6 +634,7 @@ void BlockchainDB::reset_stats()
time_tx_exists = 0;
time_add_block1 = 0;
time_add_transaction = 0;
+ time_grow_tree = 0;
time_commit1 = 0;
}
@@ -439,6 +653,8 @@ void BlockchainDB::show_stats()
<< ENDL
<< "time_add_transaction: " << time_add_transaction << "ms"
<< ENDL
+ << "time_grow_tree: " << time_grow_tree << "ms"
+ << ENDL
<< "time_commit1: " << time_commit1 << "ms"
<< ENDL
<< "*********************************"
### src/blockchain_db/blockchain_db.h
@@ -40,6 +40,8 @@
#include "cryptonote_basic/difficulty.h"
#include "cryptonote_basic/hardfork.h"
#include "cryptonote_protocol/enums.h"
+#include "fcmp_pp/curve_trees.h"
+#include "fcmp_pp/fcmp_pp_types.h"
/** \file
* Cryptonote Blockchain Database Interface
@@ -537,6 +539,113 @@ class BlockchainDB
*/
virtual void remove_spent_key(const crypto::key_image& k_image) = 0;
+ /**
+ * @brief grow the tree with provided compressed tree extension
+ *
+ * @param tree_extension tree extension containing compressed points
+ *
+ * @return the new tree edge
+ */
+ virtual std::vector<crypto::ec_point> grow_with_tree_extension(const fcmp_pp::CompressedTreeExtension &tree_extension) = 0;
+
+ /**
+ * @brief trim the leaves from the tree to the new number of leaf tuples
+ *
+ * @param new_n_leaf_tuples the expected number of leaf tuples in the tree after trimming
+ * @param trim_block_idx the block idx being trimmed
+ *
+ * @return the old number of leaf tuples
+ */
+ virtual uint64_t trim_leaves(const uint64_t new_n_leaf_tuples, const uint64_t trim_block_idx) = 0;
+
+ /**
+ * @brief trim the layers from the tree using given parameters
+ *
+ * @param new_n_leaf_tuples the expected number of leaf tuples in the tree after trimming
+ * @param new_n_elems_per_layer the expected number of elemenets in every layer of the tree after trimming
+ * @param new_tree_edge the expected new tree edge in the tree after trimming
+ * @param new_root_layer_idx the expected layer idx of the new root after trimming
+ */
+ virtual void trim_layers(const uint64_t new_n_leaf_tuples,
+ const std::vector<uint64_t> &new_n_elems_per_layer,
+ const std::vector<crypto::ec_point> &new_tree_edge,
+ const uint64_t new_root_layer_idx) = 0;
+
+ /**
+ * @brief add outs to locked outputs tables, staging them for insertion to the tree
+ *
+ * If any of this cannot be done, the subclass should throw the corresponding
+ * subclass of DB_EXCEPTION
+ *
+ * @param outs_by_last_locked_block outs grouped by last locked block
+ * @param timelocked_outputs custom timelocked outputs
+ */
+ virtual void add_locked_outs(const fcmp_pp::OutsByLastLockedBlock& outs_by_last_locked_block, const std::unordered_map<uint64_t/*unified_id*/, uint64_t/*last locked block_id*/>& timelocked_outputs) = 0;
+
+ /**
+ * @brief get the outputs that are staged for insertion to the tree with last locked block provided
+ * Note: outputs are deleted from this table after they are inserted into the tree. Default locked
+ * outputs may or may not be included in this table too.
+ *
+ * @param block_id the block idx
+ */
+ virtual std::vector<fcmp_pp::UnifiedOutput> get_outs_at_last_locked_block_idx(uint64_t block_id) const = 0;
+
+ /**
+ * @brief delete the locked outputs that were staged for insertion to the tree for a given last locked block
+ *
+ * @param block_id the outputs' last locked block idx
+ */
+ virtual void del_locked_outs_at_block_idx(uint64_t block_idx) = 0;
+
+ /**
+ * @brief get the total number of leaf tuples in the tree
+ *
+ * @return the total number of leaf tuples in the tree
+ */
+ virtual uint64_t get_n_leaf_tuples() const = 0;
+
+ /**
+ * @brief get the total number of leaf tuples in the tree when the given block idx was tip of the chain
+ *
+ * @param block_idx assumed tip of the chain
+ *
+ * @return total number of leaf tuples in the tree when given block idx was chain tip
+ */
+ virtual uint64_t get_block_n_leaf_tuples(const uint64_t block_idx) const = 0;
+
+ /**
+ * @brief get the block idx corresponding to the last block used to update the tree
+ *
+ * @return the last block used to update the tree
+ */
+ virtual uint64_t get_tree_block_idx() const = 0;
+
+ /**
+ * @brief get the most recently updated edge of the tree (the last element of every layer)
+ * when the provided block is the tip of the chain
+ *
+ * @param block_id the block idx
+ *
+ * @return the last element of every layer
+ */
+ virtual std::vector<crypto::ec_point> get_tree_edge(uint64_t block_id) const = 0;
+
+ /**
+ * @brief save the tree's metadata at the time the provided block is the tip of the chain
+ *
+ * @param block_idx the provided block idx (should be the tip)
+ * @param n_leaf_tuples the number of leaf tuples in the tree
+ * @param tree_edge the last element of every layer in the tree
+ */
+ virtual void save_tree_meta(const uint64_t block_idx, const uint64_t n_leaf_tuples, const std::vector<crypto::ec_point> &tree_edge) = 0;
+
+ /**
+ * @brief delete the tree's metadata saved for a corresponding tip block idx
+ *
+ * @param block_id the block idx
+ */
+ virtual void del_tree_meta(const uint64_t block_idx) = 0;
/*********************************************************************
* private concrete members
@@ -551,10 +660,43 @@ class BlockchainDB
*/
void remove_transaction(const crypto::hash& tx_hash);
+ /**
+ * @brief handle the FCMP++ curve tree as the provided block_idx enters the chain. This handles
+ * saving staged locked outputs to the db, and advancing the FCMP++ tree.
+ *
+ * @param block_idx the block being entering the chain
+ * @param first_unified_id the first unified id of the first new default locked output from the block entering the chain
+ * @param miner_tx the block's miner transaction
+ * @param txs the block's non-miner transactions
+ * @param transparent_amount_commitments pre-calculated transparent amount commitments
+ */
+ void handle_fcmp_tree(const uint64_t block_idx,
+ const uint64_t first_unified_id,
+ const transaction &miner_tx,
+ const std::vector<std::pair<transaction, blobdata>> &txs,
+ const std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments);
+
+ /**
+ * @brief use the provided outputs to grow the FCMP++ curve tree and save all relevant data in the db
+ *
+ * @param block_idx the block being entering the chain
+ * @param new_outputs all new outputs to grow the tree with
+ */
+ void grow_tree(const uint64_t block_idx, std::vector<fcmp_pp::UnifiedOutput> &&new_outputs);
+
+ /**
+ * @brief trim the tree to the new number of leaf tuples
+ *
+ * @param new_n_leaf_tuples the expected number of leaf tuples in the tree after trimming
+ * @param trim_block_idx the block idx being trimmed
+ */
+ void trim_tree(const uint64_t new_n_leaf_tuples, const uint64_t trim_block_idx);
+
uint64_t num_calls = 0; //!< a performance metric
uint64_t time_blk_hash = 0; //!< a performance metric
uint64_t time_add_block1 = 0; //!< a performance metric
uint64_t time_add_transaction = 0; //!< a performance metric
+ uint64_t time_grow_tree = 0; //!< a performance metric
protected:
@@ -568,23 +710,40 @@ class BlockchainDB
* @param blk_hash hash of the block which has the transaction
* @param tx the transaction to add
* @param blob for `tx`
+ * @param transparent_amount_commitments pre-calculated transparent amount commitments
* @param tx_hash_ptr the hash of the transaction, if already calculated
* @param tx_prunable_hash_ptr the hash of the prunable part of the transaction, if already calculated
*/
- void add_transaction(const crypto::hash& blk_hash, const transaction& tx, epee::span<const std::uint8_t> blob, const crypto::hash* tx_hash_ptr = NULL, const crypto::hash* tx_prunable_hash_ptr = NULL);
+ void add_transaction(const crypto::hash& blk_hash, const transaction& tx, epee::span<const std::uint8_t> blob, const std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments, const crypto::hash* tx_hash_ptr = NULL, const crypto::hash* tx_prunable_hash_ptr = NULL);
+
+ /**
+ * @brief advance the FCMP++ curve tree as the provided block_idx enters the chain. This handles
+ * reading the db for any outputs staged for insertion not already provided.
+ *
+ * @param block_idx the block being entering the chain
+ * @param known_new_outputs the default locked new outputs to grow the tree with
+ */
+ void advance_tree(const uint64_t block_idx, const std::vector<fcmp_pp::UnifiedOutput> &known_new_outputs);
+
+ /**
+ * @brief trim a block's worth of data from the tree, placing locked outputs back in the db
+ */
+ void trim_block();
mutable uint64_t time_tx_exists = 0; //!< a performance metric
uint64_t time_commit1 = 0; //!< a performance metric
bool m_auto_remove_logs = true; //!< whether or not to automatically remove old logs
HardFork* m_hardfork;
+ std::shared_ptr<fcmp_pp::curve_trees::CurveTreesV1> m_curve_trees;
+
public:
/**
* @brief An empty constructor.
*/
- BlockchainDB(): m_hardfork(NULL), m_open(false) { }
+ BlockchainDB(): m_hardfork(NULL), m_open(false), m_curve_trees() { }
/**
* @brief An empty destructor.
@@ -815,6 +974,7 @@ class BlockchainDB
* @param cumulative_difficulty the accumulated difficulty after this block
* @param coins_generated the number of coins generated total after this block
* @param txs the transactions in the block
+ * @param transparent_amount_commitments pre-calculated transparent amount commitments
*
* @return the height of the chain post-addition
*/
@@ -824,6 +984,7 @@ class BlockchainDB
, const difficulty_type& cumulative_difficulty
, const uint64_t& coins_generated
, const std::vector<std::pair<transaction, blobdata>>& txs
+ , const std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments
);
/**
@@ -1376,6 +1537,15 @@ class BlockchainDB
*/
virtual uint64_t get_tx_block_height(const crypto::hash& h) const = 0;
+ /**
+ * @brief fetches the number of outputs in the chain
+ *
+ * The subclass should return a count of outputs, or zero if there are none.
+ *
+ * @return the total number of outputs in the chain (of all amounts)
+ */
+ virtual uint64_t total_outputs() const = 0;
+
/**
* @brief fetches the number of outputs of a given amount
*
@@ -1420,13 +1590,13 @@ class BlockchainDB
* @brief gets an output's tx hash and index
*
* The subclass should return the hash of the transaction which created the
- * output with the global index given, as well as its index in that transaction.
+ * output with the unified index given, as well as its index in that transaction.
*
- * @param index an output's global index
+ * @param index an output's unified index
*
* @return the tx hash and output index
*/
- virtual tx_out_index get_output_tx_and_index_from_global(const uint64_t& index) const = 0;
+ virtual tx_out_index get_output_tx_and_index_from_unified(const uint64_t& index) const = 0;
/**
* @brief gets an output's tx hash and index
@@ -1783,6 +1953,33 @@ class BlockchainDB
*/
virtual bool for_all_alt_blocks(std::function<bool(const crypto::hash &blkid, const alt_block_data_t &data, const cryptonote::blobdata_ref *blob)> f, bool include_blob = false) const = 0;
+ /**
+ * @brief return tree's root and n_tree_layers at a specific block idx
+ *
+ * Gets the tree root and n_tree_layers composed of all valid spendable
+ * outputs when blk_idx is the tip of the chain.
+ *
+ * If the chain tip is block index n, and `blk_idx == n`, then this will
+ * return the tree root and n layers in a tree composed of all valid
+ * spendable outputs in the chain at that time.
+ *
+ * If the chain tip is block index n, and `blk_idx == n-1`, then this will
+ * return the tree root and n layers in a tree composed of all valid
+ * spendable outputs in the chain *when the chain tip was block index n - 1*.
+ *
+ * Note that the tree stored in the database may not match up with the tree
+ * root returned here, since the tree stored in the db may have grown past
+ * the chain tip's tree, with outputs that will unlock in future blocks.
+ *
+ * This function throws if the db does not have a tree root stored for the
+ * given blk_idx.
+ *
+ * @param blk_idx the state of the tree as of this block index
+ * @param tree_root_out return-by-reference tree root
+ *
+ * @return n tree layers when blk_idx was chain tip
+ */
+ virtual uint8_t get_tree_root_at_blk_idx(const uint64_t blk_idx, crypto::ec_point &tree_root_out) const = 0;
//
// Hard fork related storage
### src/blockchain_db/lmdb/db_lmdb.cpp
@@ -27,6 +27,7 @@
#include "db_lmdb.h"
+#include <boost/endian/conversion.hpp>
#include <boost/filesystem.hpp>
#include <boost/filesystem/fstream.hpp>
#include <boost/format.hpp>
@@ -43,9 +44,11 @@
#include "common/pruning.h"
#include "cryptonote_basic/cryptonote_format_utils.h"
#include "crypto/crypto.h"
+#include "fcmp_pp/fcmp_pp_serialization.h"
#include "misc_language.h"
#include "profile_tools.h"
#include "ringct/rctOps.h"
+#include "serialization/binary_utils.h"
#undef MONERO_DEFAULT_LOG_CATEGORY
#define MONERO_DEFAULT_LOG_CATEGORY "blockchain.db.lmdb"
@@ -59,7 +62,11 @@ using epee::string_tools::pod_to_hex;
using namespace crypto;
// Increase when the DB structure changes
+#ifdef ENABLE_FCMP_INTEGRATION
+#define VERSION 7
+#else
#define VERSION 5
+#endif
namespace
{
@@ -120,6 +127,16 @@ int BlockchainLMDB::compare_uint64(const MDB_val *a, const MDB_val *b)
return (va < vb) ? -1 : va > vb;
}
+int BlockchainLMDB::compare_little_endian_uint64(const MDB_val *a, const MDB_val *b)
+{
+ uint64_t va, vb;
+ memcpy(&va, a->mv_data, sizeof(va));
+ memcpy(&vb, b->mv_data, sizeof(vb));
+ boost::endian::little_to_native_inplace(va);
+ boost::endian::little_to_native_inplace(vb);
+ return (va < vb) ? -1 : va > vb;
+}
+
int BlockchainLMDB::compare_hash32(const MDB_val *a, const MDB_val *b)
{
uint32_t *va = (uint32_t*) a->mv_data;
@@ -174,6 +191,13 @@ namespace
*
* spent_keys input hash -
*
+ * locked_outputs block ID [{UnifiedOutput}...]
+ * timelocked_outputs block ID [{UnifiedOutput}...]
+ * leaves leaf_idx {mdb_leaf}
+ * layers layer_idx [{child_chunk_idx, child_chunk_hash}...]
+ * tree_edges block ID [child_chunk_hash]
+ * tree_meta block ID n_leaf_tuples
+ *
* txpool_meta txn hash txn metadata
* txpool_blob txn hash txn blob
*
@@ -185,7 +209,8 @@ namespace
* attached as a prefix on the Data to serve as the DUPSORT key.
* (DUPFIXED saves 8 bytes per record.)
*
- * The output_amounts table doesn't use a dummy key, but uses DUPSORT.
+ * The output_amounts, locked_outputs, layers, and timelocked_outputs
+ * tables don't use a dummy key, but use DUPSORT.
*/
const char* const LMDB_BLOCKS = "blocks";
const char* const LMDB_BLOCK_HEIGHTS = "block_heights";
@@ -203,6 +228,14 @@ const char* const LMDB_OUTPUT_TXS = "output_txs";
const char* const LMDB_OUTPUT_AMOUNTS = "output_amounts";
const char* const LMDB_SPENT_KEYS = "spent_keys";
+// Curve trees merkle tree tables
+const char* const LMDB_LOCKED_OUTPUTS = "locked_outputs";
+const char* const LMDB_TIMELOCKED_OUTPUTS = "timelocked_outputs";
+const char* const LMDB_LEAVES = "leaves";
+const char* const LMDB_LAYERS = "layers";
+const char* const LMDB_TREE_EDGES = "tree_edges";
+const char* const LMDB_TREE_META = "tree_meta";
+
const char* const LMDB_TXPOOL_META = "txpool_meta";
const char* const LMDB_TXPOOL_BLOB = "txpool_blob";
@@ -310,22 +343,40 @@ typedef struct blk_height {
typedef struct pre_rct_outkey {
uint64_t amount_index;
- uint64_t output_id;
+ uint64_t unified_id;
pre_rct_output_data_t data;
} pre_rct_outkey;
typedef struct outkey {
uint64_t amount_index;
- uint64_t output_id;
+ uint64_t unified_id;
output_data_t data;
} outkey;
typedef struct outtx {
- uint64_t output_id;
+ uint64_t unified_id;
crypto::hash tx_hash;
uint64_t local_index;
} outtx;
+#pragma pack(push, 1)
+typedef struct mdb_leaf {
+ uint64_t leaf_idx;
+ uint64_t unified_id;
+} mdb_leaf;
+#pragma pack(pop)
+static_assert(sizeof(mdb_leaf) == (8+8), "mdb_leaf unexpected size");
+
+typedef struct layer_val {
+ uint64_t child_chunk_idx;
+ crypto::ec_point child_chunk_hash;
+} layer_val;
+static_assert(sizeof(layer_val) == (8+32), "layer_val unexpected size");
+
+typedef struct mdb_tree_meta {
+ uint64_t n_leaf_tuples;
+} mdb_tree_meta;
+
std::atomic<uint64_t> mdb_txn_safe::num_active_txns{0};
thread_local uint64_t mdb_txn_safe::num_active_txns_per_thread = 0;
std::atomic_flag mdb_txn_safe::creation_gate = ATOMIC_FLAG_INIT;
@@ -838,6 +889,11 @@ void BlockchainLMDB::remove_block()
CURSOR(block_info)
CURSOR(block_heights)
CURSOR(blocks)
+
+#ifdef ENABLE_FCMP_INTEGRATION
+ BlockchainDB::trim_block();
+#endif
+
MDB_val_copy<uint64_t> k(m_height - 1);
MDB_val h = k;
if ((result = mdb_cursor_get(m_cur_block_info, (MDB_val *)&zerokval, &h, MDB_GET_BOTH)))
@@ -1034,7 +1090,7 @@ uint64_t BlockchainLMDB::add_output(const crypto::hash& tx_hash,
check_open();
mdb_txn_cursors *m_cursors = &m_wcursors;
uint64_t m_height = height();
- uint64_t m_num_outputs = num_outputs();
+ uint64_t m_num_outputs = total_outputs();
int result = 0;
@@ -1070,7 +1126,7 @@ uint64_t BlockchainLMDB::add_output(const crypto::hash& tx_hash,
throw0(DB_ERROR(lmdb_error("Failed to get output amount in db transaction: ", result).c_str()));
else
ok.amount_index = 0;
- ok.output_id = m_num_outputs;
+ ok.unified_id = m_num_outputs;
ok.data.pubkey = output_public_key;
ok.data.unlock_time = unlock_time;
ok.data.height = m_height;
@@ -1133,11 +1189,11 @@ void BlockchainLMDB::remove_tx_outputs(const uint64_t tx_id, const transaction&
for (size_t i = tx.vout.size(); i-- > 0;)
{
uint64_t amount = is_pseudo_rct ? 0 : tx.vout[i].amount;
- remove_output(amount, amount_output_indices[i]);
+ remove_output(tx.is_coinbase(), amount, amount_output_indices[i]);
}
}
-void BlockchainLMDB::remove_output(const uint64_t amount, const uint64_t& out_index)
+void BlockchainLMDB::remove_output(const bool is_coinbase, const uint64_t amount, const uint64_t& out_index)
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
check_open();
@@ -1155,7 +1211,12 @@ void BlockchainLMDB::remove_output(const uint64_t amount, const uint64_t& out_in
throw0(DB_ERROR(lmdb_error("DB error attempting to get an output", result).c_str()));
const pre_rct_outkey *ok = (const pre_rct_outkey *)v.mv_data;
- MDB_val_set(otxk, ok->output_id);
+
+ const uint64_t unlock_time = ok->data.unlock_time;
+ const uint64_t output_height = ok->data.height;
+ const uint64_t unified_id = ok->unified_id;
+
+ MDB_val_set(otxk, unified_id);
result = mdb_cursor_get(m_cur_output_txs, (MDB_val *)&zerokval, &otxk, MDB_GET_BOTH);
if (result == MDB_NOTFOUND)
{
@@ -1173,6 +1234,71 @@ void BlockchainLMDB::remove_output(const uint64_t amount, const uint64_t& out_in
result = mdb_cursor_del(m_cur_output_amounts, 0);
if (result)
throw0(DB_ERROR(lmdb_error(std::string("Error deleting amount for output index ").append(boost::lexical_cast<std::string>(out_index).append(": ")).c_str(), result).c_str()));
+
+#ifdef ENABLE_FCMP_INTEGRATION
+ // Remove output from locked outputs table if present. We expect all valid
+ // outputs to be in the locked outputs table because remove_output is called
+ // when removing the top block from the chain, and all outputs from the top
+ // block are expected to be locked until they are at least 10 blocks old (10
+ // is the lower bound). An output might not be in the locked outputs table if
+ // it is invalid, then gets removed from the locked outputs table upon growing
+ // the tree.
+ // TODO: test case where we add an invalid output to the chain, grow the tree
+ // in the block in which that output unlocks, pop blocks to remove that output
+ // from the chain, then progress the chain again.
+ CURSOR(locked_outputs);
+
+ bool is_custom_timelocked = false;
+ const uint64_t last_locked_block = cryptonote::get_last_locked_block_index_for_tree(is_coinbase, unlock_time, output_height, is_custom_timelocked);
+
+ // Since the UnifiedOutput unified_id is stored little endian always, we have
+ // to convert to LE here for the lookup to work as expected for BE systems.
+ const uint64_t unified_id_le = boost::endian::native_to_little(unified_id);
+
+ MDB_val_set(k_block_id, last_locked_block);
+ MDB_val_set(v_output, unified_id_le);
+
+ result = mdb_cursor_get(m_cur_locked_outputs, &k_block_id, &v_output, MDB_GET_BOTH);
+ if (result == MDB_NOTFOUND)
+ {
+ // We expect this output is invalid
+ }
+ else if (result)
+ {
+ throw1(DB_ERROR(lmdb_error("Error adding removal of locked output to db transaction", result).c_str()));
+ }
+ else
+ {
+ result = mdb_cursor_del(m_cur_locked_outputs, 0);
+ if (result)
+ throw0(DB_ERROR(lmdb_error(std::string("Error deleting locked output index ").append(boost::lexical_cast<std::string>(out_index).append(": ")).c_str(), result).c_str()));
+ }
+
+ // Remove output from custom timelocked outputs table if expected present
+ if (!is_custom_timelocked)
+ return;
+
+ CURSOR(timelocked_outputs);
+
+ MDB_val_set(k_timelocked_block_id, last_locked_block);
+ MDB_val_set(v_timelocked_output, unified_id_le);
+
+ result = mdb_cursor_get(m_cur_timelocked_outputs, &k_timelocked_block_id, &v_timelocked_output, MDB_GET_BOTH);
+ if (result == MDB_NOTFOUND)
+ {
+ // Output is either not timelocked or is invalid
+ }
+ else if (result)
+ {
+ throw1(DB_ERROR(lmdb_error("Error adding removal of timelocked output to db transaction", result).c_str()));
+ }
+ else
+ {
+ result = mdb_cursor_del(m_cur_timelocked_outputs, 0);
+ if (result)
+ throw0(DB_ERROR(lmdb_error(std::string("Error deleting timelocked output index ").append(boost::lexical_cast<std::string>(out_index).append(": ")).c_str(), result).c_str()));
+ }
+#endif
}
void BlockchainLMDB::prune_outputs(uint64_t amount)
@@ -1197,29 +1323,29 @@ void BlockchainLMDB::prune_outputs(uint64_t amount)
mdb_size_t num_elems;
mdb_cursor_count(m_cur_output_amounts, &num_elems);
MINFO(num_elems << " outputs found");
- std::vector<uint64_t> output_ids;
- output_ids.reserve(num_elems);
+ std::vector<uint64_t> unified_ids;
+ unified_ids.reserve(num_elems);
while (1)
{
const pre_rct_outkey *okp = (const pre_rct_outkey *)v.mv_data;
- output_ids.push_back(okp->output_id);
- MDEBUG("output id " << okp->output_id);
+ unified_ids.push_back(okp->unified_id);
+ MDEBUG("unified id " << okp->unified_id);
result = mdb_cursor_get(m_cur_output_amounts, &k, &v, MDB_NEXT_DUP);
if (result == MDB_NOTFOUND)
break;
if (result)
throw0(DB_ERROR(lmdb_error("Error counting outputs: ", result).c_str()));
}
- if (output_ids.size() != num_elems)
+ if (unified_ids.size() != num_elems)
throw0(DB_ERROR("Unexpected number of outputs"));
result = mdb_cursor_del(m_cur_output_amounts, MDB_NODUPDATA);
if (result)
throw0(DB_ERROR(lmdb_error("Error deleting outputs: ", result).c_str()));
- for (uint64_t output_id: output_ids)
+ for (uint64_t unified_id: unified_ids)
{
- MDB_val_set(v, output_id);
+ MDB_val_set(v, unified_id);
result = mdb_cursor_get(m_cur_output_txs, (MDB_val *)&zerokval, &v, MDB_GET_BOTH);
if (result)
throw0(DB_ERROR(lmdb_error("Error looking up output: ", result).c_str()));
@@ -1266,6 +1392,661 @@ void BlockchainLMDB::remove_spent_key(const crypto::key_image& k_image)
}
}
+void BlockchainLMDB::add_locked_outs(const fcmp_pp::OutsByLastLockedBlock& outs_by_last_locked_block, const std::unordered_map<uint64_t/*unified_id*/, uint64_t/*last locked block_id*/>& timelocked_outputs)
+{
+ LOG_PRINT_L3("BlockchainLMDB::" << __func__);
+ check_open();
+ mdb_txn_cursors *m_cursors = &m_wcursors;
+
+ CURSOR(locked_outputs)
+ CURSOR(timelocked_outputs)
+
+ // Add to the locked outputs and custom timelocked tables, keyed by last locked block
+ for (const auto &last_locked_block : outs_by_last_locked_block)
+ {
+ const uint64_t last_locked_block_idx = last_locked_block.first;
+ for (const fcmp_pp::UnifiedOutput &locked_output : last_locked_block.second)
+ {
+ const cryptonote::blobdata output_blob = cryptonote::t_serializable_object_to_blob(locked_output);
+ if (output_blob.size() != SIZEOF_SERIALIZED_UNIFIED_OUTPUT)
+ throw0(DB_ERROR(("Out " + std::to_string(locked_output.unified_id) + " has unexpected blob size" + std::to_string(output_blob.size())).c_str()));
+ static_assert(SIZEOF_SERIALIZED_UNIFIED_OUTPUT == 73, "Unified output is stored serialized in 73 bytes");
+
+ MDB_val_set(k_block_id, last_locked_block_idx);
+ MDB_val_sized(v_output, output_blob);
+ int result = mdb_cursor_put(m_cur_locked_outputs, &k_block_id, &v_output, MDB_APPENDDUP);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to add locked output: ", result).c_str()));
+
+ if (timelocked_outputs.find(locked_output.unified_id) == timelocked_outputs.end())
+ continue;
+
+ // Add to custom timelocked outputs table also so it does not get removed in del_locked_outs_at_block_idx
+ MDB_val_set(k_timelocked_block_id, last_locked_block_idx);
+ MDB_val_sized(v_timelocked_output, output_blob);
+ result = mdb_cursor_put(m_cur_timelocked_outputs, &k_timelocked_block_id, &v_timelocked_output, MDB_APPENDDUP);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to add timelocked output: ", result).c_str()));
+ }
+ }
+}
+
+std::vector<fcmp_pp::UnifiedOutput> BlockchainLMDB::get_outs_at_last_locked_block_idx(uint64_t block_idx) const
+{
+ LOG_PRINT_L3("BlockchainLMDB::" << __func__);
+ check_open();
+
+ TXN_PREFIX_RDONLY();
+ RCURSOR(locked_outputs)
+
+ MDB_val_set(k_block_idx, block_idx);
+ MDB_val v_output;
+
+ // Get all the locked outputs at the provided block id
+ std::vector<fcmp_pp::UnifiedOutput> outs;
+
+ MDB_cursor_op op = MDB_SET;
+ while (1)
+ {
+ int result = mdb_cursor_get(m_cur_locked_outputs, &k_block_idx, &v_output, op);
+ if (result == MDB_NOTFOUND)
+ break;
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to get next locked outputs: ", result).c_str()));
+ if (op == MDB_SET)
+ {
+ op = MDB_GET_MULTIPLE;
+ continue;
+ }
+
+ const uint64_t blk_id = *(const uint64_t*)k_block_idx.mv_data;
+ if (blk_id != block_idx)
+ throw0(DB_ERROR(("Blk id " + std::to_string(blk_id) + " not the expected" + std::to_string(block_idx)).c_str()));
+
+ const char *range_begin = (const char*)v_output.mv_data;
+ const char *range_end = range_begin + v_output.mv_size;
+
+ if (v_output.mv_size % SIZEOF_SERIALIZED_UNIFIED_OUTPUT != 0)
+ throw0(DB_ERROR(("Blk id " + std::to_string(blk_id) + " page of outs has unexpected data size" + std::to_string(v_output.mv_size)).c_str()));
+ static_assert(SIZEOF_SERIALIZED_UNIFIED_OUTPUT == 73, "Unified output is stored serialized in 73 bytes");
+
+ auto it = range_begin;
+ while (it < range_end)
+ {
+ if ((it + SIZEOF_SERIALIZED_UNIFIED_OUTPUT) > range_end)
+ throw0(DB_ERROR("Out of bounds reading locked outputs"));
+
+ cryptonote::blobdata bd;
+ bd.assign(it, SIZEOF_SERIALIZED_UNIFIED_OUTPUT);
+
+ fcmp_pp::UnifiedOutput out;
+ if (!::serialization::parse_binary(bd, out))
+ throw0(DB_ERROR("Failed to de-serialize locked output"));
+
+ outs.emplace_back(std::move(out));
+ it += SIZEOF_SERIALIZED_UNIFIED_OUTPUT;
+ }
+ op = MDB_NEXT_MULTIPLE;
+ }
+
+ TXN_POSTFIX_RDONLY();
+
+ return outs;
+}
+
+void BlockchainLMDB::del_locked_outs_at_block_idx(uint64_t block_idx)
+{
+ LOG_PRINT_L3("BlockchainLMDB::" << __func__);
+ check_open();
+ mdb_txn_cursors *m_cursors = &m_wcursors;
+
+ CURSOR(locked_outputs)
+
+ MDB_val_set(k_block_idx, block_idx);
+
+ int result = mdb_cursor_get(m_cur_locked_outputs, &k_block_idx, NULL, MDB_SET);
+ if (result == MDB_NOTFOUND)
+ return;
+ if (result != MDB_SUCCESS)
+ throw1(DB_ERROR(lmdb_error("Error finding locked outputs to remove: ", result).c_str()));
+
+ result = mdb_cursor_del(m_cur_locked_outputs, MDB_NODUPDATA);
+ if (result)
+ throw1(DB_ERROR(lmdb_error("Error removing locked outputs: ", result).c_str()));
+}
+
+uint64_t BlockchainLMDB::get_n_leaf_tuples() const
+{
+ LOG_PRINT_L3("BlockchainLMDB::" << __func__);
+ check_open();
+
+ TXN_PREFIX_RDONLY();
+ RCURSOR(leaves)
+
+ // Get the number of leaf tuples in the tree
+ std::uint64_t n_leaf_tuples = 0;
+
+ {
+ MDB_val k, v;
+ int result = mdb_cursor_get(m_cur_leaves, &k, &v, MDB_LAST);
+ if (result == MDB_NOTFOUND)
+ n_leaf_tuples = 0;
+ else if (result == MDB_SUCCESS)
+ n_leaf_tuples = 1 + ((const mdb_leaf*)v.mv_data)->leaf_idx;
+ else
+ throw0(DB_ERROR(lmdb_error("Failed to get last leaf: ", result).c_str()));
+ }
+
+ TXN_POSTFIX_RDONLY();
+
+ return n_leaf_tuples;
+}
+
+uint64_t BlockchainLMDB::get_block_n_leaf_tuples(const uint64_t block_idx) const
+{
+ LOG_PRINT_L3("BlockchainLMDB::" << __func__);
+ check_open();
+
+ TXN_PREFIX_RDONLY();
+ RCURSOR(tree_meta);
+
+ MDB_val_set(k_block_id, block_idx);
+ MDB_val v;
+ int result = mdb_cursor_get(m_cur_tree_meta, &k_block_id, &v, MDB_SET);
+ if (result == MDB_NOTFOUND)
+ throw0(BLOCK_DNE(std::string("Attempt to get tree meta from blk idx ").append(boost::lexical_cast<std::string>(block_idx)).append(" failed -- tree meta not in db").c_str()));
+ if (result != MDB_SUCCESS)
+ throw1(DB_ERROR(lmdb_error("Error getting tree meta n leaf tuples: ", result).c_str()));
+
+ uint64_t n_leaf_tuples = ((mdb_tree_meta *)v.mv_data)->n_leaf_tuples;
+
+ TXN_POSTFIX_RDONLY();
+
+ return n_leaf_tuples;
+}
+
+uint8_t BlockchainLMDB::get_tree_root_at_blk_idx(const uint64_t blk_idx, crypto::ec_point &tree_root_out) const
+{
+ const std::vector<crypto::ec_point> tree_edge = this->get_tree_edge(blk_idx);
+ if (tree_edge.empty())
+ {
+ tree_root_out = crypto::ec_point{};
+ return 0;
+ }
+ tree_root_out = tree_edge.back();
+ static_assert(sizeof(std::size_t) >= sizeof(uint8_t), "unexpected size of size_t");
+ return (uint8_t) tree_edge.size();
+}
+
+uint64_t BlockchainLMDB::get_tree_block_idx() const
+{
+ LOG_PRINT_L3("BlockchainLMDB::" << __func__);
+ check_open();
+
+ TXN_PREFIX_RDONLY();
+
+ RCURSOR(tree_meta)
+
+ MDB_val k;
+ MDB_val v;
+ int result = mdb_cursor_get(m_cur_tree_meta, &k, &v, MDB_LAST);
+ if (result != MDB_SUCCESS)
+ throw1(DB_ERROR(lmdb_error("Error finding last tree meta: ", result).c_str()));
+
+ uint64_t block_idx = *(uint64_t *)k.mv_data;
+
+ TXN_POSTFIX_RDONLY();
+
+ return block_idx;
+}
+
+std::vector<crypto::ec_point> BlockchainLMDB::get_tree_edge(uint64_t block_id) const
+{
+ LOG_PRINT_L3("BlockchainLMDB::" << __func__);
+ check_open();
+
+ TXN_PREFIX_RDONLY();
+ RCURSOR(tree_edges)
+
+ MDB_val_set(k_block_id, block_id);
+ MDB_val v;
+ int result = mdb_cursor_get(m_cur_tree_edges, &k_block_id, &v, MDB_SET);
+ if (result != MDB_SUCCESS)
+ throw1(DB_ERROR(lmdb_error("Error finding tree edge at block " + std::to_string(block_id) + ": ", result).c_str()));
+
+ crypto::ec_point* tree_edge = (crypto::ec_point*)v.mv_data;
+ const std::size_t n_layers = v.mv_size / sizeof(crypto::ec_point);
+
+ std::vector<crypto::ec_point> res;
+ res.reserve(n_layers);
+ for (std::size_t i = 0; i < n_layers; ++i)
+ res.emplace_back(std::move(tree_edge[i]));
+
+ TXN_POSTFIX_RDONLY();
+
+ return res;
+}
+
+void BlockchainLMDB::save_tree_meta(const uint64_t block_idx, const uint64_t n_leaf_tuples, const std::vector<crypto::ec_point> &tree_edge)
+{
+ LOG_PRINT_L3("BlockchainLMDB::" << __func__);
+ check_open();
+ mdb_txn_cursors *m_cursors = &m_wcursors;
+
+ CURSOR(tree_edges);
+ CURSOR(tree_meta);
+
+ CHECK_AND_ASSERT_THROW_MES(m_curve_trees != nullptr, "curve trees must be set");
+
+ if (tree_edge.size() > std::numeric_limits<uint8_t>::max())
+ throw0(DB_ERROR("too many tree edge members"));
+ if (tree_edge.size() != m_curve_trees->n_layers(n_leaf_tuples))
+ throw0(DB_ERROR("Number of layers in the tree edge does not match expected for the saved n leaf tuples"));
+
+ // 1. Save the tree edge
+ MDB_val_copy<uint64_t> k_tee(block_idx);
+ MDB_val v;
+ const std::size_t n_layers = tree_edge.size();
+ v.mv_data = n_layers ? (void *)tree_edge.data() : (void*)"";
+ v.mv_size = sizeof(crypto::ec_point) * n_layers;
+
+ int result = mdb_cursor_put(m_cur_tree_edges, &k_tee, &v, MDB_NOOVERWRITE);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to set last hash: ", result).c_str()));
+
+ // 2. Save the tree meta
+ MDB_val_copy<uint64_t> k_meta(block_idx);
+ mdb_tree_meta tree_meta;
+ tree_meta.n_leaf_tuples = n_leaf_tuples;
+ MDB_val_set(v_meta, tree_meta);
+
+ MDEBUG("Saving tree meta for block idx " << block_idx << " (n_leaf_tuples=" << n_leaf_tuples
+ << ", root=" << (tree_edge.size() ? tree_edge.back() : crypto::ec_point{}) << ")");
+
+ result = mdb_cursor_put(m_cur_tree_meta, &k_meta, &v_meta, MDB_NOOVERWRITE);
+ if (result != MDB_SUCCESS)
+ throw1(DB_ERROR(lmdb_error("Error setting tree meta: ", result).c_str()));
+}
+
+void BlockchainLMDB::del_tree_meta(const uint64_t block_idx)
+{
+ LOG_PRINT_L3("BlockchainLMDB::" << __func__);
+ check_open();
+ mdb_txn_cursors *m_cursors = &m_wcursors;
+
+ CURSOR(tree_edges)
+ CURSOR(tree_meta)
+
+ MDB_val_set(k_block_id, block_idx);
+
+ // 1. Delete tree edge at this block
+ int result = mdb_cursor_get(m_cur_tree_edges, &k_block_id, NULL, MDB_SET);
+ if (result != MDB_SUCCESS)
+ throw1(DB_ERROR(lmdb_error("Error finding tree edge to remove at block " + std::to_string(block_idx) + ": ", result).c_str()));
+ result = mdb_cursor_del(m_cur_tree_edges, 0);
+ if (result)
+ throw1(DB_ERROR(lmdb_error("Error removing tree edge: ", result).c_str()));
+
+ // 2. Delete tree meta at this block
+ result = mdb_cursor_get(m_cur_tree_meta, &k_block_id, NULL, MDB_SET);
+ if (result != MDB_SUCCESS)
+ throw1(DB_ERROR(lmdb_error("Error finding tree meta to remove at block " + std::to_string(block_idx) + ": ", result).c_str()));
+ result = mdb_cursor_del(m_cur_tree_meta, 0);
+ if (result)
+ throw1(DB_ERROR(lmdb_error("Error removing tree meta: ", result).c_str()));
+}
+
+std::vector<crypto::ec_point> BlockchainLMDB::grow_with_tree_extension(const fcmp_pp::CompressedTreeExtension &tree_extension)
+{
+ LOG_PRINT_L3("BlockchainLMDB::" << __func__);
+ check_open();
+ mdb_txn_cursors *m_cursors = &m_wcursors;
+
+ CHECK_AND_ASSERT_THROW_MES(m_write_txn != nullptr, "Must have m_write_txn set to grow tree");
+
+ CURSOR(leaves)
+
+ // Insert the leaves
+ auto &leaves = tree_extension.leaves;
+ for (uint64_t i = 0; i < leaves.tuples.size(); ++i)
+ {
+ const uint64_t leaf_idx = i + leaves.start_idx;
+ mdb_leaf val{.leaf_idx = leaf_idx, .unified_id = leaves.tuples.at(i).unified_id};
+ MDB_val_set(v, val);
+
+ int result = mdb_cursor_put(m_cur_leaves, (MDB_val *)&zerokval, &v, MDB_APPENDDUP);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to add leaf: ", result).c_str()));
+ }
+
+ // Grow the layers
+ const auto &layer_extensions = tree_extension.layer_extensions;
+ if (layer_extensions.empty())
+ throw0(DB_ERROR("Unexpected 0 n layers"));
+
+ std::vector<crypto::ec_point> tree_edge;
+ tree_edge.reserve(layer_extensions.size());
+ for (uint64_t layer_idx = 0; layer_idx < layer_extensions.size(); ++layer_idx)
+ {
+ MTRACE("Growing layer " << layer_idx);
+ tree_edge.emplace_back(this->grow_layer(layer_extensions.at(layer_idx), layer_idx));
+ }
+
+ return tree_edge;
+}
+
+crypto::ec_point BlockchainLMDB::grow_layer(const fcmp_pp::CompressedLayerExtension &layer_extension,
+ const uint64_t layer_idx)
+{
+ LOG_PRINT_L3("BlockchainLMDB::" << __func__);
+ check_open();
+ mdb_txn_cursors *m_cursors = &m_wcursors;
+
+ CURSOR(layers)
+
+ CHECK_AND_ASSERT_THROW_MES(!layer_extension.hashes.empty(), "empty layer extension");
+ std::vector<crypto::ec_point> hashes;
+ hashes.reserve(layer_extension.hashes.size());
+
+ // Expected: layer_extension.start_idx should line up with the end of the layer
+
+ MDB_val_copy<uint64_t> k(layer_idx);
+
+ // 1. Update the existing last hash if necessary
+ if (layer_extension.update_existing_last_hash)
+ {
+ hashes.emplace_back(layer_extension.hashes.front());
+
+ // We updated the last hash, so update it
+ layer_val lv;
+ lv.child_chunk_idx = layer_extension.start_idx;
+ lv.child_chunk_hash = hashes.back();
+ MDB_val_set(v, lv);
+
+ // We expect to overwrite the existing hash
+ // Expected: the hash should already exist and be the expected existing last hash
+ int result = mdb_cursor_put(m_cur_layers, &k, &v, 0);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to update chunk hash: ", result).c_str()));
+ }
+
+ // 2. Add all the new hashes found in the extension
+ for (uint64_t i = layer_extension.update_existing_last_hash ? 1 : 0; i < layer_extension.hashes.size(); ++i)
+ {
+ hashes.emplace_back(layer_extension.hashes[i]);
+
+ layer_val lv;
+ lv.child_chunk_idx = i + layer_extension.start_idx;
+ lv.child_chunk_hash = hashes.back();
+ MDB_val_set(v, lv);
+
+ int result = mdb_cursor_put(m_cur_layers, &k, &v, MDB_APPENDDUP);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to add hash: ", result).c_str()));
+ }
+
+ return hashes.back();
+}
+
+uint64_t BlockchainLMDB::trim_leaves(const uint64_t new_n_leaf_tuples, const uint64_t trim_block_idx)
+{
+ LOG_PRINT_L3("BlockchainLMDB::" << __func__);
+ check_open();
+ mdb_txn_cursors *m_cursors = &m_wcursors;
+
+ CURSOR(leaves)
+ CURSOR(locked_outputs)
+
+ CHECK_AND_ASSERT_THROW_MES(m_write_txn != nullptr, "Must have m_write_txn set to trim tree");
+
+ const uint64_t old_n_leaf_tuples = this->get_n_leaf_tuples();
+ if (new_n_leaf_tuples > old_n_leaf_tuples)
+ throw1(DB_ERROR("Cannot have more leaves in tree after trimming than exist in the tree already"));
+
+ // Return if we don't need to trim any leaves
+ if (new_n_leaf_tuples == old_n_leaf_tuples)
+ return old_n_leaf_tuples;
+
+ // Trim the leaves, re-adding to locked outputs table
+ std::vector<uint64_t> unified_ids;
+ unified_ids.reserve(old_n_leaf_tuples - new_n_leaf_tuples);
+ for (uint64_t i = new_n_leaf_tuples; i < old_n_leaf_tuples; ++i)
+ {
+ MDB_val_copy<uint64_t> k(i);
+ MDB_val v = k;
+ int result = mdb_cursor_get(m_cur_leaves, (MDB_val *)&zerokval, &v, MDB_GET_BOTH);
+ if (result == MDB_NOTFOUND)
+ throw0(DB_ERROR("leaf not found"));
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to get leaf: ", result).c_str()));
+
+ const auto *o = (mdb_leaf *)v.mv_data;
+ unified_ids.push_back(o->unified_id);
+
+ // Delete the leaf
+ result = mdb_cursor_del(m_cur_leaves, 0);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Error removing leaf: ", result).c_str()));
+
+ MDEBUG("Successfully removed leaf at leaf_tuple_idx: " << i);
+ }
+
+ const auto unified_outputs = this->get_unified_output_by_id(unified_ids);
+
+ MDB_val_set(k_block_id, trim_block_idx);
+ for (const fcmp_pp::UnifiedOutput &unified_output : unified_outputs)
+ {
+ // Re-add the output to the locked output table in order. The output should
+ // still be in the outputs tables.
+ const cryptonote::blobdata output_blob = cryptonote::t_serializable_object_to_blob(unified_output);
+ if (output_blob.size() != SIZEOF_SERIALIZED_UNIFIED_OUTPUT)
+ throw0(DB_ERROR(("Output " + std::to_string(unified_output.unified_id) + " has unexpected blob size" + std::to_string(output_blob.size())).c_str()));
+ static_assert(SIZEOF_SERIALIZED_UNIFIED_OUTPUT == 73, "Unified output is stored serialized in 73 bytes");
+
+ MDB_val_sized(v_output, output_blob);
+ MDEBUG("Re-adding locked unified_id: " << unified_output.unified_id << " , last locked block: " << trim_block_idx);
+ int result = mdb_cursor_put(m_cur_locked_outputs, &k_block_id, &v_output, MDB_APPENDDUP);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to re-add locked output: ", result).c_str()));
+ }
+
+ return old_n_leaf_tuples;
+}
+
+void BlockchainLMDB::trim_layers(const uint64_t new_n_leaf_tuples,
+ const std::vector<uint64_t> &new_n_elems_in_layer,
+ const std::vector<crypto::ec_point> &new_tree_edge,
+ const uint64_t new_root_layer_idx)
+{
+ LOG_PRINT_L3("BlockchainLMDB::" << __func__);
+ check_open();
+ mdb_txn_cursors *m_cursors = &m_wcursors;
+
+ CURSOR(layers)
+
+ CHECK_AND_ASSERT_THROW_MES(m_write_txn != nullptr, "Must have m_write_txn set for trim_layers");
+
+ if (new_n_elems_in_layer.size() != new_tree_edge.size())
+ throw1(DB_ERROR("trim_layers: n_elems_per_layer.size() != new_tree_edge.size()"));
+
+ // If the tree is supposed to be empty, empty the tree
+ if (new_n_leaf_tuples == 0)
+ {
+ // Empty the layers table, no elems should remain
+ int result = mdb_drop(*m_write_txn, m_layers, 0);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Error emptying layers table: ", result).c_str()));
+ return;
+ }
+
+ // Shrink the layer sizes and update the last hash in each layer using the prev block's cached tree edge
+ for (std::size_t layer_idx = 0; layer_idx < new_n_elems_in_layer.size(); ++layer_idx)
+ {
+ const uint64_t n_new_elems = new_n_elems_in_layer[layer_idx];
+ if (n_new_elems == 0)
+ throw0(DB_ERROR("Unexpected 0 new elems"));
+ if (layer_idx >= new_tree_edge.size())
+ throw0(DB_ERROR("Tree edge is too small"));
+
+ // Delete all excess elems in layer
+ this->trim_layer(n_new_elems, layer_idx);
+
+ // Set the new last elem using the old tree edge elem
+ MDB_val_copy<uint64_t> k_layer_idx(layer_idx);
+ layer_val lv;
+ lv.child_chunk_idx = n_new_elems - 1;
+ lv.child_chunk_hash = std::move(new_tree_edge.at(layer_idx));
+ MDB_val_set(v_lv, lv);
+
+ // Overwrite layer last elem
+ MDEBUG("Re-setting elem " << lv.child_chunk_idx << " at layer idx " << layer_idx << ": " << epee::string_tools::pod_to_hex(lv.child_chunk_hash));
+ int result = mdb_cursor_put(m_cur_layers, &k_layer_idx, &v_lv, 0);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to update chunk hash: ", result).c_str()));
+ }
+
+ // Delete any remaining layers in layers after the root
+ while (1)
+ {
+ MDB_val k, v;
+ int result = mdb_cursor_get(m_cur_layers, &k, &v, MDB_LAST);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to get last elem: ", result).c_str()));
+
+ const uint64_t last_layer_idx = *(uint64_t *)k.mv_data;
+ if (last_layer_idx > new_root_layer_idx)
+ {
+ // Delete all elements in layers after the root
+ result = mdb_cursor_del(m_cur_layers, MDB_NODUPDATA);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Error removing elems after root: ", result).c_str()));
+ }
+ else if (last_layer_idx < new_root_layer_idx)
+ {
+ throw0(DB_ERROR("Encountered unexpected last elem in tree before the root"));
+ }
+ else // last_layer_idx == new_root_layer_idx
+ {
+ // We've trimmed all layers past the root, we're done
+ break;
+ }
+ }
+}
+
+void BlockchainLMDB::trim_layer(const uint64_t new_n_elems_in_layer, const uint64_t layer_idx)
+{
+ LOG_PRINT_L3("BlockchainLMDB::" << __func__);
+ check_open();
+ mdb_txn_cursors *m_cursors = &m_wcursors;
+
+ CURSOR(layers)
+
+ MDEBUG("Trimming layer " << layer_idx << ", new elems in layer: " << new_n_elems_in_layer);
+ MDB_val_copy<uint64_t> k(layer_idx);
+
+ // Get the number of existing elements in the layer
+ uint64_t old_n_elems_in_layer = 0;
+ {
+ // Get the first record in a layer so we can then get the last record
+ MDB_val v;
+ int result = mdb_cursor_get(m_cur_layers, &k, &v, MDB_SET);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to get first record in layer: ", result).c_str()));
+
+ result = mdb_cursor_get(m_cur_layers, &k, &v, MDB_LAST_DUP);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to get layer last elem on trim: ", result).c_str()));
+
+ const auto *lv = (layer_val *)v.mv_data;
+ old_n_elems_in_layer = (1 + lv->child_chunk_idx);
+ }
+
+ CHECK_AND_ASSERT_THROW_MES(old_n_elems_in_layer >= new_n_elems_in_layer, "unexpected old n elems in layer");
+ const uint64_t trim_n_elems_in_layer = old_n_elems_in_layer - new_n_elems_in_layer;
+
+ // Delete the elements
+ for (uint64_t i = 0; i < trim_n_elems_in_layer; ++i)
+ {
+ uint64_t last_elem_idx = (old_n_elems_in_layer - 1 - i);
+ MDB_val_set(v, last_elem_idx);
+
+ int result = mdb_cursor_get(m_cur_layers, &k, &v, MDB_GET_BOTH);
+ if (result == MDB_NOTFOUND)
+ throw0(DB_ERROR("leaf not found"));
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to get elem: ", result).c_str()));
+
+ result = mdb_cursor_del(m_cur_layers, 0);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Error removing elem: ", result).c_str()));
+
+ MDEBUG("Successfully removed elem at layer_idx: " << layer_idx << " , last_elem_idx: " << last_elem_idx);
+ }
+}
+
+std::vector<fcmp_pp::UnifiedOutput> BlockchainLMDB::get_unified_output_by_id(
+ const std::vector<uint64_t> &unified_ids) const
+{
+ LOG_PRINT_L3("BlockchainLMDB::" << __func__);
+ check_open();
+
+ TXN_PREFIX_RDONLY();
+
+ std::vector<fcmp_pp::UnifiedOutput> unified_outputs;
+ unified_outputs.reserve(unified_ids.size());
+
+ // Collect tx hashes from output id's, reading db output_txs
+ std::vector<tx_out_index> tois;
+ this->get_output_tx_and_index_from_unified(unified_ids, tois);
+
+ // Collect pruned txs from tx hashes, reading db tx_indices, and txs_pruned
+ std::unordered_map<crypto::hash, cryptonote::transaction> txs;
+ for (const auto &toi : tois)
+ {
+ if (txs.find(toi.first) == txs.end())
+ txs[toi.first] = this->get_pruned_tx(toi.first);
+ }
+
+ // Collect unified outputs from pruned tx data
+ std::unordered_map<rct::xmr_amount, rct::key> transparent_amount_commitments;
+ for (std::size_t i = 0; i < unified_ids.size(); ++i)
+ {
+ const uint64_t unified_id = unified_ids.at(i);
+ const auto &toi = tois.at(i);
+
+ const auto tx_it = txs.find(toi.first);
+ if (tx_it == txs.end())
+ throw0(DB_ERROR("Missing tx for provided output id."));
+ const auto &tx = tx_it->second;
+
+ // Amount commitment
+ const auto &out = tx.vout.at(toi.second);
+ rct::key commitment;
+ if (cryptonote::commitment_is_in_rct_signatures(tx))
+ {
+ commitment = tx.rct_signatures.outPk.at(toi.second).mask;
+ }
+ else
+ {
+ if (transparent_amount_commitments.find(out.amount) == transparent_amount_commitments.end())
+ transparent_amount_commitments[out.amount] = rct::zeroCommitVartime(out.amount);
+ commitment = transparent_amount_commitments[out.amount];
+ }
+
+ unified_outputs.emplace_back(fcmp_pp::UnifiedOutput{
+ .unified_id = unified_id,
+ .output_pair = cryptonote::to_output_pair(out.target, commitment)
+ });
+ }
+
+ if (unified_ids.size() != unified_outputs.size())
+ throw0(DB_ERROR("get_unified_output_by_id: unified_ids <> unified_outputs size mismatch"));
+
+ TXN_POSTFIX_RDONLY();
+
+ return unified_outputs;
+}
+
BlockchainLMDB::~BlockchainLMDB()
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
@@ -1280,7 +2061,7 @@ BlockchainLMDB::~BlockchainLMDB()
BlockchainLMDB::close();
}
-BlockchainLMDB::BlockchainLMDB(bool batch_transactions): BlockchainDB()
+BlockchainLMDB::BlockchainLMDB(bool batch_transactions, std::shared_ptr<fcmp_pp::curve_trees::CurveTreesV1> curve_trees): BlockchainDB()
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
// initialize folder to something "safe" just in case
@@ -1297,6 +2078,8 @@ BlockchainLMDB::BlockchainLMDB(bool batch_transactions): BlockchainDB()
// reset may also need changing when initialize things here
m_hardfork = nullptr;
+
+ m_curve_trees = curve_trees;
}
#ifdef WIN32
@@ -1475,6 +2258,31 @@ void BlockchainLMDB::open(const std::string& filename, const int db_flags)
lmdb_db_open(txn, LMDB_SPENT_KEYS, MDB_INTEGERKEY | MDB_CREATE | MDB_DUPSORT | MDB_DUPFIXED, m_spent_keys, "Failed to open db handle for m_spent_keys");
+#ifdef ENABLE_FCMP_INTEGRATION
+ try
+ {
+ lmdb_db_open(txn, LMDB_LOCKED_OUTPUTS, MDB_INTEGERKEY | MDB_DUPSORT | MDB_DUPFIXED | MDB_CREATE, m_locked_outputs, "Failed to open db handle for m_locked_outputs");
+ }
+ catch (cryptonote::DB_OPEN_FAILURE&)
+ {
+ if (mdb_flags & MDB_RDONLY)
+ {
+ txn.abort();
+ mdb_env_close(m_env);
+ m_open = false;
+ MFATAL("Existing lmdb database needs to be converted, which cannot be done on a read-only database.");
+ MFATAL("Please run monerod once to convert the database.");
+ return;
+ }
+ throw;
+ }
+ lmdb_db_open(txn, LMDB_TIMELOCKED_OUTPUTS, MDB_INTEGERKEY | MDB_DUPSORT | MDB_DUPFIXED | MDB_CREATE, m_timelocked_outputs, "Failed to open db handle for m_timelocked_outputs");
+ lmdb_db_open(txn, LMDB_LEAVES, MDB_INTEGERKEY | MDB_DUPSORT | MDB_DUPFIXED | MDB_CREATE, m_leaves, "Failed to open db handle for m_leaves");
+ lmdb_db_open(txn, LMDB_LAYERS, MDB_INTEGERKEY | MDB_DUPSORT | MDB_DUPFIXED | MDB_CREATE, m_layers, "Failed to open db handle for m_layers");
+ lmdb_db_open(txn, LMDB_TREE_EDGES, MDB_INTEGERKEY | MDB_CREATE, m_tree_edges, "Failed to open db handle for m_tree_edges");
+ lmdb_db_open(txn, LMDB_TREE_META, MDB_INTEGERKEY | MDB_CREATE, m_tree_meta, "Failed to open db handle for m_tree_meta");
+#endif
+
lmdb_db_open(txn, LMDB_TXPOOL_META, MDB_CREATE, m_txpool_meta, "Failed to open db handle for m_txpool_meta");
lmdb_db_open(txn, LMDB_TXPOOL_BLOB, MDB_CREATE, m_txpool_blob, "Failed to open db handle for m_txpool_blob");
@@ -1494,6 +2302,17 @@ void BlockchainLMDB::open(const std::string& filename, const int db_flags)
mdb_set_dupsort(txn, m_block_heights, compare_hash32);
mdb_set_dupsort(txn, m_tx_indices, compare_hash32);
mdb_set_dupsort(txn, m_output_amounts, compare_uint64);
+#ifdef ENABLE_FCMP_INTEGRATION
+ // We have to use little endian uint64 compare because we're using the binary serializer
+ // to serialize these types to/from compact blobs, which uses LE order for fixed size uints.
+ mdb_set_dupsort(txn, m_locked_outputs, compare_little_endian_uint64);
+ mdb_set_dupsort(txn, m_timelocked_outputs, compare_little_endian_uint64);
+
+ mdb_set_dupsort(txn, m_leaves, compare_uint64);
+ mdb_set_dupsort(txn, m_layers, compare_uint64);
+ mdb_set_compare(txn, m_tree_edges, compare_uint64);
+ mdb_set_compare(txn, m_tree_meta, compare_uint64);
+#endif
mdb_set_dupsort(txn, m_output_txs, compare_uint64);
mdb_set_dupsort(txn, m_block_info, compare_uint64);
if (!(mdb_flags & MDB_RDONLY))
@@ -1549,6 +2368,7 @@ void BlockchainLMDB::open(const std::string& filename, const int db_flags)
// See commit e5d2680094ee15889934fe28901e4e133cda56f2 2015/07/10
// We don't handle the old format previous to that commit.
txn.commit();
+ txn.uncheck();
m_open = true;
migrate(db_version);
return;
@@ -1671,6 +2491,20 @@ void BlockchainLMDB::reset()
throw0(DB_ERROR(lmdb_error("Failed to drop m_output_amounts: ", result).c_str()));
if (auto result = mdb_drop(txn, m_spent_keys, 0))
throw0(DB_ERROR(lmdb_error("Failed to drop m_spent_keys: ", result).c_str()));
+#ifdef ENABLE_FCMP_INTEGRATION
+ if (auto result = mdb_drop(txn, m_locked_outputs, 0))
+ throw0(DB_ERROR(lmdb_error("Failed to drop m_locked_outputs: ", result).c_str()));
+ if (auto result = mdb_drop(txn, m_timelocked_outputs, 0))
+ throw0(DB_ERROR(lmdb_error("Failed to drop m_timelocked_outputs: ", result).c_str()));
+ if (auto result = mdb_drop(txn, m_leaves, 0))
+ throw0(DB_ERROR(lmdb_error("Failed to drop m_leaves: ", result).c_str()));
+ if (auto result = mdb_drop(txn, m_layers, 0))
+ throw0(DB_ERROR(lmdb_error("Failed to drop m_layers: ", result).c_str()));
+ if (auto result = mdb_drop(txn, m_tree_edges, 0))
+ throw0(DB_ERROR(lmdb_error("Failed to drop m_tree_edges: ", result).c_str()));
+ if (auto result = mdb_drop(txn, m_tree_meta, 0))
+ throw0(DB_ERROR(lmdb_error("Failed to drop m_tree_meta: ", result).c_str()));
+#endif
(void)mdb_drop(txn, m_hf_starting_heights, 0); // this one is dropped in new code
if (auto result = mdb_drop(txn, m_hf_versions, 0))
throw0(DB_ERROR(lmdb_error("Failed to drop m_hf_versions: ", result).c_str()));
@@ -2943,7 +3777,7 @@ uint64_t BlockchainLMDB::height() const
return db_stats.ms_entries;
}
-uint64_t BlockchainLMDB::num_outputs() const
+uint64_t BlockchainLMDB::total_outputs() const
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
check_open();
@@ -2958,7 +3792,7 @@ uint64_t BlockchainLMDB::num_outputs() const
if (result == MDB_NOTFOUND)
num = 0;
else if (result == 0)
- num = 1 + ((const outtx*)v.mv_data)->output_id;
+ num = 1 + ((const outtx*)v.mv_data)->unified_id;
else
throw0(DB_ERROR(lmdb_error("Failed to query m_output_txs: ", result).c_str()));
@@ -3512,15 +4346,15 @@ output_data_t BlockchainLMDB::get_output_key(const uint64_t& amount, const uint6
return ret;
}
-tx_out_index BlockchainLMDB::get_output_tx_and_index_from_global(const uint64_t& output_id) const
+tx_out_index BlockchainLMDB::get_output_tx_and_index_from_unified(const uint64_t& unified_id) const
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
check_open();
TXN_PREFIX_RDONLY();
RCURSOR(output_txs);
- MDB_val_set(v, output_id);
+ MDB_val_set(v, unified_id);
auto get_result = mdb_cursor_get(m_cur_output_txs, (MDB_val *)&zerokval, &v, MDB_GET_BOTH);
if (get_result == MDB_NOTFOUND)
@@ -3843,7 +4677,7 @@ bool BlockchainLMDB::for_all_outputs(std::function<bool(uint64_t amount, const c
throw0(DB_ERROR("Failed to enumerate outputs"));
uint64_t amount = *(const uint64_t*)k.mv_data;
outkey *ok = (outkey *)v.mv_data;
- tx_out_index toi = get_output_tx_and_index_from_global(ok->output_id);
+ tx_out_index toi = get_output_tx_and_index_from_unified(ok->unified_id);
if (!f(amount, toi.first, ok->data.height, toi.second)) {
fret = false;
break;
@@ -4177,7 +5011,7 @@ void BlockchainLMDB::block_rtxn_abort() const
}
uint64_t BlockchainLMDB::add_block(const std::pair<block, blobdata>& blk, size_t block_weight, uint64_t long_term_block_weight, const difficulty_type& cumulative_difficulty, const uint64_t& coins_generated,
- const std::vector<std::pair<transaction, blobdata>>& txs)
+ const std::vector<std::pair<transaction, blobdata>>& txs, const std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments)
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
check_open();
@@ -4195,7 +5029,7 @@ uint64_t BlockchainLMDB::add_block(const std::pair<block, blobdata>& blk, size_t
try
{
- BlockchainDB::add_block(blk, block_weight, long_term_block_weight, cumulative_difficulty, coins_generated, txs);
+ BlockchainDB::add_block(blk, block_weight, long_term_block_weight, cumulative_difficulty, coins_generated, txs, transparent_amount_commitments);
}
catch (const DB_ERROR_TXN_START& e)
{
@@ -4224,7 +5058,7 @@ void BlockchainLMDB::pop_block(block& blk, std::vector<transaction>* txs)
}
}
-void BlockchainLMDB::get_output_tx_and_index_from_global(const std::vector<uint64_t> &global_indices,
+void BlockchainLMDB::get_output_tx_and_index_from_unified(const std::vector<uint64_t> &global_indices,
std::vector<tx_out_index> &tx_out_indices) const
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
@@ -4235,9 +5069,9 @@ void BlockchainLMDB::get_output_tx_and_index_from_global(const std::vector<uint6
TXN_PREFIX_RDONLY();
RCURSOR(output_txs);
- for (const uint64_t &output_id : global_indices)
+ for (const uint64_t &unified_id : global_indices)
{
- MDB_val_set(v, output_id);
+ MDB_val_set(v, unified_id);
auto get_result = mdb_cursor_get(m_cur_output_txs, (MDB_val *)&zerokval, &v, MDB_GET_BOTH);
if (get_result == MDB_NOTFOUND)
@@ -4331,13 +5165,13 @@ void BlockchainLMDB::get_output_tx_and_index(const uint64_t& amount, const std::
throw0(DB_ERROR(lmdb_error("Error attempting to retrieve an output from the db", get_result).c_str()));
const outkey *okp = (const outkey *)v.mv_data;
- tx_indices.push_back(okp->output_id);
+ tx_indices.push_back(okp->unified_id);
}
TIME_MEASURE_START(db3);
if(tx_indices.size() > 0)
{
- get_output_tx_and_index_from_global(tx_indices, indices);
+ get_output_tx_and_index_from_unified(tx_indices, indices);
}
TIME_MEASURE_FINISH(db3);
LOG_PRINT_L3("db3: " << db3);
@@ -5193,7 +6027,8 @@ void BlockchainLMDB::migrate_0_1()
throw0(DB_ERROR("Failed to parse block from blob retrieved from the db"));
const auto miner_blob = tx_to_blob(b.miner_tx);
- add_transaction(null_hash, b.miner_tx, epee::strspan<std::uint8_t>(miner_blob));
+ // Empty transparent amount commitments is ok bc only needed for v2 txs, which v0 db's can't have
+ add_transaction(null_hash, b.miner_tx, epee::strspan<std::uint8_t>(miner_blob), {});
for (unsigned int j = 0; j<b.tx_hashes.size(); j++) {
transaction tx;
hk.mv_data = &b.tx_hashes[j];
@@ -5203,7 +6038,7 @@ void BlockchainLMDB::migrate_0_1()
bd = {reinterpret_cast<char*>(v.mv_data), v.mv_size};
if (!parse_and_validate_tx_from_blob(bd, tx))
throw0(DB_ERROR("Failed to parse tx from blob retrieved from the db"));
- add_transaction(null_hash, std::move(tx), epee::strspan<std::uint8_t>(bd), &b.tx_hashes[j]);
+ add_transaction(null_hash, std::move(tx), epee::strspan<std::uint8_t>(bd), {}, &b.tx_hashes[j]);
result = mdb_cursor_del(c_txs, 0);
if (result)
throw0(DB_ERROR(lmdb_error("Failed to get record from txs: ", result).c_str()));
@@ -5800,6 +6635,376 @@ void BlockchainLMDB::migrate_4_5()
txn.commit();
}
+void BlockchainLMDB::migrate_5_7()
+{
+ LOG_PRINT_L3("BlockchainLMDB::" << __func__);
+ uint64_t i;
+ int result;
+ mdb_txn_safe txn(false);
+
+ MGINFO_YELLOW("Migrating blockchain from DB version 5 to 7 - this may take a while:");
+
+ const uint32_t DB_VERSION_IN_PROGRESS = 6;
+ const uint32_t DB_VERSION_ONCE_COMPLETE = 7;
+
+ const auto update_db_version = [this, &txn](uint32_t version)
+ {
+ MDB_val v;
+ v.mv_data = (void *)&version;
+ v.mv_size = sizeof(version);
+ MDB_val_str(vk, "version");
+ int result = mdb_txn_begin(m_env, NULL, 0, txn);
+ if (result)
+ throw0(DB_ERROR(lmdb_error("Failed to create a transaction for the db: ", result).c_str()));
+ result = mdb_put(txn, m_properties, &vk, &v, 0);
+ if (result)
+ throw0(DB_ERROR(lmdb_error("Failed to update version for the db: ", result).c_str()));
+ };
+
+ // Start by setting DB version to 6, which indicates the migration is in progress and prevents
+ // a monerod at a lower version from modifying the db once the migration is already in progress.
+ update_db_version(DB_VERSION_IN_PROGRESS);
+ txn.commit();
+
+ MDB_dbi m_tmp_last_output;
+ do
+ {
+ // 1. Prepare all valid outputs to be inserted into the merkle tree and
+ // place them in a locked outputs table. The key to this new table is the
+ // block id in which the outputs unlock.
+ {
+ MGINFO("Setting up a locked outputs table (step 1/2 of the full-chain membership proof migration)");
+
+ result = mdb_txn_begin(m_env, NULL, 0, txn);
+ if (result)
+ throw0(DB_ERROR(lmdb_error("Failed to create a transaction for the db: ", result).c_str()));
+ lmdb_db_open(txn, "tmp_last_output", MDB_INTEGERKEY | MDB_CREATE, m_tmp_last_output, "Failed to open db handle for m_tmp_last_output");
+ txn.commit();
+
+ if (!m_batch_transactions)
+ set_batch_transactions(true);
+ const std::size_t BATCH_SIZE = 10000;
+ batch_start();
+ txn.m_txn = m_write_txn->m_txn;
+
+ // Use this cache to know how to restart the migration if the process is killed
+ struct tmp_output_cache { uint64_t n_outputs_read; uint64_t amount; outkey ok; };
+ tmp_output_cache last_output;
+
+ MDB_cursor *c_output_amounts, *c_locked_outputs, *c_tmp_last_output, *c_timelocked_outputs;
+ MDB_val k, v;
+
+ i = 0;
+ const uint64_t n_outputs = this->total_outputs();
+ {
+ // Since we're using the output amounts table for an efficient migration, make
+ // sure we have the expected number of outputs saved in it. Pruning known
+ // spent data using the monero-blockchain-prune-known-spent-data tool would
+ // remove the data necessary to complete the migration. Normal pruned nodes
+ // are unaffected and should be able to proceed with the migration fine.
+ MDB_stat db_stats;
+ result = mdb_stat(txn, m_output_amounts, &db_stats);
+ if (result)
+ throw0(DB_ERROR(lmdb_error("Failed to query m_output_amounts: ", result).c_str()));
+ if (n_outputs != db_stats.ms_entries)
+ throw0(DB_ERROR("Missing output data, can't complete efficient migration. Delete your database and re-sync."));
+ }
+ std::size_t progress_width = 0;
+ MDB_cursor_op op = MDB_FIRST;
+ while (1)
+ {
+ if (!(i % BATCH_SIZE))
+ {
+ if (i)
+ {
+ LOGIF(el::Level::Info)
+ {
+ const uint64_t percent = std::min((i * 100) / n_outputs, (uint64_t)99);
+ const std::string progress_line = std::to_string(i) + " / " + std::to_string(n_outputs) +
+ " outputs (" + std::to_string(percent) + "% of step 1/2)";
+ std::cout << '\r' << std::string(progress_width, ' ') << '\r' << progress_line << std::flush;
+ progress_width = std::max(progress_width, progress_line.size());
+ }
+
+ // Update last output read
+ MDB_val_set(v_last_output, last_output);
+ result = mdb_cursor_put(c_tmp_last_output, (MDB_val*)&zerokval, &v_last_output, 0);
+ if (result)
+ throw0(DB_ERROR(lmdb_error("Failed to update max output id: ", result).c_str()));
+
+ // Commit and start a new txn
+ batch_stop();
+ batch_start();
+ txn.m_txn = m_write_txn->m_txn;
+
+ // Reset k and v so we continue migration from the last output
+ k = {sizeof(last_output.amount), (void *)&last_output.amount};
+
+ const std::size_t outkey_size = (last_output.amount == 0) ? sizeof(outkey) : sizeof(pre_rct_outkey);
+ v = {outkey_size, (void *)&last_output.ok};
+ }
+
+ // Open all cursors
+ result = mdb_cursor_open(txn, m_output_amounts, &c_output_amounts);
+ if (result)
+ throw0(DB_ERROR(lmdb_error("Failed to open a cursor for output amounts: ", result).c_str()));
+ result = mdb_cursor_open(txn, m_locked_outputs, &c_locked_outputs);
+ if (result)
+ throw0(DB_ERROR(lmdb_error("Failed to open a cursor for locked outputs: ", result).c_str()));
+ result = mdb_cursor_open(txn, m_tmp_last_output, &c_tmp_last_output);
+ if (result)
+ throw0(DB_ERROR(lmdb_error("Failed to open a cursor for temp last output: ", result).c_str()));
+ result = mdb_cursor_open(txn, m_timelocked_outputs, &c_timelocked_outputs);
+ if (result)
+ throw0(DB_ERROR(lmdb_error("Failed to open a cursor for timelocked outputs: ", result).c_str()));
+
+ // Get the cached last output from the db
+ bool found_cached_output = false;
+ tmp_output_cache cached_last_o;
+ if (i == 0)
+ {
+ MDB_val v_last_output;
+ result = mdb_cursor_get(c_tmp_last_output, (MDB_val*)&zerokval, &v_last_output, MDB_SET);
+ if (result != MDB_SUCCESS && result != MDB_NOTFOUND)
+ throw0(DB_ERROR(lmdb_error("Failed to get max output id: ", result).c_str()));
+ if (result != MDB_NOTFOUND)
+ {
+ cached_last_o = *(const tmp_output_cache*)v_last_output.mv_data;
+
+ if (n_outputs < cached_last_o.n_outputs_read)
+ throw0(DB_ERROR("Unexpected n_outputs_read on cached last output"));
+ if (n_outputs == cached_last_o.n_outputs_read)
+ break;
+
+ MDEBUG("Found cached output " << cached_last_o.ok.unified_id
+ << ", migrated " << cached_last_o.n_outputs_read << " outputs already");
+ found_cached_output = true;
+
+ // Set k and v so we can continue the migration from that output
+ k = {sizeof(cached_last_o.amount), (void *)&cached_last_o.amount};
+
+ const std::size_t outkey_size = (cached_last_o.amount == 0) ? sizeof(outkey) : sizeof(pre_rct_outkey);
+ v = {outkey_size, (void *)&cached_last_o.ok};
+
+ i = cached_last_o.n_outputs_read;
+ op = MDB_NEXT;
+ }
+ }
+
+ // Advance the output_amounts cursor to the last output read
+ if (i || found_cached_output)
+ {
+ result = mdb_cursor_get(c_output_amounts, &k, &v, MDB_GET_BOTH);
+ if (result)
+ throw0(DB_ERROR(lmdb_error("Failed to advance cursor for output amounts: ", result).c_str()));
+ }
+ }
+
+ // Get the next output from the db
+ result = mdb_cursor_get(c_output_amounts, &k, &v, op);
+ op = MDB_NEXT;
+ if (result == MDB_NOTFOUND)
+ {
+ // Indicate we've read all outputs so we know the migration step is complete
+ last_output.n_outputs_read = n_outputs;
+ MDB_val_set(v_last_output, last_output);
+ result = mdb_cursor_put(c_tmp_last_output, (MDB_val*)&zerokval, &v_last_output, 0);
+ if (result)
+ throw0(DB_ERROR(lmdb_error("Failed to update max output id: ", result).c_str()));
+
+ if (progress_width)
+ std::cout << '\r' << std::string(progress_width, ' ') << '\r' << std::flush;
+ //std::cout << " 150000000 / 150000000 outputs (100% of step 1/2)\r" << std::flush; // just showing that chars are erased effectively
+ batch_stop();
+ break;
+ }
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to get a record from output amounts: ", result).c_str()));
+
+ ++i;
+ const bool commit_next_iter = i && !(i % BATCH_SIZE);
+
+ // Read the output data
+ uint64_t amount = *(const uint64_t*)k.mv_data;
+ output_data_t output_data;
+ uint64_t unified_id;
+ if (amount == 0)
+ {
+ const outkey *okp = (const outkey *)v.mv_data;
+ output_data = okp->data;
+ unified_id = okp->unified_id;
+ if (commit_next_iter)
+ memcpy(&last_output.ok, okp, sizeof(outkey));
+ }
+ else
+ {
+ const pre_rct_outkey *okp = (const pre_rct_outkey *)v.mv_data;
+ memcpy(&output_data, &okp->data, sizeof(pre_rct_output_data_t));
+ output_data.commitment = rct::zeroCommitVartime(amount);
+ unified_id = okp->unified_id;
+ if (commit_next_iter)
+ memcpy(&last_output.ok, okp, sizeof(pre_rct_outkey));
+ }
+
+ if (commit_next_iter)
+ {
+ // Set last output metadata
+ last_output.amount = amount;
+ last_output.n_outputs_read = i;
+ }
+
+ // Prepare the output for insertion to the tree (all outputs in the db at this point must be legacy)
+ fcmp_pp::LegacyOutputPair output_pair{{
+ output_data.pubkey,
+ rct::rct2pt(output_data.commitment),
+ }};
+
+ const fcmp_pp::UnifiedOutput unified_output{
+ .unified_id = unified_id,
+ .output_pair = std::move(output_pair)
+ };
+ const cryptonote::blobdata output_blob = cryptonote::t_serializable_object_to_blob(unified_output);
+
+ // Check if the output is a coinbase output
+ bool is_coinbase = false;
+ const bool has_coinbase_unlock_time = output_data.unlock_time == (output_data.height + CRYPTONOTE_MINED_MONEY_UNLOCK_WINDOW);
+ if (has_coinbase_unlock_time)
+ {
+ // Only coinbase outputs could potentially have the coinbase unlock_time (see prevalidate_miner_transaction)
+ auto toi = this->get_output_tx_and_index_from_unified(unified_id);
+ auto tx = this->get_pruned_tx(toi.first);
+ is_coinbase = tx.is_coinbase();
+ }
+
+ // Get the output's last locked block
+ bool is_custom_timelocked = false;
+ const uint64_t last_locked_block = cryptonote::get_last_locked_block_index_for_tree(is_coinbase,
+ output_data.unlock_time,
+ output_data.height,
+ is_custom_timelocked);
+
+ // Add the output to the locked outputs table
+ MDB_val_set(k_block_id, last_locked_block);
+ MDB_val_sized(v_output, output_blob);
+
+ // MDB_NODUPDATA because all output id's should be unique
+ // Can't use MDB_APPENDDUP because outputs aren't inserted in order sorted by unified_id
+ result = mdb_cursor_put(c_locked_outputs, &k_block_id, &v_output, MDB_NODUPDATA);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to add locked output: ", result).c_str()));
+
+ // Add custom timelocked outputs to the timelocked outputs table
+ if (!is_custom_timelocked)
+ continue;
+
+ MDB_val_set(k_timelocked_block_id, last_locked_block);
+ MDB_val_sized(v_timelocked_output, output_blob);
+
+ // MDB_NODUPDATA because all output id's should be unique
+ // Can't use MDB_APPENDDUP because outputs aren't inserted in order sorted by unified_id
+ result = mdb_cursor_put(c_timelocked_outputs, &k_timelocked_block_id, &v_timelocked_output, MDB_NODUPDATA);
+ if (result != MDB_SUCCESS)
+ throw0(DB_ERROR(lmdb_error("Failed to add timelocked output: ", result).c_str()));
+ }
+ }
+
+ // 2. Set up the curve trees merkle tree by growing the tree block by block,
+ // with leaves that are spendable in each respective block
+ {
+ MGINFO("Setting up a merkle tree using existing cryptonote outputs (step 2/2 of the full-chain membership proof migration)");
+
+ if (!m_batch_transactions)
+ set_batch_transactions(true);
+ const std::size_t BATCH_SIZE = 50;
+ batch_start();
+ txn.m_txn = m_write_txn->m_txn;
+
+ MDB_cursor *c_locked_outputs;
+
+ i = 0;
+ const uint64_t n_blocks = height();
+ std::size_t progress_width = 0;
+ while (i < n_blocks)
+ {
+ if (!(i % BATCH_SIZE))
+ {
+ if (i)
+ {
+ LOGIF(el::Level::Info)
+ {
+ const uint64_t percent = std::min((i * 100) / n_blocks, (uint64_t)99);
+ const std::string progress_line = std::to_string(i) + " / " + std::to_string(n_blocks) +
+ " blocks (" + std::to_string(percent) + "% of step 2/2)";
+ std::cout << '\r' << std::string(progress_width, ' ') << '\r' << progress_line << std::flush;
+ progress_width = std::max(progress_width, progress_line.size());
+ }
+
+ batch_stop();
+ batch_start();
+ txn.m_txn = m_write_txn->m_txn;
+ }
+
+ // Open all cursors
+ result = mdb_cursor_open(txn, m_locked_outputs, &c_locked_outputs);
+ if (result)
+ throw0(DB_ERROR(lmdb_error("Failed to open a cursor for locked outputs: ", result).c_str()));
+
+ // See what the last block inserted into the new table was
+ if (i == 0)
+ {
+ MDB_stat db_stats;
+ result = mdb_stat(txn, m_tree_meta, &db_stats);
+ if (result)
+ throw0(DB_ERROR(lmdb_error("Failed to query m_tree_meta: ", result).c_str()));
+ const uint64_t n_tree_blocks = db_stats.ms_entries;
+ const uint64_t tree_block_idx = n_tree_blocks - std::min<uint64_t>(1, n_tree_blocks);
+ const uint64_t last_added_block_idx = tree_block_idx - std::min<uint64_t>((CRYPTONOTE_DEFAULT_TX_SPENDABLE_AGE - 1), tree_block_idx);
+ CHECK_AND_ASSERT_THROW_MES(n_tree_blocks == 0 || cryptonote::get_default_last_locked_block_index(last_added_block_idx) == tree_block_idx,
+ "Unexpected tree block idx mismatch to last added block idx");
+ if (n_tree_blocks != 0)
+ {
+ MINFO("Continuing from last added block " << last_added_block_idx);
+ i = last_added_block_idx + 1;
+ }
+ if (i == n_blocks)
+ break;
+ }
+ }
+
+ this->advance_tree(i, {});
+
+ LOGIF(el::Level::Info)
+ {
+ if ((i % 1000) == 0)
+ {
+ const uint64_t n_leaf_tuples = this->get_block_n_leaf_tuples(i);
+ crypto::ec_point tree_root;
+ this->get_tree_root_at_blk_idx(i, tree_root);
+ const std::string tree_root_str = epee::string_tools::pod_to_hex(tree_root);
+ MINFO("Block: " << i << ", tree root: " << tree_root_str << ", leaves: " << n_leaf_tuples);
+ }
+ }
+
+ ++i;
+ if (i == n_blocks)
+ if (progress_width)
+ std::cout << '\r' << std::string(progress_width, ' ') << '\r' << std::flush;
+ }
+ batch_stop();
+ }
+ } while(0);
+
+ // Migration complete, now bump db version to 7
+ update_db_version(DB_VERSION_ONCE_COMPLETE);
+
+ // We only needed the temp last output table for this migration, drop it
+ result = mdb_drop(txn, m_tmp_last_output, 1);
+ if (result)
+ throw0(DB_ERROR(lmdb_error("Failed to drop temp last output table: ", result).c_str()));
+
+ txn.commit();
+}
+
void BlockchainLMDB::migrate(const uint32_t oldversion)
{
if (oldversion < 1)
@@ -5812,6 +7017,10 @@ void BlockchainLMDB::migrate(const uint32_t oldversion)
migrate_3_4();
if (oldversion < 5)
migrate_4_5();
+#ifdef ENABLE_FCMP_INTEGRATION
+ if (oldversion < 7)
+ migrate_5_7();
+#endif
}
} // namespace cryptonote
### src/blockchain_db/lmdb/db_lmdb.h
@@ -30,6 +30,7 @@
#include "blockchain_db/blockchain_db.h"
#include "cryptonote_basic/blobdatatype.h" // for type blobdata
+#include "fcmp_pp/fcmp_pp_types.h"
#include "ringct/rctTypes.h"
#include <boost/thread/tss.hpp>
@@ -64,6 +65,13 @@ typedef struct mdb_txn_cursors
MDB_cursor *m_txc_spent_keys;
+ MDB_cursor *m_txc_locked_outputs;
+ MDB_cursor *m_txc_timelocked_outputs;
+ MDB_cursor *m_txc_leaves;
+ MDB_cursor *m_txc_layers;
+ MDB_cursor *m_txc_tree_edges;
+ MDB_cursor *m_txc_tree_meta;
+
MDB_cursor *m_txc_txpool_meta;
MDB_cursor *m_txc_txpool_blob;
@@ -87,6 +95,12 @@ typedef struct mdb_txn_cursors
#define m_cur_tx_indices m_cursors->m_txc_tx_indices
#define m_cur_tx_outputs m_cursors->m_txc_tx_outputs
#define m_cur_spent_keys m_cursors->m_txc_spent_keys
+#define m_cur_locked_outputs m_cursors->m_txc_locked_outputs
+#define m_cur_timelocked_outputs m_cursors->m_txc_timelocked_outputs
+#define m_cur_leaves m_cursors->m_txc_leaves
+#define m_cur_layers m_cursors->m_txc_layers
+#define m_cur_tree_edges m_cursors->m_txc_tree_edges
+#define m_cur_tree_meta m_cursors->m_txc_tree_meta
#define m_cur_txpool_meta m_cursors->m_txc_txpool_meta
#define m_cur_txpool_blob m_cursors->m_txc_txpool_blob
#define m_cur_alt_blocks m_cursors->m_txc_alt_blocks
@@ -109,6 +123,12 @@ typedef struct mdb_rflags
bool m_rf_tx_indices;
bool m_rf_tx_outputs;
bool m_rf_spent_keys;
+ bool m_rf_locked_outputs;
+ bool m_rf_timelocked_outputs;
+ bool m_rf_leaves;
+ bool m_rf_layers;
+ bool m_rf_tree_edges;
+ bool m_rf_tree_meta;
bool m_rf_txpool_meta;
bool m_rf_txpool_blob;
bool m_rf_alt_blocks;
@@ -184,7 +204,7 @@ struct mdb_txn_safe
class BlockchainLMDB final: public BlockchainDB
{
public:
- BlockchainLMDB(bool batch_transactions=true);
+ BlockchainLMDB(bool batch_transactions=true, std::shared_ptr<fcmp_pp::curve_trees::CurveTreesV1> curve_trees = fcmp_pp::curve_trees::curve_trees_v1());
~BlockchainLMDB();
void open(const std::string& filename, const int mdb_flags=0) override;
@@ -267,13 +287,15 @@ class BlockchainLMDB final: public BlockchainDB
uint64_t get_tx_block_height(const crypto::hash& h) const override;
+ uint64_t total_outputs() const;
+
uint64_t get_num_outputs(const uint64_t& amount) const override;
output_data_t get_output_key(const uint64_t& amount, const uint64_t& index, bool include_commitmemt) const override;
void get_output_key(const epee::span<const uint64_t> &amounts, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs, bool allow_partial = false) const override;
- tx_out_index get_output_tx_and_index_from_global(const uint64_t& index) const override;
- void get_output_tx_and_index_from_global(const std::vector<uint64_t> &global_indices,
+ tx_out_index get_output_tx_and_index_from_unified(const uint64_t& index) const override;
+ void get_output_tx_and_index_from_unified(const std::vector<uint64_t> &global_indices,
std::vector<tx_out_index> &tx_out_indices) const;
tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) const override;
@@ -326,6 +348,7 @@ class BlockchainLMDB final: public BlockchainDB
, const difficulty_type& cumulative_difficulty
, const uint64_t& coins_generated
, const std::vector<std::pair<transaction, blobdata>>& txs
+ , const std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments
) override;
void set_batch_transactions(bool batch_transactions) override;
@@ -363,6 +386,7 @@ class BlockchainLMDB final: public BlockchainDB
// helper functions
static int compare_uint64(const MDB_val *a, const MDB_val *b);
+ static int compare_little_endian_uint64(const MDB_val *a, const MDB_val *b);
static int compare_hash32(const MDB_val *a, const MDB_val *b);
static int compare_string(const MDB_val *a, const MDB_val *b);
@@ -401,15 +425,52 @@ class BlockchainLMDB final: public BlockchainDB
void remove_tx_outputs(const uint64_t tx_id, const transaction& tx);
- void remove_output(const uint64_t amount, const uint64_t& out_index);
+ void remove_output(const bool is_coinbase, const uint64_t amount, const uint64_t& out_index);
void prune_outputs(uint64_t amount) override;
void add_spent_key(const crypto::key_image& k_image) override;
void remove_spent_key(const crypto::key_image& k_image) override;
- uint64_t num_outputs() const;
+ //
+ // Curve tree related db calls (private)
+ //
+
+ std::vector<crypto::ec_point> grow_with_tree_extension(const fcmp_pp::CompressedTreeExtension &tree_extension) override;
+
+ crypto::ec_point grow_layer(const fcmp_pp::CompressedLayerExtension &layer_extension, const uint64_t layer_idx);
+
+ uint64_t trim_leaves(const uint64_t new_n_leaf_tuples, const uint64_t trim_block_idx) override;
+
+ void trim_layers(const uint64_t new_n_leaf_tuples,
+ const std::vector<uint64_t> &new_n_elems_per_layer,
+ const std::vector<crypto::ec_point> &new_tree_edge,
+ const uint64_t new_root_layer_idx) override;
+
+ void trim_layer(const uint64_t new_n_elems_in_layer, const uint64_t layer_idx);
+
+ void add_locked_outs(const fcmp_pp::OutsByLastLockedBlock& outs_by_last_locked_block, const std::unordered_map<uint64_t/*unified_id*/, uint64_t/*last locked block_id*/>& timelocked_outputs) override;
+
+ std::vector<fcmp_pp::UnifiedOutput> get_outs_at_last_locked_block_idx(uint64_t block_id) const override;
+
+ void del_locked_outs_at_block_idx(uint64_t block_idx) override;
+
+ uint64_t get_n_leaf_tuples() const override;
+
+ uint64_t get_block_n_leaf_tuples(const uint64_t block_idx) const override;
+
+ uint8_t get_tree_root_at_blk_idx(const uint64_t blk_idx, crypto::ec_point &tree_root_out) const override;
+
+ uint64_t get_tree_block_idx() const override;
+
+ std::vector<crypto::ec_point> get_tree_edge(uint64_t block_id) const override;
+
+ void save_tree_meta(const uint64_t block_idx, const uint64_t n_leaf_tuples, const std::vector<crypto::ec_point> &tree_edge) override;
+
+ void del_tree_meta(const uint64_t block_idx) override;
+
+ std::vector<fcmp_pp::UnifiedOutput> get_unified_output_by_id(const std::vector<uint64_t> &unified_ids) const;
// Hard fork
void set_hard_fork_version(uint64_t height, uint8_t version) override;
@@ -448,6 +509,9 @@ class BlockchainLMDB final: public BlockchainDB
// migrate from DB version 4 to 5
void migrate_4_5();
+ // migrate from DB version 5 to 7
+ void migrate_5_7();
+
void cleanup_batch();
private:
@@ -470,6 +534,13 @@ class BlockchainLMDB final: public BlockchainDB
MDB_dbi m_spent_keys;
+ MDB_dbi m_locked_outputs;
+ MDB_dbi m_timelocked_outputs;
+ MDB_dbi m_leaves;
+ MDB_dbi m_layers;
+ MDB_dbi m_tree_edges;
+ MDB_dbi m_tree_meta;
+
MDB_dbi m_txpool_meta;
MDB_dbi m_txpool_blob;
### src/blockchain_db/testdb.h
@@ -35,6 +35,7 @@
#include <map>
#include <cryptonote_basic/cryptonote_format_utils.h>
+#include "fcmp_pp/fcmp_pp_types.h"
#include "blockchain_db.h"
@@ -101,10 +102,11 @@ class BaseTestDB: public cryptonote::BlockchainDB {
virtual uint64_t get_tx_count() const override { return 0; }
virtual std::vector<cryptonote::transaction> get_tx_list(const std::vector<crypto::hash>& hlist) const override { return std::vector<cryptonote::transaction>(); }
virtual uint64_t get_tx_block_height(const crypto::hash& h) const override { return 0; }
+ virtual uint64_t total_outputs() const override { return 1; }
virtual uint64_t get_num_outputs(const uint64_t& amount) const override { return 1; }
virtual uint64_t get_indexing_base() const override { return 0; }
virtual cryptonote::output_data_t get_output_key(const uint64_t& amount, const uint64_t& index, bool include_commitmemt) const override { return cryptonote::output_data_t(); }
- virtual cryptonote::tx_out_index get_output_tx_and_index_from_global(const uint64_t& index) const override { return cryptonote::tx_out_index(); }
+ virtual cryptonote::tx_out_index get_output_tx_and_index_from_unified(const uint64_t& index) const override { return cryptonote::tx_out_index(); }
virtual cryptonote::tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) const override { return cryptonote::tx_out_index(); }
virtual void get_output_tx_and_index(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<cryptonote::tx_out_index> &indices) const override {}
virtual void get_output_key(const epee::span<const uint64_t> &amounts, const std::vector<uint64_t> &offsets, std::vector<cryptonote::output_data_t> &outputs, bool allow_partial = false) const override {}
@@ -119,6 +121,20 @@ class BaseTestDB: public cryptonote::BlockchainDB {
virtual void add_spent_key(const crypto::key_image& k_image) override {}
virtual void remove_spent_key(const crypto::key_image& k_image) override {}
+ virtual std::vector<crypto::ec_point> grow_with_tree_extension(const fcmp_pp::CompressedTreeExtension &tree_extension) override { return std::vector<crypto::ec_point>{}; };
+ virtual uint64_t trim_leaves(const uint64_t new_n_leaf_tuples, const uint64_t trim_block_idx) override { return 0; };
+ virtual void trim_layers(const uint64_t new_n_leaf_tuples, const std::vector<uint64_t> &new_n_elems_per_layer, const std::vector<crypto::ec_point> &new_tree_edge, const uint64_t new_root_layer_idx) override {};
+ virtual void add_locked_outs(const fcmp_pp::OutsByLastLockedBlock& outs_by_last_locked_block, const std::unordered_map<uint64_t/*unified_id*/, uint64_t/*last locked block_id*/>& timelocked_outputs) override {};
+ virtual std::vector<fcmp_pp::UnifiedOutput> get_outs_at_last_locked_block_idx(uint64_t block_id) const override { return std::vector<fcmp_pp::UnifiedOutput>{}; };
+ virtual void del_locked_outs_at_block_idx(uint64_t block_idx) override {};
+ virtual uint64_t get_n_leaf_tuples() const override { return 0; };
+ virtual uint64_t get_block_n_leaf_tuples(const uint64_t block_idx) const override { return 0; };
+ virtual uint8_t get_tree_root_at_blk_idx(const uint64_t blk_idx, crypto::ec_point &tree_root_out) const override { return {}; };
+ virtual uint64_t get_tree_block_idx() const override { return 0; };
+ virtual std::vector<crypto::ec_point> get_tree_edge(uint64_t block_id) const override { return {}; };
+ virtual void save_tree_meta(const uint64_t block_idx, const uint64_t n_leaf_tuples, const std::vector<crypto::ec_point> &tree_edge) override {};
+ virtual void del_tree_meta(const uint64_t block_idx) override {};
+
virtual bool for_all_key_images(std::function<bool(const crypto::key_image&)>) const override { return true; }
virtual bool for_blocks_range(const uint64_t&, const uint64_t&, std::function<bool(uint64_t, const crypto::hash&, const cryptonote::block&)>) const override { return true; }
virtual bool for_all_block_info(uint64_t, uint64_t, const std::function<bool(uint64_t, const crypto::hash&, uint64_t)> &f) const override { return true; }
@@ -146,7 +162,8 @@ class BaseTestDB: public cryptonote::BlockchainDB {
, const cryptonote::difficulty_type& cumulative_difficulty
, const uint64_t& coins_generated
, const std::vector<std::pair<cryptonote::transaction, cryptonote::blobdata>>& txs
- ) override { return cryptonote::BlockchainDB::add_block(blk, block_weight, long_term_block_weight, cumulative_difficulty, coins_generated, txs); }
+ , const std::unordered_map<uint64_t, rct::key>& transparent_amount_commitments
+ ) override { return cryptonote::BlockchainDB::add_block(blk, block_weight, long_term_block_weight, cumulative_difficulty, coins_generated, txs, transparent_amount_commitments); }
virtual void add_block( const cryptonote::block& blk
, size_t block_weight
, uint64_t long_term_block_weight
### src/blockchain_utilities/blockchain_import.cpp
@@ -42,6 +42,7 @@
#include "serialization/binary_utils.h" // dump_binary(), parse_binary()
#include "include_base_utils.h"
#include "cryptonote_core/cryptonote_core.h"
+#include "cryptonote_core/tx_verification_utils.h"
#include "cryptonote_protocol/block_entry_utils.h"
#undef MONERO_DEFAULT_LOG_CATEGORY
@@ -488,10 +489,13 @@ int import_from_file(cryptonote::core& core, const std::string& import_file_path
cumulative_difficulty = bp.cumulative_difficulty;
coins_generated = bp.coins_generated;
+ std::unordered_map<uint64_t, rct::key> transparent_amount_commitments;
+ collect_transparent_amount_commitments(b.miner_tx, txs, transparent_amount_commitments);
+
try
{
uint64_t long_term_block_weight = core.get_blockchain_storage().get_next_long_term_block_weight(block_weight);
- core.get_blockchain_storage().get_db().add_block(std::make_pair(b, block_to_blob(b)), block_weight, long_term_block_weight, cumulative_difficulty, coins_generated, txs);
+ core.get_blockchain_storage().get_db().add_block(std::make_pair(b, block_to_blob(b)), block_weight, long_term_block_weight, cumulative_difficulty, coins_generated, txs, transparent_amount_commitments);
}
catch (const std::exception& e)
{
### src/blockchain_utilities/blockchain_prune.cpp
@@ -53,7 +53,7 @@ static std::string db_path;
static uint64_t records_per_sync = 16 * 65536;
static const size_t slack = 512 * 1024 * 1024;
-static constexpr uint32_t MAX_SUPPORTED_DB_VERSION = 5;
+static constexpr uint32_t MAX_SUPPORTED_DB_VERSION = 7;
static std::vector<bool> is_v1;
@@ -704,6 +704,14 @@ int main(int argc, char* argv[])
copy_table(env0, env1, "tx_outputs", MDB_INTEGERKEY, 0);
copy_table(env0, env1, "output_txs", MDB_INTEGERKEY | MDB_DUPSORT | MDB_DUPFIXED, 0, BlockchainLMDB::compare_uint64);
copy_table(env0, env1, "output_amounts", MDB_INTEGERKEY | MDB_DUPSORT | MDB_DUPFIXED, 0, BlockchainLMDB::compare_uint64);
+#ifdef ENABLE_FCMP_INTEGRATION
+ copy_table(env0, env1, "locked_outputs", MDB_INTEGERKEY | MDB_DUPSORT | MDB_DUPFIXED, 0, BlockchainLMDB::compare_little_endian_uint64);
+ copy_table(env0, env1, "timelocked_outputs", MDB_INTEGERKEY | MDB_DUPSORT | MDB_DUPFIXED, 0, BlockchainLMDB::compare_little_endian_uint64);
+ copy_table(env0, env1, "leaves", MDB_INTEGERKEY | MDB_DUPSORT | MDB_DUPFIXED, 0, BlockchainLMDB::compare_uint64);
+ copy_table(env0, env1, "layers", MDB_INTEGERKEY | MDB_DUPSORT | MDB_DUPFIXED, 0, BlockchainLMDB::compare_uint64);
+ copy_table(env0, env1, "tree_edges", MDB_INTEGERKEY, 0, BlockchainLMDB::compare_uint64);
+ copy_table(env0, env1, "tree_meta", MDB_INTEGERKEY, 0, BlockchainLMDB::compare_uint64);
+#endif
copy_table(env0, env1, "spent_keys", MDB_INTEGERKEY | MDB_DUPSORT | MDB_DUPFIXED, 0, BlockchainLMDB::compare_hash32);
copy_table(env0, env1, "txpool_meta", 0, 0, BlockchainLMDB::compare_hash32);
copy_table(env0, env1, "txpool_blob", 0, 0, BlockchainLMDB::compare_hash32);
### src/cryptonote_basic/cryptonote_basic.h
@@ -72,9 +72,22 @@ namespace cryptonote
END_SERIALIZE()
};
- struct txout_to_scripthash
+ struct txout_to_carrot_v1
{
- crypto::hash hash;
+ crypto::public_key key; // K_o
+ // TODO: the rest of Carrot
+ // carrot::view_tag_t view_tag; // vt
+ // carrot::encrypted_janus_anchor_t encrypted_janus_anchor; // anchor_enc
+
+ // // Encrypted amount a_enc and amount commitment C_a are stored in rct::rctSigBase
+ // // This allows for reuse of this output type between coinbase and non-coinbase txs
+
+ BEGIN_SERIALIZE_OBJECT()
+ FIELD(key)
+ // TODO: the rest of Carrot
+ // FIELD(view_tag)
+ // FIELD(encrypted_janus_anchor)
+ END_SERIALIZE()
};
// outputs <= HF_VERSION_VIEW_TAGS
@@ -154,7 +167,7 @@ namespace cryptonote
typedef boost::variant<txin_gen, txin_to_script, txin_to_scripthash, txin_to_key> txin_v;
- typedef boost::variant<txout_to_script, txout_to_scripthash, txout_to_key, txout_to_tagged_key> txout_target_v;
+ typedef boost::variant<txout_to_script, txout_to_carrot_v1, txout_to_key, txout_to_tagged_key> txout_target_v;
struct tx_out
{
@@ -169,6 +182,22 @@ namespace cryptonote
};
+ inline const crypto::public_key &output_pubkey_cref(const txout_target_v &tx_out)
+ {
+ struct tx_out_visitor
+ {
+ const crypto::public_key &operator()(const cryptonote::txout_to_script&) const
+ { throw std::runtime_error("Unexpected usage of txout to script"); }
+ const crypto::public_key &operator()(const cryptonote::txout_to_carrot_v1 &out) const
+ { return out.key; }
+ const crypto::public_key &operator()(const cryptonote::txout_to_tagged_key &out) const
+ { return out.key; }
+ const crypto::public_key &operator()(const cryptonote::txout_to_key &out) const
+ { return out.key; }
+ };
+ return boost::apply_visitor(tx_out_visitor{}, tx_out);
+ }
+
class transaction_prefix
{
@@ -729,14 +758,13 @@ namespace std {
}
BLOB_SERIALIZER(cryptonote::txout_to_key);
-BLOB_SERIALIZER(cryptonote::txout_to_scripthash);
VARIANT_TAG(binary_archive, cryptonote::txin_gen, 0xff);
VARIANT_TAG(binary_archive, cryptonote::txin_to_script, 0x0);
VARIANT_TAG(binary_archive, cryptonote::txin_to_scripthash, 0x1);
VARIANT_TAG(binary_archive, cryptonote::txin_to_key, 0x2);
VARIANT_TAG(binary_archive, cryptonote::txout_to_script, 0x0);
-VARIANT_TAG(binary_archive, cryptonote::txout_to_scripthash, 0x1);
+VARIANT_TAG(binary_archive, cryptonote::txout_to_carrot_v1, 0x1);
VARIANT_TAG(binary_archive, cryptonote::txout_to_key, 0x2);
VARIANT_TAG(binary_archive, cryptonote::txout_to_tagged_key, 0x3);
VARIANT_TAG(binary_archive, cryptonote::transaction, 0xcc);
@@ -747,7 +775,7 @@ VARIANT_TAG(json_archive, cryptonote::txin_to_script, "script");
VARIANT_TAG(json_archive, cryptonote::txin_to_scripthash, "scripthash");
VARIANT_TAG(json_archive, cryptonote::txin_to_key, "key");
VARIANT_TAG(json_archive, cryptonote::txout_to_script, "script");
-VARIANT_TAG(json_archive, cryptonote::txout_to_scripthash, "scripthash");
+VARIANT_TAG(json_archive, cryptonote::txout_to_carrot_v1, "carrot_v1");
VARIANT_TAG(json_archive, cryptonote::txout_to_key, "key");
VARIANT_TAG(json_archive, cryptonote::txout_to_tagged_key, "tagged_key");
VARIANT_TAG(json_archive, cryptonote::transaction, "tx");
@@ -758,7 +786,7 @@ VARIANT_TAG(debug_archive, cryptonote::txin_to_script, "script");
VARIANT_TAG(debug_archive, cryptonote::txin_to_scripthash, "scripthash");
VARIANT_TAG(debug_archive, cryptonote::txin_to_key, "key");
VARIANT_TAG(debug_archive, cryptonote::txout_to_script, "script");
-VARIANT_TAG(debug_archive, cryptonote::txout_to_scripthash, "scripthash");
+VARIANT_TAG(debug_archive, cryptonote::txout_to_carrot_v1, "carrot_v1");
VARIANT_TAG(debug_archive, cryptonote::txout_to_key, "key");
VARIANT_TAG(debug_archive, cryptonote::txout_to_tagged_key, "tagged_key");
VARIANT_TAG(debug_archive, cryptonote::transaction, "tx");
### src/cryptonote_basic/cryptonote_boost_serialization.h
@@ -114,9 +114,12 @@ namespace boost
}
template <class Archive>
- inline void serialize(Archive &a, cryptonote::txout_to_scripthash &x, const boost::serialization::version_type ver)
+ inline void serialize(Archive &a, cryptonote::txout_to_carrot_v1 &x, const boost::serialization::version_type ver)
{
- a & x.hash;
+ a & x.key;
+ // TODO: the rest of Carrot
+ // a & x.view_tag.bytes;
+ // a & x.encrypted_janus_anchor.bytes;
}
template <class Archive>
### src/cryptonote_basic/cryptonote_format_utils.cpp
@@ -1083,6 +1083,33 @@ namespace cryptonote
: boost::optional<crypto::view_tag>();
}
//---------------------------------------------------------------
+ bool commitment_is_in_rct_signatures(const transaction_prefix& tx)
+ {
+ return tx.version >= 2 && !tx.is_coinbase();
+ }
+ //---------------------------------------------------------------
+ bool get_commitment(const transaction& tx, const std::size_t o_idx, const std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments, rct::key &c_out)
+ {
+ static_assert(CURRENT_TRANSACTION_VERSION == 2, "This section of code was written with 2 tx versions in mind. "
+ "Revisit this section and update for the new tx version.");
+ CHECK_AND_ASSERT_THROW_MES(tx.version == 1 || tx.version == 2, "encountered unexpected tx version");
+
+ if (commitment_is_in_rct_signatures(tx))
+ {
+ CHECK_AND_ASSERT_MES(tx.rct_signatures.outPk.size() > o_idx, false, "get_commitment: o_idx must be < tx.rct_signatures.outPk.size()");
+ c_out = tx.rct_signatures.outPk.at(o_idx).mask;
+ return true;
+ }
+
+ // tx version 1 OR miner tx
+ // return the pre-calculated transparent amount commitment
+ CHECK_AND_ASSERT_MES(tx.vout.size() > o_idx, false, "get_commitment: o_idx must be < tx.vout.size()");
+ const auto it = transparent_amount_commitments.find(tx.vout.at(o_idx).amount);
+ CHECK_AND_ASSERT_MES(it != transparent_amount_commitments.end(), false, "get_commitment: transparent amount commitment missing");
+ c_out = it->second;
+ return true;
+ }
+ //---------------------------------------------------------------
//---------------------------------------------------------------
void set_tx_out(const uint64_t amount, const crypto::public_key& output_public_key, const bool use_view_tags, const crypto::view_tag& view_tag, tx_out& out)
{
@@ -1841,4 +1868,218 @@ namespace cryptonote
sc_sub((unsigned char*)key.data, (const unsigned char*)key.data, (const unsigned char*)hash.data);
return key;
}
+ //---------------------------------------------------------------
+ uint64_t get_default_last_locked_block_index(const uint64_t block_included_in_chain)
+ {
+ static_assert(CRYPTONOTE_DEFAULT_TX_SPENDABLE_AGE > 0, "unexpected default spendable age");
+ return block_included_in_chain + (CRYPTONOTE_DEFAULT_TX_SPENDABLE_AGE - 1);
+ }
+ //---------------------------------------------------------------
+ uint64_t get_last_locked_block_index_for_tree(const bool is_coinbase,
+ const uint64_t unlock_time,
+ const uint64_t block_included_in_chain,
+ bool &is_custom_timelocked_out)
+ {
+ uint64_t last_locked_block_index = 0;
+ is_custom_timelocked_out = false;
+
+ const uint64_t default_block_index = get_default_last_locked_block_index(block_included_in_chain);
+
+ /* Custom unlock times were deprecated as of June 1, 2026. Any non-coinbase unlock_time created after is ignored.
+ See: https://www.getmonero.org/2026/05/10/deprecating-unlock-time.html
+
+ Mainnet block 05b31aa5b25ece01c3474d9a6be627c844519b2c237f252d748318bc85c1a89f is the first block with UTC
+ timestamp of June 1, 2026, and is block index 3686382. So we set that as the deprecation block.
+
+ We use a mainnet block index here for all networks even though this block does not correspond to June 1, 2026
+ on other networks because it's simpler to manage (this function doesn't need to be network-aware) and won't
+ have a material consequence on other networks.
+
+ Note: coinbase txs are always expected to have unlock time set to
+ `block_included_in_chain + CRYPTONOTE_MINED_MONEY_UNLOCK_WINDOW` as of this writing.
+ */
+ const uint64_t UNLOCK_TIME_DEPRECATION_BLOCK = 3686382;
+ if (!is_coinbase && block_included_in_chain >= UNLOCK_TIME_DEPRECATION_BLOCK)
+ {
+ return default_block_index;
+ }
+
+ if (unlock_time == 0)
+ {
+ last_locked_block_index = default_block_index;
+ }
+ else if (unlock_time < CRYPTONOTE_MAX_BLOCK_NUMBER)
+ {
+ // The unlock_time in this case is supposed to be the chain height at which the output unlocks.
+ // The chain height is 1 higher than the highest block index, so we subtract 1 for this delta.
+ last_locked_block_index = unlock_time > 0 ? (unlock_time - 1) : 0;
+ }
+ else
+ {
+ /* Here we interpret the unlock_time as time. There are a few things to consider in explaining how we
+ implement this here.
+
+ 1) We use the hf16 block 84785e719b9bcbb763b103bb04b46565cea1c82a4995d984ae30817b8c6ac922 timestamp as an anchor
+ point to determine the expected block last locked block index. We use the timestamp that was actually
+ recorded in the mainnet chain for this so that it is more accurate.
+
+ 2) As of this writing (chain height 3783990), there are no expected unlock times that are materially affected
+ by this function's logic. Either some outputs are going to unlock way in the future (because of some
+ apparent mistake by users or intended burns), or they have already unlocked before this function's logic
+ is expected to be used to unlock outputs. Thus, we don't need to take into account when the block interval
+ was 60s (instead of the current 120s).
+
+ 3) We use the mainnet timestamp for all networks even though it won't be accurate for other networks
+ (testnet, stagenet, etc). We do this because it's simpler to manage (this function doesn't need to be
+ network-aware) and doesn't have a material consequence on the other networks.
+ */
+ const uint64_t HF_V16_HEIGHT = 2689608;
+ const uint64_t HF_V16_TIMESTAMP = 1660505834;
+
+ // Use the last hard fork's time and block combo to convert the time-based timelock into an last locked block
+ if (HF_V16_TIMESTAMP > unlock_time)
+ {
+ const uint64_t seconds_since_unlock = HF_V16_TIMESTAMP - unlock_time;
+ const uint64_t blocks_since_unlock = seconds_since_unlock / DIFFICULTY_TARGET_V2;
+
+ last_locked_block_index = HF_V16_HEIGHT > blocks_since_unlock
+ ? (HF_V16_HEIGHT - blocks_since_unlock)
+ : default_block_index;
+ }
+ else
+ {
+ const uint64_t seconds_until_unlock = unlock_time - HF_V16_TIMESTAMP;
+ const uint64_t blocks_until_unlock = seconds_until_unlock / DIFFICULTY_TARGET_V2;
+ last_locked_block_index = HF_V16_HEIGHT + blocks_until_unlock;
+ }
+
+ /* Note: since this function was introduced for the hf that included fcmp's, it's possible for an output to be
+ spent before it reaches the last_locked_block_index going by the old rules; this is ok. It can't be spent again b/c
+ it'll have a duplicate key image. It's also possible for an output to unlock by old rules, and then re-lock
+ again at the fork. This is also ok, we just need to be sure that the new hf rules use this last_locked_block_index
+ starting at the fork for fcmp's.
+ */
+
+ MDEBUG("unlock time: " << unlock_time << " , last_locked_block_index: " << last_locked_block_index);
+ }
+
+ is_custom_timelocked_out = !is_coinbase && last_locked_block_index > default_block_index;
+
+ // Can't unlock earlier than the default last locked block
+ return std::max(last_locked_block_index, default_block_index);
+ }
+ //---------------------------------------------------------------
+ // Helper function to group outputs by last locked block idx
+ static uint64_t set_tx_outs_by_last_locked_block(const cryptonote::transaction &tx,
+ const std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments,
+ const uint64_t &first_unified_id,
+ const uint64_t block_idx,
+ fcmp_pp::OutsByLastLockedBlock &outs_by_last_locked_block_inout,
+ std::unordered_map<uint64_t/*unified_id*/, uint64_t/*last locked block_id*/> &timelocked_outputs_inout)
+ {
+ bool has_custom_timelock = false;
+ const uint64_t last_locked_block = cryptonote::get_last_locked_block_index_for_tree(tx.is_coinbase(),
+ tx.unlock_time,
+ block_idx,
+ has_custom_timelock);
+
+ for (std::size_t i = 0; i < tx.vout.size(); ++i)
+ {
+ const uint64_t unified_id = first_unified_id + i;
+ const auto &out = tx.vout[i];
+
+ rct::key commitment;
+ CHECK_AND_ASSERT_THROW_MES(cryptonote::get_commitment(tx, i, transparent_amount_commitments, commitment),
+ "failed to get tx commitment");
+
+ const fcmp_pp::UnifiedOutput unified_output{
+ .unified_id = unified_id,
+ .output_pair = cryptonote::to_output_pair(out.target, commitment)
+ };
+
+ if (has_custom_timelock)
+ {
+ timelocked_outputs_inout[unified_id] = last_locked_block;
+ }
+
+ outs_by_last_locked_block_inout[last_locked_block].emplace_back(unified_output);
+ }
+
+ return tx.vout.size();
+ }
+ //---------------------------------------------------------------
+ static OutsByLastLockedBlockMeta get_outs_by_last_locked_block(
+ const std::vector<std::reference_wrapper<const cryptonote::transaction>> &txs,
+ const std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments,
+ const uint64_t first_unified_id,
+ const uint64_t block_idx)
+ {
+ OutsByLastLockedBlockMeta outs;
+ outs.next_unified_id = first_unified_id;
+
+ for (const auto &tx : txs)
+ {
+ outs.next_unified_id += set_tx_outs_by_last_locked_block(
+ tx.get(),
+ transparent_amount_commitments,
+ outs.next_unified_id,
+ block_idx,
+ outs.outs_by_last_locked_block,
+ outs.timelocked_outputs);
+ }
+
+ return outs;
+ }
+ //---------------------------------------------------------------
+ OutsByLastLockedBlockMeta get_outs_by_last_locked_block(
+ const cryptonote::transaction &miner_tx,
+ const std::vector<cryptonote::transaction> &txs,
+ const std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments,
+ const uint64_t first_unified_id,
+ const uint64_t block_idx)
+ {
+ std::vector<std::reference_wrapper<const transaction>> tx_refs;
+ tx_refs.reserve(1 + txs.size());
+ tx_refs.push_back(std::cref(miner_tx));
+ for (const auto &tx : txs)
+ tx_refs.push_back(std::cref(tx));
+ return get_outs_by_last_locked_block(tx_refs, transparent_amount_commitments, first_unified_id, block_idx);
+ }
+
+ OutsByLastLockedBlockMeta get_outs_by_last_locked_block(
+ const cryptonote::transaction &miner_tx,
+ const std::vector<std::pair<transaction, blobdata>> &tx_pairs,
+ const std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments,
+ const uint64_t first_unified_id,
+ const uint64_t block_idx)
+ {
+ std::vector<std::reference_wrapper<const transaction>> tx_refs;
+ tx_refs.reserve(1 + tx_pairs.size());
+ tx_refs.push_back(std::cref(miner_tx));
+ for (const auto &tx : tx_pairs)
+ tx_refs.push_back(std::cref(tx.first));
+ return get_outs_by_last_locked_block(tx_refs, transparent_amount_commitments, first_unified_id, block_idx);
+ }
+ //---------------------------------------------------------------
+ fcmp_pp::OutputPair to_output_pair(const cryptonote::txout_target_v &tx_out, const rct::key &commitment)
+ {
+ struct tx_out_visitor
+ {
+ const crypto::public_key &O;
+ const crypto::ec_point &C;
+
+ fcmp_pp::OutputPair operator()(const cryptonote::txout_to_carrot_v1&) const
+ { return fcmp_pp::CarrotOutputPairV1{{O, C}}; }
+ fcmp_pp::OutputPair operator()(const cryptonote::txout_to_tagged_key&) const
+ { return fcmp_pp::LegacyOutputPair{{O, C}}; }
+ fcmp_pp::OutputPair operator()(const cryptonote::txout_to_key&) const
+ { return fcmp_pp::LegacyOutputPair{{O, C}}; }
+ fcmp_pp::OutputPair operator()(const cryptonote::txout_to_script&) const
+ { throw std::logic_error("cannot convert txout to script into an output pair"); }
+ };
+
+ const crypto::public_key &O = cryptonote::output_pubkey_cref(tx_out);
+ const crypto::ec_point &C = rct::rct2pt(commitment);
+ return boost::apply_visitor(tx_out_visitor{O, C}, tx_out);
+ }
}
### src/cryptonote_basic/cryptonote_format_utils.h
@@ -37,6 +37,7 @@
#include "include_base_utils.h"
#include "crypto/crypto.h"
#include "crypto/hash.h"
+#include "fcmp_pp/fcmp_pp_types.h"
#include <unordered_map>
#include <boost/multiprecision/cpp_int.hpp>
@@ -157,6 +158,8 @@ namespace cryptonote
uint64_t get_outs_money_amount(const transaction& tx);
bool get_output_public_key(const cryptonote::tx_out& out, crypto::public_key& output_public_key);
boost::optional<crypto::view_tag> get_output_view_tag(const cryptonote::tx_out& out);
+ bool commitment_is_in_rct_signatures(const transaction_prefix& tx);
+ bool get_commitment(const transaction& tx, const std::size_t o_idx, const std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments, rct::key &c_out);
bool check_inputs_types_supported(const transaction& tx);
bool check_outs_valid(const transaction& tx);
bool parse_amount(uint64_t& amount, const std::string& str_amount);
@@ -294,6 +297,38 @@ namespace cryptonote
crypto::secret_key encrypt_key(crypto::secret_key key, const epee::wipeable_string &passphrase);
crypto::secret_key decrypt_key(crypto::secret_key key, const epee::wipeable_string &passphrase);
+
+ uint64_t get_default_last_locked_block_index(const uint64_t block_included_in_chain);
+ // Returns the last locked block index for the provided unlock_time using rules applied at the FCMP++ fork.
+ // Warning: this function is ONLY expected to be used to determine when outputs should enter the FCMP tree.
+ uint64_t get_last_locked_block_index_for_tree(const bool is_coinbase,
+ const uint64_t unlock_time,
+ const uint64_t block_included_in_chain,
+ bool &is_custom_timelocked_out);
+
+ struct OutsByLastLockedBlockMeta
+ {
+ fcmp_pp::OutsByLastLockedBlock outs_by_last_locked_block;
+ std::unordered_map<uint64_t/*unified_id*/, uint64_t/*last locked block_id*/> timelocked_outputs;
+ uint64_t next_unified_id;
+ };
+
+ OutsByLastLockedBlockMeta get_outs_by_last_locked_block(
+ const cryptonote::transaction &miner_tx,
+ const std::vector<cryptonote::transaction> &txs,
+ const std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments,
+ const uint64_t first_unified_id,
+ const uint64_t block_idx);
+
+ OutsByLastLockedBlockMeta get_outs_by_last_locked_block(
+ const cryptonote::transaction &miner_tx,
+ const std::vector<std::pair<transaction, blobdata>> &tx_pairs,
+ const std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments,
+ const uint64_t first_unified_id,
+ const uint64_t block_idx);
+
+ fcmp_pp::OutputPair to_output_pair(const cryptonote::txout_target_v &tx_out, const rct::key &commitment);
+
#define CHECKED_GET_SPECIFIC_VARIANT(variant_var, specific_type, variable_name, fail_return_val) \
CHECK_AND_ASSERT_MES(variant_var.type() == typeid(specific_type), fail_return_val, "wrong variant type: " << variant_var.type().name() << ", expected " << typeid(specific_type).name()); \
specific_type& variable_name = boost::get<specific_type>(variant_var);
### src/cryptonote_core/blockchain.cpp
@@ -4266,6 +4266,13 @@ bool Blockchain::handle_block_to_main_chain(const block& bl, const crypto::hash&
cumulative_block_weight = m_blocks_hash_check[blockchain_height].second;
}
+ TIME_MEASURE_START(tac);
+
+ std::unordered_map<uint64_t, rct::key> transparent_amount_commitments;
+ collect_transparent_amount_commitments(bl.miner_tx, txs, transparent_amount_commitments);
+
+ TIME_MEASURE_FINISH(tac);
+
TIME_MEASURE_START(vmt);
uint64_t base_reward = 0;
uint64_t already_generated_coins = blockchain_height ? m_db->get_block_already_generated_coins(blockchain_height - 1) : 0;
@@ -4305,7 +4312,7 @@ bool Blockchain::handle_block_to_main_chain(const block& bl, const crypto::hash&
{
uint64_t long_term_block_weight = get_next_long_term_block_weight(block_weight);
cryptonote::blobdata bd = cryptonote::block_to_blob(bl);
- new_height = m_db->add_block(std::make_pair(bl, std::move(bd)), block_weight, long_term_block_weight, cumulative_difficulty, already_generated_coins, txs);
+ new_height = m_db->add_block(std::make_pair(bl, std::move(bd)), block_weight, long_term_block_weight, cumulative_difficulty, already_generated_coins, txs, transparent_amount_commitments);
}
catch (const KEY_IMAGE_EXISTS& e)
{
@@ -4347,7 +4354,7 @@ bool Blockchain::handle_block_to_main_chain(const block& bl, const crypto::hash&
<< cumulative_block_weight << " p/t: " << block_processing_time << " ("
<< target_calculating_time << "/" << longhash_calculating_time << "/"
<< t1 << "/" << t2 << "/" << t3 << "/" << t_exists << "/" << t_pool
- << "/" << t_checktx << "/" << t_dblspnd << "/" << vmt << "/" << addblock << ")ms");
+ << "/" << t_checktx << "/" << t_dblspnd << "/" << vmt << "/" << tac << "/" << addblock << ")ms");
}
bvc.m_added_to_main_chain = true;
### src/cryptonote_core/tx_verification_utils.cpp
@@ -35,6 +35,8 @@
#include "hardforks/hardforks.h"
#include "ringct/rctSigs.h"
+#include <functional>
+
#undef MONERO_DEFAULT_LOG_CATEGORY
#define MONERO_DEFAULT_LOG_CATEGORY "verify"
@@ -311,11 +313,46 @@ static bool ver_non_input_consensus_templated(TxForwardIt tx_begin, TxForwardIt
return true;
}
+static void collect_transparent_amount_commitments_static(
+ const std::vector<std::reference_wrapper<const transaction>> &txs,
+ std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments_inout)
+{
+ // Note: we do not clear transparent_amount_commitments_inout because it may be a rolling cache
+
+ for (const auto &tx_ref : txs)
+ {
+ const auto &tx = tx_ref.get();
+
+ // We only need commitments for transparent amounts, which are tx version 1 || coinbase txs
+ if (tx.version > 1 && !tx.is_coinbase())
+ continue;
+ for (const auto &tx_out : tx.vout)
+ {
+ const uint64_t amount = tx_out.amount;
+ if (transparent_amount_commitments_inout.find(amount) == transparent_amount_commitments_inout.end())
+ transparent_amount_commitments_inout[amount] = rct::zeroCommitVartime(amount);
+ }
+ }
+}
+
////////////////////////////////////////////////////////////////////////////////////////////////////
namespace cryptonote
{
+void collect_transparent_amount_commitments(
+ const transaction &miner_tx,
+ const std::vector<std::pair<transaction, blobdata>> &tx_pairs,
+ std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments_inout)
+{
+ std::vector<std::reference_wrapper<const transaction>> tx_refs;
+ tx_refs.reserve(1 + tx_pairs.size());
+ tx_refs.push_back(std::cref(miner_tx));
+ for (const auto &tx : tx_pairs)
+ tx_refs.push_back(std::cref(tx.first));
+ collect_transparent_amount_commitments_static(tx_refs, transparent_amount_commitments_inout);
+}
+
uint64_t get_transaction_weight_limit(const uint8_t hf_version)
{
// from v8, limit a tx to 50% of the minimum block weight
### src/cryptonote_core/tx_verification_utils.h
@@ -35,6 +35,11 @@
namespace cryptonote
{
+void collect_transparent_amount_commitments(
+ const transaction &miner_tx,
+ const std::vector<std::pair<transaction, blobdata>> &tx_pairs,
+ std::unordered_map<uint64_t, rct::key> &transparent_amount_commitments_inout);
+
/**
* @brief Get the maximum transaction weight for a given hardfork
*
### src/fcmp_pp/curve_trees.cpp
@@ -1085,5 +1085,88 @@ std::size_t CurveTrees<C1, C2>::n_layers(const uint64_t n_leaf_tuples) const
// Explicit instantiation
template std::size_t CurveTrees<Selene, Helios>::n_layers(const uint64_t n_leaf_tuples) const;
//----------------------------------------------------------------------------------------------------------------------
+template<typename C1, typename C2>
+typename CurveTrees<C1, C2>::LastHashes CurveTrees<C1, C2>::tree_edge_to_last_hashes(
+ const std::vector<crypto::ec_point> &tree_edge) const
+{
+ typename CurveTrees<C1, C2>::LastHashes last_hashes;
+
+ bool parent_is_c1 = true;
+ for (const auto &last_hash : tree_edge)
+ {
+ if (parent_is_c1)
+ last_hashes.c1_last_hashes.push_back(m_c1->from_bytes(last_hash));
+ else
+ last_hashes.c2_last_hashes.push_back(m_c2->from_bytes(last_hash));
+ parent_is_c1 = !parent_is_c1;
+ }
+
+ return last_hashes;
+}
+
+// Explicit instantiation
+template CurveTrees<Selene, Helios>::LastHashes CurveTrees<Selene, Helios>::tree_edge_to_last_hashes(
+ const std::vector<crypto::ec_point> &tree_edge) const;
+//----------------------------------------------------------------------------------------------------------------------
+template<typename C1, typename C2>
+CompressedTreeExtension CurveTrees<C1, C2>::compress_tree_extension(
+ typename CurveTrees<C1, C2>::TreeExtension &&tree_extension) const
+{
+ std::vector<CompressedLayerExtension> layer_extensions;
+ const std::size_t n_layers = tree_extension.c1_layer_extensions.size() + tree_extension.c2_layer_extensions.size();
+ layer_extensions.reserve(n_layers);
+
+ bool parent_is_c1 = true;
+ std::size_t c1_idx = 0, c2_idx = 0;
+ for (std::size_t i = 0; i < n_layers; ++i)
+ {
+ if (parent_is_c1)
+ {
+ const auto &c1_layer_ext = tree_extension.c1_layer_extensions.at(c1_idx);
+
+ std::vector<crypto::ec_point> hashes;
+ hashes.reserve(c1_layer_ext.hashes.size());
+ for (const auto &h : c1_layer_ext.hashes)
+ hashes.emplace_back(m_c1->to_bytes(h));
+
+ layer_extensions.emplace_back(fcmp_pp::CompressedLayerExtension{
+ .start_idx = c1_layer_ext.start_idx,
+ .update_existing_last_hash = c1_layer_ext.update_existing_last_hash,
+ .hashes = std::move(hashes)
+ });
+
+ ++c1_idx;
+ }
+ else
+ {
+ const auto &c2_layer_ext = tree_extension.c2_layer_extensions.at(c2_idx);
+
+ std::vector<crypto::ec_point> hashes;
+ hashes.reserve(c2_layer_ext.hashes.size());
+ for (const auto &h : c2_layer_ext.hashes)
+ hashes.emplace_back(m_c2->to_bytes(h));
+
+ layer_extensions.emplace_back(fcmp_pp::CompressedLayerExtension{
+ .start_idx = c2_layer_ext.start_idx,
+ .update_existing_last_hash = c2_layer_ext.update_existing_last_hash,
+ .hashes = std::move(hashes)
+ });
+
+ ++c2_idx;
+ }
+
+ parent_is_c1 = !parent_is_c1;
+ }
+
+ return CompressedTreeExtension{
+ .leaves = std::move(tree_extension.leaves),
+ .layer_extensions = std::move(layer_extensions)
+ };
+}
+
+// Explicit instantiation
+template CompressedTreeExtension CurveTrees<Selene, Helios>::compress_tree_extension(
+ typename CurveTrees<Selene, Helios>::TreeExtension &&tree_ext) const;
+//----------------------------------------------------------------------------------------------------------------------
} //namespace curve_trees
} //namespace fcmp_pp
### src/fcmp_pp/curve_trees.h
@@ -172,6 +172,11 @@ class CurveTrees
// Doesn't include the leaf layer.
std::size_t n_layers(const uint64_t n_leaf_tuples) const;
+ // Decompress the compressed points from the provided tree edge
+ LastHashes tree_edge_to_last_hashes(const std::vector<crypto::ec_point> &tree_edge_to_last_hashes) const;
+
+ // Compress all the points in the tree extension
+ CompressedTreeExtension compress_tree_extension(TreeExtension &&tree_extension) const;
private:
// Multithreaded helper function to convert valid outputs to leaf tuples ready for insertion to the tree & db
void outputs_to_leaves(std::vector<UnifiedOutput> &&new_outputs,
### src/fcmp_pp/fcmp_pp_rust/fcmp++.h
@@ -135,6 +135,10 @@ void helios_point_to_bytes(const struct HeliosPoint *helios_point, uint8_t bytes
void selene_point_to_bytes(const struct SelenePoint *selene_point, uint8_t bytes_out[32]);
+int helios_point_from_bytes(const uint8_t *helios_point_bytes, struct HeliosPoint *helios_point_out);
+
+int selene_point_from_bytes(const uint8_t *selene_point_bytes, struct SelenePoint *selene_point_out);
+
int hash_grow_helios(struct HeliosPoint existing_hash,
uintptr_t offset,
struct HeliosScalar existing_child_at_offset,
### src/fcmp_pp/fcmp_pp_rust/src/lib.rs
@@ -10,8 +10,8 @@ use ec_divisors::DivisorCurve;
use full_chain_membership_proofs::tree::hash_grow;
use helioselene::{
- Field25519 as SeleneScalar, HeliosPoint, HelioseleneField as HeliosScalar, Selene,
- SelenePoint,
+ Field25519 as SeleneScalar, Helios, HeliosPoint, HelioseleneField as HeliosScalar,
+ Selene, SelenePoint,
};
use monero_fcmp_plus_plus::{
@@ -116,6 +116,8 @@ ec_elem_to_bytes!(helios_point_to_bytes, HeliosPoint, to_bytes);
ec_elem_to_bytes!(selene_point_to_bytes, SelenePoint, to_bytes);
ec_elem_from_bytes!(selene_scalar_from_bytes, SeleneScalar, Selene, read_F);
+ec_elem_from_bytes!(helios_point_from_bytes, HeliosPoint, Helios, read_G);
+ec_elem_from_bytes!(selene_point_from_bytes, SelenePoint, Selene, read_G);
macro_rules! point_to_cycle_scalar {
($fn_name:ident, $Point:ty, $Scalar:ty) => {
### src/fcmp_pp/fcmp_pp_serialization.h
@@ -0,0 +1,61 @@
+// Copyright (c) 2025, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+// conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+// of conditions and the following disclaimer in the documentation and/or other
+// materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+// used to endorse or promote products derived from this software without specific
+// prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#pragma once
+
+#include "fcmp_pp_types.h"
+#include "serialization/serialization.h"
+#include "serialization/binary_archive.h"
+#include "serialization/debug_archive.h"
+#include "serialization/json_archive.h"
+#include "serialization/variant.h"
+
+
+BEGIN_SERIALIZE_OBJECT_FN(fcmp_pp::LegacyOutputPair)
+ FIELD_F(output_pubkey)
+ FIELD_F(commitment)
+END_SERIALIZE()
+
+BEGIN_SERIALIZE_OBJECT_FN(fcmp_pp::CarrotOutputPairV1)
+ FIELD_F(output_pubkey)
+ FIELD_F(commitment)
+END_SERIALIZE()
+
+VARIANT_TAG(binary_archive, fcmp_pp::LegacyOutputPair, 0x70);
+VARIANT_TAG(binary_archive, fcmp_pp::CarrotOutputPairV1, 0x71);
+
+VARIANT_TAG(json_archive, fcmp_pp::LegacyOutputPair, "legacy_out_pair");
+VARIANT_TAG(json_archive, fcmp_pp::CarrotOutputPairV1, "carrot_out_pair_v1");
+
+VARIANT_TAG(debug_archive, fcmp_pp::LegacyOutputPair, "legacy_out_pair");
+VARIANT_TAG(debug_archive, fcmp_pp::CarrotOutputPairV1, "carrot_out_pair_v1");
+
+BEGIN_SERIALIZE_OBJECT_FN(fcmp_pp::UnifiedOutput)
+ FIELD_F(unified_id)
+ FIELD_F(output_pair)
+END_SERIALIZE()
### src/fcmp_pp/fcmp_pp_types.h
@@ -30,6 +30,7 @@
#include <cstring>
#include <type_traits>
+#include <unordered_map>
#include <variant>
#include <vector>
@@ -141,6 +142,10 @@ struct UnifiedOutput final
}
};
+#define SIZEOF_SERIALIZED_UNIFIED_OUTPUT 73 // 8+1+32+32
+
+using OutsByLastLockedBlock = std::unordered_map<uint64_t, std::vector<UnifiedOutput>>;
+
// Contiguous leaves in the tree, starting at a specified start_idx in the leaf layer
struct ContiguousLeaves final
{
@@ -149,6 +154,25 @@ struct ContiguousLeaves final
// Contiguous leaves in a tree that start at the start_idx
std::vector<UnifiedOutput> tuples;
};
+
+/* The "Compressed" prefix means all points contained in the struct are compressed points */
+
+// A layer of contiguous hashes starting from a specific start_idx in the tree
+struct CompressedLayerExtension final
+{
+ uint64_t start_idx{0};
+ bool update_existing_last_hash;
+ std::vector<crypto::ec_point> hashes;
+};
+
+// A struct useful to extend an existing tree
+// - layers alternate between C1 and C2
+// - layer_extensions[0] is C1 first layer after leaves, then layer_extensions[1] is C2, layer_extensions[2] is C1, etc.
+struct CompressedTreeExtension final
+{
+ ContiguousLeaves leaves;
+ std::vector<CompressedLayerExtension> layer_extensions;
+};
//----------------------------------------------------------------------------------------------------------------------
//----------------------------------------------------------------------------------------------------------------------
}//namespace fcmp_pp
### src/fcmp_pp/tower_cycle.cpp
@@ -149,6 +149,22 @@ crypto::ec_point Helios::to_bytes(const Helios::Point &point) const
return res;
}
//----------------------------------------------------------------------------------------------------------------------
+Selene::Point Selene::from_bytes(const crypto::ec_point &bytes) const
+{
+ Selene::Point selene_point;
+ int r = ::selene_point_from_bytes(reinterpret_cast<const uint8_t*>(&bytes), &selene_point);
+ CHECK_FFI_RES;
+ return selene_point;
+}
+//----------------------------------------------------------------------------------------------------------------------
+Helios::Point Helios::from_bytes(const crypto::ec_point &bytes) const
+{
+ Helios::Point helios_point;
+ int r = ::helios_point_from_bytes(reinterpret_cast<const uint8_t*>(&bytes), &helios_point);
+ CHECK_FFI_RES;
+ return helios_point;
+}
+//----------------------------------------------------------------------------------------------------------------------
std::string Selene::to_string(const typename Selene::Scalar &scalar) const
{
return epee::string_tools::pod_to_hex(this->to_bytes(scalar));
### src/fcmp_pp/tower_cycle.h
@@ -63,6 +63,8 @@ class Curve
virtual crypto::ec_scalar to_bytes(const typename C::Scalar &scalar) const = 0;
virtual crypto::ec_point to_bytes(const typename C::Point &point) const = 0;
+ virtual typename C::Point from_bytes(const crypto::ec_point &bytes) const = 0;
+
virtual std::string to_string(const typename C::Scalar &scalar) const = 0;
virtual std::string to_string(const typename C::Point &point) const = 0;
};
@@ -93,6 +95,8 @@ class Selene final : public Curve<SeleneT>
crypto::ec_scalar to_bytes(const Scalar &scalar) const override;
crypto::ec_point to_bytes(const Point &point) const override;
+ Point from_bytes(const crypto::ec_point &bytes) const override;
+
std::string to_string(const Scalar &scalar) const override;
std::string to_string(const Point &point) const override;
};
@@ -123,6 +127,8 @@ class Helios final : public Curve<HeliosT>
crypto::ec_scalar to_bytes(const Scalar &scalar) const override;
crypto::ec_point to_bytes(const Point &point) const override;
+ Point from_bytes(const crypto::ec_point &bytes) const override;
+
std::string to_string(const Scalar &scalar) const override;
std::string to_string(const Point &point) const override;
};
### src/serialization/crypto.h
@@ -86,6 +86,7 @@ BLOB_SERIALIZER(crypto::key_derivation);
BLOB_SERIALIZER(crypto::key_image);
BLOB_SERIALIZER(crypto::signature);
BLOB_SERIALIZER(crypto::view_tag);
+BLOB_SERIALIZER(crypto::ec_point);
VARIANT_TAG(debug_archive, crypto::hash, "hash");
VARIANT_TAG(debug_archive, crypto::hash8, "hash8");
VARIANT_TAG(debug_archive, crypto::public_key, "public_key");
@@ -94,4 +95,4 @@ VARIANT_TAG(debug_archive, crypto::key_derivation, "key_derivation");
VARIANT_TAG(debug_archive, crypto::key_image, "key_image");
VARIANT_TAG(debug_archive, crypto::signature, "signature");
VARIANT_TAG(debug_archive, crypto::view_tag, "view_tag");
-
+VARIANT_TAG(debug_archive, crypto::ec_point, "ec_point");
### src/serialization/json_object.cpp
@@ -527,23 +527,29 @@ void fromJsonValue(const rapidjson::Value& val, cryptonote::txout_to_script& txo
}
-void toJsonValue(rapidjson::Writer<epee::byte_stream>& dest, const cryptonote::txout_to_scripthash& txout)
+void toJsonValue(rapidjson::Writer<epee::byte_stream>& dest, const cryptonote::txout_to_carrot_v1& txout)
{
dest.StartObject();
- INSERT_INTO_JSON_OBJECT(dest, hash, txout.hash);
+ INSERT_INTO_JSON_OBJECT(dest, key, txout.key);
+ // TODO: the rest of Carrot
+ // INSERT_INTO_JSON_OBJECT(dest, view_tag, txout.view_tag);
+ // INSERT_INTO_JSON_OBJECT(dest, encrypted_janus_anchor, txout.encrypted_janus_anchor);
dest.EndObject();
}
-void fromJsonValue(const rapidjson::Value& val, cryptonote::txout_to_scripthash& txout)
+void fromJsonValue(const rapidjson::Value& val, cryptonote::txout_to_carrot_v1& txout)
{
if (!val.IsObject())
{
throw WRONG_TYPE("json object");
}
- GET_FROM_JSON_OBJECT(val, txout.hash, hash);
+ GET_FROM_JSON_OBJECT(val, txout.key, key);
+ // TODO: the rest of Carrot
+ // GET_FROM_JSON_OBJECT(val, txout.view_tag, view_tag);
+ // GET_FROM_JSON_OBJECT(val, txout.encrypted_janus_anchor, encrypted_janus_anchor);
}
@@ -610,9 +616,9 @@ void toJsonValue(rapidjson::Writer<epee::byte_stream>& dest, const cryptonote::t
{
INSERT_INTO_JSON_OBJECT(dest, to_script, output);
}
- void operator()(cryptonote::txout_to_scripthash const& output) const
+ void operator()(cryptonote::txout_to_carrot_v1 const& output) const
{
- INSERT_INTO_JSON_OBJECT(dest, to_scripthash, output);
+ INSERT_INTO_JSON_OBJECT(dest, to_carrot_v1, output);
}
};
boost::apply_visitor(add_output{dest}, txout.target);
@@ -656,9 +662,9 @@ void fromJsonValue(const rapidjson::Value& val, cryptonote::tx_out& txout)
fromJsonValue(elem.value, tmpVal);
txout.target = std::move(tmpVal);
}
- else if (elem.name == "to_scripthash")
+ else if (elem.name == "to_carrot_v1")
{
- cryptonote::txout_to_scripthash tmpVal;
+ cryptonote::txout_to_carrot_v1 tmpVal;
fromJsonValue(elem.value, tmpVal);
txout.target = std::move(tmpVal);
}
### src/serialization/json_object.h
@@ -222,8 +222,8 @@ void fromJsonValue(const rapidjson::Value& val, cryptonote::txout_target_v& txou
void toJsonValue(rapidjson::Writer<epee::byte_stream>& dest, const cryptonote::txout_to_script& txout);
void fromJsonValue(const rapidjson::Value& val, cryptonote::txout_to_script& txout);
-void toJsonValue(rapidjson::Writer<epee::byte_stream>& dest, const cryptonote::txout_to_scripthash& txout);
-void fromJsonValue(const rapidjson::Value& val, cryptonote::txout_to_scripthash& txout);
+void toJsonValue(rapidjson::Writer<epee::byte_stream>& dest, const cryptonote::txout_to_carrot_v1& txout);
+void fromJsonValue(const rapidjson::Value& val, cryptonote::txout_to_carrot_v1& txout);
void toJsonValue(rapidjson::Writer<epee::byte_stream>& dest, const cryptonote::txout_to_key& txout);
void fromJsonValue(const rapidjson::Value& val, cryptonote::txout_to_key& txout);
### src/serialization/variant.h
@@ -38,11 +38,10 @@
#include <boost/variant/variant.hpp>
#include <boost/variant/apply_visitor.hpp>
-#include <boost/variant/static_visitor.hpp>
-#include <boost/mpl/empty.hpp>
-#include <boost/mpl/if.hpp>
-#include <boost/mpl/front.hpp>
-#include <boost/mpl/pop_front.hpp>
+#include <boost/mpl/vector.hpp>
+
+#include <variant>
+
#include "serialization.h"
/*! \struct variant_serialization_traits
@@ -100,13 +99,41 @@ struct variant_reader<Archive, Variant, TBegin, TBegin>
}
};
-template <template <bool> class Archive, typename... T>
-static bool do_serialize(Archive<false> &ar, boost::variant<T...> &v) {
- using types = typename boost::variant<T...>::types;
+namespace detail
+{
+template <typename T>
+struct is_variant_serializable: std::false_type {};
+
+template <typename... T>
+struct is_variant_serializable<boost::variant<T...>>: std::true_type {};
+
+template <typename... T>
+struct is_variant_serializable<std::variant<T...>>: std::true_type {};
+
+template <typename T>
+struct get_variant_types {};
+
+template <typename... T>
+struct get_variant_types<boost::variant<T...>>
+{ using types = typename boost::variant<T...>::types; };
+
+template <typename... T>
+struct get_variant_types<std::variant<T...>>
+{ using types = boost::mpl::vector<T...>; };
+
+} //namespace detail
+
+template <
+ template <bool> class Archive,
+ typename Variant,
+ typename = std::enable_if_t<detail::is_variant_serializable<Variant>::value>
+ >
+static bool do_serialize(Archive<false> &ar, Variant &v) {
+ using types = typename detail::get_variant_types<Variant>::types;
typename Archive<false>::variant_tag_type t;
ar.begin_variant();
ar.read_variant_tag(t);
- if(!variant_reader<Archive<false>, boost::variant<T...>,
+ if(!variant_reader<Archive<false>, Variant,
typename boost::mpl::begin<types>::type,
typename boost::mpl::end<types>::type>::read(ar, v, t))
{
@@ -118,7 +145,7 @@ static bool do_serialize(Archive<false> &ar, boost::variant<T...> &v) {
}
template <template <bool> class Archive>
-struct variant_write_visitor : public boost::static_visitor<bool>
+struct variant_write_visitor
{
Archive<true> &ar;
@@ -139,8 +166,22 @@ struct variant_write_visitor : public boost::static_visitor<bool>
}
};
-template <template <bool> class Archive, typename... T>
+template <
+ template <bool> class Archive,
+ typename... T,
+ typename = std::enable_if_t<detail::is_variant_serializable<boost::variant<T...>>::value>
+ >
static bool do_serialize(Archive<true> &ar, boost::variant<T...> &v)
{
return boost::apply_visitor(variant_write_visitor<Archive>(ar), v);
}
+
+template <
+ template <bool> class Archive,
+ typename... T,
+ typename = std::enable_if_t<detail::is_variant_serializable<std::variant<T...>>::value>
+ >
+static bool do_serialize(Archive<true> &ar, std::variant<T...> &v)
+{
+ return std::visit(variant_write_visitor<Archive>(ar), v);
+}
### tests/block_weight/block_weight.cpp
@@ -54,6 +54,12 @@ class TestDB: public cryptonote::BaseTestDB
uint64_t long_term_weight;
};
+ struct tree_meta_t
+ {
+ uint64_t block_idx;
+ uint64_t n_leaf_tuples;
+ };
+
public:
TestDB() { m_open = true; }
@@ -97,13 +103,16 @@ class TestDB: public cryptonote::BaseTestDB
*block_height = h - 1;
return top;
}
- virtual void pop_block(cryptonote::block &, std::vector<cryptonote::transaction> *) override { blocks.pop_back(); }
+ virtual void pop_block(cryptonote::block &, std::vector<cryptonote::transaction> *) override { if (!blocks.empty()) { blocks.pop_back(); } if (!tree_metas.empty()) { tree_metas.pop_back(); } }
virtual void set_hard_fork_version(uint64_t height, uint8_t version) override { if (height >= hf.size()) hf.resize(height + 1); hf[height] = version; }
virtual uint8_t get_hard_fork_version(uint64_t height) const override { if (height >= hf.size()) return 255; return hf[height]; }
+ virtual void save_tree_meta(const uint64_t block_idx, const uint64_t n_leaf_tuples, const std::vector<crypto::ec_point> &tree_edge) override { tree_metas.push_back(tree_meta_t{block_idx, n_leaf_tuples}); };
+ virtual uint64_t get_tree_block_idx() const override { return tree_metas.size() ? tree_metas.back().block_idx : 0; };
private:
std::vector<block_t> blocks;
std::vector<uint8_t> hf;
+ std::vector<tree_meta_t> tree_metas;
};
}
@@ -146,7 +155,7 @@ static void test(test_t t, uint64_t blocks)
cryptonote::block b;
b.major_version = 1;
b.minor_version = 1;
- bc->get_db().add_block(std::make_pair(b, ""), 300000, 300000, bc->get_db().height(), bc->get_db().height(), {});
+ bc->get_db().add_block(std::make_pair(b, ""), 300000, 300000, bc->get_db().height(), bc->get_db().height(), {}, {});
if (!bc->update_next_cumulative_weight_limit())
{
fprintf(stderr, "Failed to update cumulative weight limit 1\n");
@@ -180,7 +189,7 @@ static void test(test_t t, uint64_t blocks)
cryptonote::block b;
b.major_version = HF_VERSION_2021_SCALING;
b.minor_version = HF_VERSION_2021_SCALING;
- bc->get_db().add_block(std::make_pair(std::move(b), ""), w, ltw, bc->get_db().height(), bc->get_db().height(), {});
+ bc->get_db().add_block(std::make_pair(std::move(b), ""), w, ltw, bc->get_db().height(), bc->get_db().height(), {}, {});
if (!bc->update_next_cumulative_weight_limit())
{
### tests/core_tests/chaingen.cpp
@@ -78,6 +78,12 @@ namespace
crypto::hash hash;
};
+ struct tree_meta_t
+ {
+ uint64_t block_idx;
+ uint64_t n_leaf_tuples;
+ };
+
public:
TestDB() { m_open = true; }
@@ -133,13 +139,17 @@ namespace
return blocks[blocks.size()-1].bl;
}
- virtual void pop_block(cryptonote::block &, std::vector<cryptonote::transaction> *) override { if (!blocks.empty()) blocks.pop_back(); }
+ virtual void pop_block(cryptonote::block &, std::vector<cryptonote::transaction> *) override { if (!blocks.empty()) { blocks.pop_back(); } if (!tree_metas.empty()) { tree_metas.pop_back(); } }
virtual void set_hard_fork_version(uint64_t height, uint8_t version) override { if (height >= hf.size()) hf.resize(height + 1); hf[height] = version; }
virtual uint8_t get_hard_fork_version(uint64_t height) const override { if (height >= hf.size()) return 255; return hf[height]; }
+ virtual void save_tree_meta(const uint64_t block_idx, const uint64_t n_leaf_tuples, const std::vector<crypto::ec_point> &tree_edge) override { tree_metas.push_back(tree_meta_t{block_idx, n_leaf_tuples}); };
+ virtual uint64_t get_tree_block_idx() const override { return tree_metas.size() ? tree_metas.back().block_idx : 0; };
+
private:
std::vector<block_t> blocks;
std::vector<uint8_t> hf;
+ std::vector<tree_meta_t> tree_metas;
};
}
### tests/core_tests/tx_validation.cpp
@@ -746,7 +746,7 @@ bool gen_tx_output_is_not_txout_to_key::generate(std::vector<test_event_entry>&
builder.m_tx.vout.push_back(tx_out());
builder.m_tx.vout.back().amount = 1;
- builder.m_tx.vout.back().target = txout_to_scripthash();
+ builder.m_tx.vout.back().target = txout_to_carrot_v1();
builder.step4_calc_hash();
builder.step5_sign(sources);
### tests/unit_tests/blockchain_db.cpp
@@ -40,6 +40,7 @@
#include "blockchain_db/lmdb/db_lmdb.h"
#include "common/pruning.h"
#include "cryptonote_basic/cryptonote_format_utils.h"
+#include "cryptonote_core/tx_verification_utils.h"
using namespace cryptonote;
using epee::string_tools::pod_to_hex;
@@ -449,8 +450,13 @@ TYPED_TEST(BlockchainDBTest, AddBlock)
// no blocks have been added yet (because genesis has no parent).
//ASSERT_THROW(this->m_db->add_block(this->m_blocks[1], t_sizes[1], t_sizes[1], t_diffs[1], t_coins[1], this->m_txs[1]), BLOCK_PARENT_DNE);
- ASSERT_NO_THROW(this->m_db->add_block(this->m_blocks[0], t_sizes[0], t_sizes[0], t_diffs[0], t_coins[0], this->m_txs[0]));
- ASSERT_NO_THROW(this->m_db->add_block(this->m_blocks[1], t_sizes[1], t_sizes[1], t_diffs[1], t_coins[1], this->m_txs[1]));
+ std::unordered_map<uint64_t, rct::key> transparent_amount_commitments1;
+ std::unordered_map<uint64_t, rct::key> transparent_amount_commitments2;
+ cryptonote::collect_transparent_amount_commitments(this->m_blocks[0].first.miner_tx, this->m_txs[0], transparent_amount_commitments1);
+ cryptonote::collect_transparent_amount_commitments(this->m_blocks[1].first.miner_tx, this->m_txs[1], transparent_amount_commitments2);
+
+ ASSERT_NO_THROW(this->m_db->add_block(this->m_blocks[0], t_sizes[0], t_sizes[0], t_diffs[0], t_coins[0], this->m_txs[0], transparent_amount_commitments1));
+ ASSERT_NO_THROW(this->m_db->add_block(this->m_blocks[1], t_sizes[1], t_sizes[1], t_diffs[1], t_coins[1], this->m_txs[1], transparent_amount_commitments2));
block b;
ASSERT_TRUE(this->m_db->block_exists(get_block_hash(this->m_blocks[0].first)));
@@ -463,7 +469,7 @@ TYPED_TEST(BlockchainDBTest, AddBlock)
ASSERT_TRUE(compare_blocks(this->m_blocks[0].first, b));
// assert that we can't add the same block twice
- ASSERT_THROW(this->m_db->add_block(this->m_blocks[0], t_sizes[0], t_sizes[0], t_diffs[0], t_coins[0], this->m_txs[0]), TX_EXISTS);
+ ASSERT_THROW(this->m_db->add_block(this->m_blocks[0], t_sizes[0], t_sizes[0], t_diffs[0], t_coins[0], this->m_txs[0], transparent_amount_commitments1), TX_EXISTS);
for (auto& h : this->m_blocks[0].first.tx_hashes)
{
@@ -489,14 +495,19 @@ TYPED_TEST(BlockchainDBTest, RetrieveBlockData)
db_wtxn_guard guard(this->m_db);
- ASSERT_NO_THROW(this->m_db->add_block(this->m_blocks[0], t_sizes[0], t_sizes[0], t_diffs[0], t_coins[0], this->m_txs[0]));
+ std::unordered_map<uint64_t, rct::key> transparent_amount_commitments1;
+ std::unordered_map<uint64_t, rct::key> transparent_amount_commitments2;
+ cryptonote::collect_transparent_amount_commitments(this->m_blocks[0].first.miner_tx, this->m_txs[0], transparent_amount_commitments1);
+ cryptonote::collect_transparent_amount_commitments(this->m_blocks[1].first.miner_tx, this->m_txs[1], transparent_amount_commitments2);
+
+ ASSERT_NO_THROW(this->m_db->add_block(this->m_blocks[0], t_sizes[0], t_sizes[0], t_diffs[0], t_coins[0], this->m_txs[0], transparent_amount_commitments1));
ASSERT_EQ(t_sizes[0], this->m_db->get_block_weight(0));
ASSERT_EQ(t_diffs[0], this->m_db->get_block_cumulative_difficulty(0));
ASSERT_EQ(t_diffs[0], this->m_db->get_block_difficulty(0));
ASSERT_EQ(t_coins[0], this->m_db->get_block_already_generated_coins(0));
- ASSERT_NO_THROW(this->m_db->add_block(this->m_blocks[1], t_sizes[1], t_sizes[1], t_diffs[1], t_coins[1], this->m_txs[1]));
+ ASSERT_NO_THROW(this->m_db->add_block(this->m_blocks[1], t_sizes[1], t_sizes[1], t_diffs[1], t_coins[1], this->m_txs[1], transparent_amount_commitments2));
ASSERT_EQ(t_diffs[1] - t_diffs[0], this->m_db->get_block_difficulty(1));
ASSERT_HASH_EQ(get_block_hash(this->m_blocks[0].first), this->m_db->get_block_hash_from_height(0));
### tests/unit_tests/long_term_block_weight.cpp
@@ -48,6 +48,12 @@ class TestDB: public cryptonote::BaseTestDB
uint64_t long_term_weight;
};
+ struct tree_meta_t
+ {
+ uint64_t block_idx;
+ uint64_t n_leaf_tuples;
+ };
+
public:
TestDB() { m_open = true; }
@@ -91,10 +97,13 @@ class TestDB: public cryptonote::BaseTestDB
*block_height = h - 1;
return top;
}
- virtual void pop_block(cryptonote::block &blk, std::vector<cryptonote::transaction> *txs) override { blocks.pop_back(); }
+ virtual void pop_block(cryptonote::block &, std::vector<cryptonote::transaction> *) override { if (!blocks.empty()) { blocks.pop_back(); } if (!tree_metas.empty()) { tree_metas.pop_back(); } }
+ virtual void save_tree_meta(const uint64_t block_idx, const uint64_t n_leaf_tuples, const std::vector<crypto::ec_point> &tree_edge) override { tree_metas.push_back(tree_meta_t{block_idx, n_leaf_tuples}); };
+ virtual uint64_t get_tree_block_idx() const override { return tree_metas.size() ? tree_metas.back().block_idx : 0; };
private:
std::vector<block_t> blocks;
+ std::vector<tree_meta_t> tree_metas;
};
static uint32_t lcg_seed = 0;
@@ -141,7 +150,7 @@ TEST(long_term_block_weight, identical_before_fork)
{
size_t w = h < CRYPTONOTE_REWARD_BLOCKS_WINDOW ? CRYPTONOTE_BLOCK_GRANTED_FULL_REWARD_ZONE_V5 : bc->get_current_cumulative_block_weight_limit();
uint64_t ltw = bc->get_next_long_term_block_weight(w);
- bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {});
+ bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {}, {});
ASSERT_TRUE(bc->update_next_cumulative_weight_limit());
}
for (uint64_t h = 0; h < 10 * TEST_LONG_TERM_BLOCK_WEIGHT_WINDOW; ++h)
@@ -158,7 +167,7 @@ TEST(long_term_block_weight, identical_after_fork_before_long_term_window)
{
size_t w = h < TEST_LONG_TERM_BLOCK_WEIGHT_WINDOW ? CRYPTONOTE_BLOCK_GRANTED_FULL_REWARD_ZONE_V5 : bc->get_current_cumulative_block_weight_limit();
uint64_t ltw = bc->get_next_long_term_block_weight(w);
- bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {});
+ bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {}, {});
ASSERT_TRUE(bc->update_next_cumulative_weight_limit());
}
for (uint64_t h = 0; h < TEST_LONG_TERM_BLOCK_WEIGHT_WINDOW; ++h)
@@ -175,7 +184,7 @@ TEST(long_term_block_weight, ceiling_at_30000000)
{
size_t w = h < TEST_LONG_TERM_BLOCK_WEIGHT_WINDOW ? CRYPTONOTE_BLOCK_GRANTED_FULL_REWARD_ZONE_V5 : bc->get_current_cumulative_block_weight_limit();
uint64_t ltw = bc->get_next_long_term_block_weight(w);
- bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {});
+ bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {}, {});
ASSERT_TRUE(bc->update_next_cumulative_weight_limit());
}
ASSERT_EQ(bc->get_current_cumulative_block_weight_median(), 15000000);
@@ -190,7 +199,7 @@ TEST(long_term_block_weight, multi_pop)
{
size_t w = h < TEST_LONG_TERM_BLOCK_WEIGHT_WINDOW ? CRYPTONOTE_BLOCK_GRANTED_FULL_REWARD_ZONE_V5 : bc->get_current_cumulative_block_weight_limit();
uint64_t ltw = bc->get_next_long_term_block_weight(w);
- bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {});
+ bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {}, {});
ASSERT_TRUE(bc->update_next_cumulative_weight_limit());
}
@@ -202,7 +211,7 @@ TEST(long_term_block_weight, multi_pop)
{
size_t w = bc->get_current_cumulative_block_weight_limit();
uint64_t ltw = bc->get_next_long_term_block_weight(w);
- bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {});
+ bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {}, {});
ASSERT_TRUE(bc->update_next_cumulative_weight_limit());
}
@@ -223,7 +232,7 @@ TEST(long_term_block_weight, multiple_updates)
{
size_t w = h < TEST_LONG_TERM_BLOCK_WEIGHT_WINDOW ? CRYPTONOTE_BLOCK_GRANTED_FULL_REWARD_ZONE_V5 : bc->get_current_cumulative_block_weight_limit();
uint64_t ltw = bc->get_next_long_term_block_weight(w);
- bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {});
+ bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {}, {});
ASSERT_TRUE(bc->update_next_cumulative_weight_limit());
const uint64_t effective_median = bc->get_current_cumulative_block_weight_median();
const uint64_t effective_limit = bc->get_current_cumulative_block_weight_limit();
@@ -247,7 +256,7 @@ TEST(long_term_block_weight, pop_invariant_max)
{
size_t w = bc->get_db().height() < TEST_LONG_TERM_BLOCK_WEIGHT_WINDOW ? CRYPTONOTE_BLOCK_GRANTED_FULL_REWARD_ZONE_V5 : bc->get_current_cumulative_block_weight_limit();
uint64_t ltw = bc->get_next_long_term_block_weight(w);
- bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {});
+ bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {}, {});
ASSERT_TRUE(bc->update_next_cumulative_weight_limit());
}
@@ -274,7 +283,7 @@ TEST(long_term_block_weight, pop_invariant_max)
{
size_t w = bc->get_db().height() < TEST_LONG_TERM_BLOCK_WEIGHT_WINDOW ? CRYPTONOTE_BLOCK_GRANTED_FULL_REWARD_ZONE_V5 : bc->get_current_cumulative_block_weight_limit();
uint64_t ltw = bc->get_next_long_term_block_weight(w);
- bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, bc->get_db().height(), bc->get_db().height(), {});
+ bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, bc->get_db().height(), bc->get_db().height(), {}, {});
ASSERT_TRUE(bc->update_next_cumulative_weight_limit());
}
@@ -296,7 +305,7 @@ TEST(long_term_block_weight, pop_invariant_random)
uint32_t r = lcg();
size_t w = bc->get_db().height() < TEST_LONG_TERM_BLOCK_WEIGHT_WINDOW ? CRYPTONOTE_BLOCK_GRANTED_FULL_REWARD_ZONE_V5 : (r % bc->get_current_cumulative_block_weight_limit());
uint64_t ltw = bc->get_next_long_term_block_weight(w);
- bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {});
+ bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {}, {});
ASSERT_TRUE(bc->update_next_cumulative_weight_limit());
}
@@ -330,7 +339,7 @@ TEST(long_term_block_weight, pop_invariant_random)
uint32_t r = lcg();
size_t w = bc->get_db().height() < TEST_LONG_TERM_BLOCK_WEIGHT_WINDOW ? CRYPTONOTE_BLOCK_GRANTED_FULL_REWARD_ZONE_V5 : (r % bc->get_current_cumulative_block_weight_limit());
uint64_t ltw = bc->get_next_long_term_block_weight(w);
- bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, bc->get_db().height(), bc->get_db().height(), {});
+ bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, bc->get_db().height(), bc->get_db().height(), {}, {});
ASSERT_TRUE(bc->update_next_cumulative_weight_limit());
const uint64_t effective_median = bc->get_current_cumulative_block_weight_median();
const uint64_t effective_limit = bc->get_current_cumulative_block_weight_limit();
@@ -358,7 +367,7 @@ TEST(long_term_block_weight, long_growth_spike_and_drop)
{
size_t w = CRYPTONOTE_BLOCK_GRANTED_FULL_REWARD_ZONE_V5;
uint64_t ltw = bc->get_next_long_term_block_weight(w);
- bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {});
+ bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {}, {});
ASSERT_TRUE(bc->update_next_cumulative_weight_limit(&long_term_effective_median_block_weight));
}
ASSERT_EQ(long_term_effective_median_block_weight, 300000);
@@ -370,7 +379,7 @@ TEST(long_term_block_weight, long_growth_spike_and_drop)
float t = h / float(365 * 720 * TEST_LONG_TERM_BLOCK_WEIGHT_WINDOW / 100000);
size_t w = 300000 + t * 30000;
uint64_t ltw = bc->get_next_long_term_block_weight(w);
- bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {});
+ bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {}, {});
ASSERT_TRUE(bc->update_next_cumulative_weight_limit(&long_term_effective_median_block_weight));
}
ASSERT_GT(long_term_effective_median_block_weight, 300000 * 1.07);
@@ -381,7 +390,7 @@ TEST(long_term_block_weight, long_growth_spike_and_drop)
{
size_t w = bc->get_current_cumulative_block_weight_limit();
uint64_t ltw = bc->get_next_long_term_block_weight(w);
- bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {});
+ bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {}, {});
ASSERT_TRUE(bc->update_next_cumulative_weight_limit(&long_term_effective_median_block_weight));
}
ASSERT_GT(long_term_effective_median_block_weight, 300000 * 1.07);
@@ -392,7 +401,7 @@ TEST(long_term_block_weight, long_growth_spike_and_drop)
{
size_t w = bc->get_current_cumulative_block_weight_median() * .25;
uint64_t ltw = bc->get_next_long_term_block_weight(w);
- bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {});
+ bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {}, {});
ASSERT_TRUE(bc->update_next_cumulative_weight_limit(&long_term_effective_median_block_weight));
}
ASSERT_GT(long_term_effective_median_block_weight, 300000 * 1.07);
@@ -408,7 +417,7 @@ TEST(long_term_block_weight, cache_matches_true_value)
{
size_t w = bc->get_current_cumulative_block_weight_limit();
uint64_t ltw = bc->get_next_long_term_block_weight(w);
- bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {});
+ bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {}, {});
bc->update_next_cumulative_weight_limit();
}
@@ -419,7 +428,7 @@ TEST(long_term_block_weight, cache_matches_true_value)
{
size_t w = (bc->get_current_cumulative_block_weight_median() * 10/17) - 1000;
uint64_t ltw = bc->get_next_long_term_block_weight(w);
- bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {});
+ bc->get_db().add_block(std::make_pair(cryptonote::block(), ""), w, ltw, h, h, {}, {});
bc->update_next_cumulative_weight_limit();
}
### tests/unit_tests/node_server.cpp
@@ -1133,6 +1133,8 @@ TEST(cryptonote_protocol_handler, race_condition)
const stat::chain &stat
){
core.get_blockchain_storage().get_db().batch_start({}, {});
+ std::unordered_map<uint64_t, rct::key> transparent_amount_commitments;
+ cryptonote::collect_transparent_amount_commitments(block.miner_tx, {}, transparent_amount_commitments);
core.get_blockchain_storage().get_db().add_block(
{block, cryptonote::block_to_blob(block)},
cryptonote::get_transaction_weight(block.miner_tx),
@@ -1141,7 +1143,8 @@ TEST(cryptonote_protocol_handler, race_condition)
),
stat.diff,
stat.reward,
- {}
+ {},
+ transparent_amount_commitments
);
core.get_blockchain_storage().get_db().batch_stop();
};Why this scored 33/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.