verify inclusion proofs for newly confirmed transactions, leaving them unconfirmed where they cannot be proved
What changed, and why it matters
This commit adds a security feature to Sparrow Wallet that verifies Bitcoin transaction inclusion proofs supplied by Electrum servers. Before this change, the wallet trusted the server when it reported a transaction as 'confirmed' in a particular block. Now, for newly confirmed transactions, the wallet asks the server for a cryptographic Merkle proof and checks it against a verified block header. If the proof is missing, wrong, or for a different block, the transaction is demoted to 'unconfirmed' and the user is warned that the server may be faulty or dishonest. The change protects users against a malicious or compromised server falsely claiming a transaction was confirmed.
This is a defensive hardening commit. Users and downstream packagers should ensure they upgrade to a release containing this commit, especially when using public Electrum servers. Operators should monitor for the new 'Transaction Verification Failed/Refused' dialogs and consider switching servers if they appear. No immediate mitigation is needed beyond applying the update; the change is enabled by default.
Security signals we found
Adds cryptographic verification of server-reported transaction confirmations (Merkle proofs against verified headers)
Demotes unverifiable confirmed transactions to unconfirmed instead of accepting server claims
Distinguishes server 'refusal' from 'dishonest/faulty' proof failure and surfaces distinct user dialogs
Adds retry/backoff logic to avoid false positives from transient server errors
Makes verification mandatory for public Electrum servers on mainnet
Disables verification for user's own Bitcoin Core node (trusted anyway)
Handles CVE-2017-12842-style 64-byte transaction forgery by rejecting inner Merkle nodes that deserialize as transactions
Clears stale announced-tip header cache above reorg fork to avoid serving orphaned block metadata
Records block hash on stored transactions to detect same-height reorgs
Evidence from the diff
The patch introduces transaction verification via Electrum blockchain.transaction.get_merkle proofs. It adds verifyConfirmedReferences() and verifySilentPaymentReferences() in ElectrumServer, which batch-request Merkle proofs for newly confirmed (height > 0) references, verify them against verifiedHistoricalHeaders/HeaderStore block headers using MerkleBranch.computeRoot(), and demote unproven references to height 0 in the wallet’s node history. It adds retry logic (proofAttempts, proofRetryDelayMillis) to distinguish transient server failures from persistent refusals, and emits TransactionProofsFailedEvent (proof does not reconstruct merkle root) or TransactionProofsRefusedEvent (server won’t substantiate the reported height). A config flag verifyTransactions defaults to true; verification is mandatory for public Electrum servers on mainnet, disabled for Bitcoin Core backends, and skipped for servers not caught up to the last checkpoint. It also clears stale retrievedBlockHeaders above a reorg fork and records the proven block hash on BlockTransaction so reorgs at the same height are detected.
Changed components
com.sparrowwallet.sparrow.net.ElectrumServercom.sparrowwallet.sparrow.net.BatchedElectrumServerRpccom.sparrowwallet.sparrow.net.ProofsUnavailableExceptioncom.sparrowwallet.sparrow.io.Configcom.sparrowwallet.sparrow.AppServicescom.sparrowwallet.sparrow.wallet.WalletFormcom.sparrowwallet.sparrow.event.TransactionProofsEvent / TransactionProofsFailedEvent / TransactionProofsRefusedEvent / RequestWalletRefreshEventInspect captured patch +2687 / −106
### build.gradle
@@ -131,6 +131,7 @@ test {
excludeTags 'checkpoint'
}
jvmArgs = ["--enable-native-access=ALL-UNNAMED"]
+ systemProperty 'sparrow.home', layout.buildDirectory.dir('test-home').get().asFile.absolutePath
}
tasks.register('verifyCheckpoint', Test) {
### src/main/java/com/sparrowwallet/sparrow/AppServices.java
@@ -1528,6 +1528,43 @@ public void walletHistoryFailed(WalletHistoryFailedEvent event) {
}
}
+ @Subscribe
+ public void transactionProofsFailed(TransactionProofsFailedEvent event) {
+ showProofsDialog(event, "Transaction Verification Failed", describeProofs(event.getReferences())
+ + (event.getReferences().size() == 1 ? " but the proof of inclusion it supplied does not match that block." : " but the proofs of inclusion it supplied do not match those blocks.")
+ + " This means the server is either faulty or dishonest, and what it reported may not have been confirmed at all.");
+ }
+
+ @Subscribe
+ public void transactionProofsRefused(TransactionProofsRefusedEvent event) {
+ showProofsDialog(event, "Transaction Verification Refused", describeProofs(event.getReferences())
+ + (event.getReferences().size() == 1 ? " which then declined to prove it at that height." : " which then declined to prove them at those heights.")
+ + " A server contradicting itself in this way may be faulty or overloaded, and what it reported cannot be taken as confirmed.");
+ }
+
+ private void showProofsDialog(TransactionProofsEvent event, String title, String content) {
+ Platform.runLater(() -> {
+ ButtonType refreshButton = new ButtonType("Refresh Wallet", ButtonBar.ButtonData.OK_DONE);
+ Optional<ButtonType> optType = showErrorDialog(title, content + (event.getReferences().size() == 1 ? " It is" : " They are")
+ + " shown as unconfirmed until verified.\n\nConsider switching servers, and refreshing the wallet afterwards.",
+ ButtonType.CANCEL, refreshButton);
+ if(optType.isPresent() && optType.get() == refreshButton) {
+ EventManager.get().post(new RequestWalletRefreshEvent(event.getWallet()));
+ }
+ });
+ }
+
+ private static String describeProofs(Set<BlockTransactionHash> references) {
+ BlockTransactionHash first = references.iterator().next();
+ String firstId = first.getHashAsString().substring(0, 8) + "..";
+ if(references.size() == 1) {
+ return "Transaction " + firstId + " was reported as confirmed in block " + first.getHeight() + " by the connected server,";
+ }
+
+ return references.size() + " transactions, the first being " + firstId + " in block " + first.getHeight()
+ + ", were reported as confirmed by the connected server,";
+ }
+
@Subscribe
public void silentPaymentsUnsubscribe(SilentPaymentsUnsubscribeEvent event) {
if(isConnected()) {
### src/main/java/com/sparrowwallet/sparrow/event/RequestWalletRefreshEvent.java
@@ -0,0 +1,18 @@
+package com.sparrowwallet.sparrow.event;
+
+import com.sparrowwallet.drongo.wallet.Wallet;
+
+/**
+ * Requests that the history of the given wallet be fetched again, for the places that can offer a refresh without holding the wallet's form.
+ */
+public class RequestWalletRefreshEvent {
+ private final Wallet wallet;
+
+ public RequestWalletRefreshEvent(Wallet wallet) {
+ this.wallet = wallet;
+ }
+
+ public Wallet getWallet() {
+ return wallet;
+ }
+}
### src/main/java/com/sparrowwallet/sparrow/event/TransactionProofsEvent.java
@@ -0,0 +1,31 @@
+package com.sparrowwallet.sparrow.event;
+
+import com.sparrowwallet.drongo.wallet.BlockTransactionHash;
+import com.sparrowwallet.drongo.wallet.Wallet;
+
+import java.util.Set;
+
+/**
+ * The transactions of one wallet whose confirmed heights the connected server did not prove in a single history pass, aggregated so that a pass
+ * surfacing many of them raises one dialog rather than one per transaction.
+ */
+public abstract class TransactionProofsEvent {
+ private final Wallet wallet;
+ private final Set<BlockTransactionHash> references;
+
+ public TransactionProofsEvent(Wallet wallet, Set<BlockTransactionHash> references) {
+ this.wallet = wallet;
+ this.references = references;
+ }
+
+ public Wallet getWallet() {
+ return wallet;
+ }
+
+ /**
+ * The transactions with the heights the server reported them at, which are no longer the heights they are held at.
+ */
+ public Set<BlockTransactionHash> getReferences() {
+ return references;
+ }
+}
### src/main/java/com/sparrowwallet/sparrow/event/TransactionProofsFailedEvent.java
@@ -0,0 +1,18 @@
+package com.sparrowwallet.sparrow.event;
+
+import com.sparrowwallet.drongo.wallet.BlockTransactionHash;
+import com.sparrowwallet.drongo.wallet.Wallet;
+
+import java.util.Set;
+
+/**
+ * Posted once per wallet history pass where the server supplied a proof that did not reconstruct the merkle root of the verified header at the height
+ * it reported - the server proven wrong rather than merely unhelpful. The transactions carry the reported heights, and are already written unconfirmed.
+ * <p>
+ * Dispatched on the wallet history thread, so a handler must hop to the application thread itself.
+ */
+public class TransactionProofsFailedEvent extends TransactionProofsEvent {
+ public TransactionProofsFailedEvent(Wallet wallet, Set<BlockTransactionHash> references) {
+ super(wallet, references);
+ }
+}
### src/main/java/com/sparrowwallet/sparrow/event/TransactionProofsRefusedEvent.java
@@ -0,0 +1,19 @@
+package com.sparrowwallet.sparrow.event;
+
+import com.sparrowwallet.drongo.wallet.BlockTransactionHash;
+import com.sparrowwallet.drongo.wallet.Wallet;
+
+import java.util.Set;
+
+/**
+ * Posted once per wallet history pass where the server reported a transaction as confirmed and then would not substantiate it at that height, by
+ * erroring, by answering for another block, or by leaving the header unverifiable. Nothing has been shown false, so this is the server contradicting
+ * itself rather than lying. The transactions are already written unconfirmed.
+ * <p>
+ * Dispatched on the wallet history thread, so a handler must hop to the application thread itself.
+ */
+public class TransactionProofsRefusedEvent extends TransactionProofsEvent {
+ public TransactionProofsRefusedEvent(Wallet wallet, Set<BlockTransactionHash> references) {
+ super(wallet, references);
+ }
+}
### src/main/java/com/sparrowwallet/sparrow/io/Config.java
@@ -56,6 +56,7 @@ public class Config {
private boolean showDeprecatedImportExport = false;
private boolean signBsmsExports = false;
private boolean preventSleep = false;
+ private boolean verifyTransactions = true;
private Boolean connectToBroadcast;
private Boolean connectToResolve;
private Boolean suggestSendToMany;
@@ -382,6 +383,15 @@ public void setPreventSleep(boolean preventSleep) {
flush();
}
+ public boolean isVerifyTransactions() {
+ return verifyTransactions;
+ }
+
+ public void setVerifyTransactions(boolean verifyTransactions) {
+ this.verifyTransactions = verifyTransactions;
+ flush();
+ }
+
public Boolean getConnectToBroadcast() {
return connectToBroadcast;
}
### src/main/java/com/sparrowwallet/sparrow/net/BatchedElectrumServerRpc.java
@@ -25,6 +25,7 @@ public class BatchedElectrumServerRpc implements ElectrumServerRpc {
static final int DEFAULT_MAX_ATTEMPTS = 5;
static final int RETRY_DELAY_SECS = 1;
static final int HEADERS_BATCH_PAGE_SIZE = 4; //Four difficulty periods of headers is ~1.3MB of hex, within every server's max response size
+ static final int MERKLE_BATCH_PAGE_SIZE = 250; //A proof is max MerkleBranch.MAX_DEPTH sibling hashes, so ~1.1KB of JSON: 250 of them is ~275KB
private final AtomicLong idCounter;
private final int maxTargetBlocks;
@@ -293,7 +294,8 @@ public Map<String, VerboseTransaction> getVerboseTransactions(Transport transpor
@Override
@SuppressWarnings("unchecked")
public Map<String, TransactionMerkleProof> getTransactionMerkleProofs(Transport transport, Wallet wallet, Collection<BlockTransactionHash> references) {
- PagedBatchRequestBuilder<String, TransactionMerkleProof> batchRequest = PagedBatchRequestBuilder.create(transport, idCounter).keysType(String.class).returnType(TransactionMerkleProof.class);
+ PagedBatchRequestBuilder<String, TransactionMerkleProof> batchRequest = PagedBatchRequestBuilder.create(transport, idCounter).keysType(String.class)
+ .returnType(TransactionMerkleProof.class).pageSize(MERKLE_BATCH_PAGE_SIZE);
EventManager.get().post(new WalletHistoryStatusEvent(wallet, true, "Verifying " + references.size() + " transactions"));
for(BlockTransactionHash reference : references) {
### src/main/java/com/sparrowwallet/sparrow/net/ElectrumServer.java
@@ -43,6 +43,7 @@
import java.nio.charset.StandardCharsets;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.Condition;
import java.util.concurrent.locks.ReentrantLock;
import java.util.regex.Matcher;
@@ -76,11 +77,11 @@ public class ElectrumServer {
private static Server previousServer;
- private static final Map<String, String> retrievedScriptHashes = Collections.synchronizedMap(new HashMap<>());
+ static final Map<String, String> retrievedScriptHashes = Collections.synchronizedMap(new HashMap<>());
private static final Map<Sha256Hash, BlockTransaction> retrievedTransactions = new ConcurrentHashMap<>();
- private static final Map<Integer, BlockHeader> retrievedBlockHeaders = new ConcurrentHashMap<>();
+ static final Map<Integer, BlockHeader> retrievedBlockHeaders = new ConcurrentHashMap<>();
private static final Map<Sha256Hash, BlockTransaction> broadcastedTransactions = new ConcurrentHashMap<>();
@@ -130,20 +131,47 @@ public class ElectrumServer {
private static volatile long lastTipWarningLoggedAt;
+ //A server refusing for capacity recovers within these attempts, one that cannot substantiate a height never does. Not final so tests need not wait
+ static int proofAttempts = 4;
+
+ static long proofRetryDelayMillis = 2000;
+
+ //Filters the dialogs rather than the verification, so the passes that re-attempt a refused transaction do not raise it again
+ static final Set<String> proofWarnedPairs = ConcurrentHashMap.newKeySet();
+
+ //Tracked apart, since being proven wrong must still be raised where the pair was only refused on an earlier pass, while the reverse adds nothing
+ static final Set<String> proofsShownFalseWarnedPairs = ConcurrentHashMap.newKeySet();
+
private final static Map<String, Integer> subscribedRecent = new ConcurrentHashMap<>();
private final static Map<String, String> broadcastRecent = new ConcurrentHashMap<>();
+ final static Map<String, BlockTransaction> confirmingRecent = new ConcurrentHashMap<>();
+
static ElectrumServerRpc electrumServerRpc = new SimpleElectrumServerRpc();
private static Cormorant cormorant;
private static Server coreElectrumServer;
- private static ServerCapability serverCapability;
+ static ServerCapability serverCapability;
private static final Pattern RPC_WALLET_LOADING_PATTERN = Pattern.compile(".*\"(Wallet loading failed[:.][^\"]*)\".*");
+ //Per pass, since one ElectrumServer is built per history task. A demoted height reads as changed to the next of the several gated calls in a
+ //pass, so without this memo each would spend the retries on it again
+ private final Set<BlockTransactionHash> refusedThisPass = new HashSet<>();
+
+ //Keyed by wallet, since one task runs the master and each nested child. Held no longer than the task: postProofEvents removes each key in the
+ //finally that closes the wallet it was added under, and the instance is a local of a Task.call()
+ private final Map<Wallet, Set<BlockTransactionHash>> failed = new LinkedHashMap<>();
+
+ private final Map<Wallet, Set<BlockTransactionHash>> refused = new LinkedHashMap<>();
+
+ //The nodes this task demoted a height in. Their outputs disagree with the server's status by construction, so the changed history check must not
+ //count them: a wallet whose used nodes are all demoted would otherwise abort into a full refresh on every open, the demotion being stored
+ private final Set<String> demotedScriptHashes = new HashSet<>();
+
private static synchronized CloseableTransport getTransport() throws ServerException {
if(transport == null) {
try {
@@ -180,11 +208,7 @@ private static synchronized CloseableTransport getTransport() throws ServerExcep
//If changing server, don't rely on previous transaction history
if(previousServer != null && !electrumServer.equals(previousServer)) {
- retrievedScriptHashes.clear();
- retrievedTransactions.clear();
- retrievedBlockHeaders.clear();
- reorgInvalidatedScriptHashes.clear();
- walletSyncLocks.values().forEach(syncLock -> syncLock.scriptHashesInitialized = false);
+ clearPreviousServerState();
}
previousServer = electrumServer;
@@ -214,6 +238,20 @@ private static synchronized CloseableTransport getTransport() throws ServerExcep
return transport;
}
+ /**
+ * Forgets what the previous server told us on changing to another, including which of its claims were shown unproven: the dialogs ask the user to
+ * switch servers, so the new one is judged afresh. Verified headers are not forgotten, being claims about the chain rather than about the server.
+ */
+ static void clearPreviousServerState() {
+ retrievedScriptHashes.clear();
+ retrievedTransactions.clear();
+ retrievedBlockHeaders.clear();
+ reorgInvalidatedScriptHashes.clear();
+ proofWarnedPairs.clear();
+ proofsShownFalseWarnedPairs.clear();
+ walletSyncLocks.values().forEach(syncLock -> syncLock.scriptHashesInitialized = false);
+ }
+
public void connect() throws ServerException {
CloseableTransport closeableTransport = getTransport();
closeableTransport.connect();
@@ -297,7 +335,7 @@ private static void addScriptHashStatus(Map<String, String> calculatedScriptHash
calculatedScriptHashes.put(scriptHash, scriptHashStatus);
}
- private static String getScriptHashStatus(String scriptHash, WalletNode walletNode) {
+ static String getScriptHashStatus(String scriptHash, WalletNode walletNode) {
List<ScriptHashTx> scriptHashTxes = getScriptHashes(scriptHash, walletNode);
return getScriptHashStatus(scriptHashTxes);
}
@@ -329,7 +367,7 @@ private static List<ScriptHashTx> getScriptHashes(String scriptHash, WalletNode
return txos.stream().map(txo -> new ScriptHashTx(txo.getHeight(), txo.getHashAsString(), txo.getFee() == null ? 0 : txo.getFee())).toList();
}
- private static String getScriptHashStatus(List<ScriptHashTx> scriptHashTxes) {
+ static String getScriptHashStatus(List<ScriptHashTx> scriptHashTxes) {
if(!scriptHashTxes.isEmpty()) {
StringBuilder scriptHashStatus = new StringBuilder();
for(ScriptHashTx scriptHashTx : scriptHashTxes) {
@@ -415,55 +453,74 @@ private boolean fetchAndCalculateWalletHistory(Wallet wallet, List<Wallet> filte
}
if(isConnected()) {
- Map<String, String> previousScriptHashes = getCalculatedScriptHashes(wallet);
- Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = (nodes == null ? getHistory(wallet) : getHistory(wallet, nodes));
- getReferencedTransactions(wallet, nodeTransactionMap);
- calculateNodeHistory(wallet, nodeTransactionMap);
-
- //A node invalidated by a reorg has no retrieved status left to compare against, so it must not count as changed history below
- Set<String> invalidatedScriptHashes = Set.copyOf(reorgInvalidatedScriptHashes);
+ try {
+ //Taken before the fetch, so an invalidation arriving while this pass runs can be told from one the pass is acting on
+ Set<String> invalidatedBeforeFetch = Set.copyOf(reorgInvalidatedScriptHashes);
+ Map<String, String> previousScriptHashes = getCalculatedScriptHashes(wallet);
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = (nodes == null ? getHistory(wallet) : getHistory(wallet, nodes));
+ getReferencedTransactions(wallet, nodeTransactionMap);
+ calculateNodeHistory(wallet, nodeTransactionMap);
+
+ //A node invalidated by a reorg has no retrieved status left to compare against, so it must not count as changed history below
+ Set<String> invalidatedScriptHashes = Set.copyOf(reorgInvalidatedScriptHashes);
+
+ //Add all of the script hashes we have now fetched the history for so we don't need to fetch again until the script hash status changes
+ Set<WalletNode> updatedNodes = new HashSet<>();
+ Map<WalletNode, Set<BlockTransactionHashIndex>> walletNodes = wallet.getWalletNodes();
+ for(WalletNode node : (nodes == null ? walletNodes.keySet() : nodes)) {
+ String scriptHash = getScriptHash(node);
+ String subscribedStatus = getSubscribedScriptHashStatus(scriptHash);
+ if(!Objects.equals(subscribedStatus, retrievedScriptHashes.get(scriptHash))) {
+ updatedNodes.add(node);
+ }
- //Add all of the script hashes we have now fetched the history for so we don't need to fetch again until the script hash status changes
- Set<WalletNode> updatedNodes = new HashSet<>();
- Map<WalletNode, Set<BlockTransactionHashIndex>> walletNodes = wallet.getWalletNodes();
- for(WalletNode node : (nodes == null ? walletNodes.keySet() : nodes)) {
- String scriptHash = getScriptHash(node);
- String subscribedStatus = getSubscribedScriptHashStatus(scriptHash);
- if(!Objects.equals(subscribedStatus, retrievedScriptHashes.get(scriptHash))) {
- updatedNodes.add(node);
+ //A reorg detected while this pass was in flight - which is what happens when a history thread is the one to reconcile -
+ //invalidated data the pass had already fetched, so its status stays cleared for the refresh the reorg triggers. Restoring it
+ //would leave the node looking up to date while it still holds what the orphaned block proved, and nothing would fetch it again
+ if(invalidatedBeforeFetch.contains(scriptHash) || !invalidatedScriptHashes.contains(scriptHash)) {
+ retrievedScriptHashes.put(scriptHash, subscribedStatus);
+ }
}
- retrievedScriptHashes.put(scriptHash, subscribedStatus);
- }
- //If wallet was not empty, check if all used updated nodes have changed history
- if(nodes == null && previousScriptHashes.values().stream().anyMatch(Objects::nonNull)) {
- Set<WalletNode> changedNodes = updatedNodes.stream().filter(node -> !invalidatedScriptHashes.contains(getScriptHash(node))).collect(Collectors.toSet());
- if(!changedNodes.isEmpty()
- && changedNodes.equals(walletNodes.entrySet().stream().filter(entry -> !entry.getValue().isEmpty()).map(Map.Entry::getKey).collect(Collectors.toSet()))
- && !sameHeightTxioScriptHashes.containsAll(changedNodes.stream().map(ElectrumServer::getScriptHash).collect(Collectors.toSet()))) {
- //All used nodes on a non-empty wallet have changed history. Abort and trigger a full refresh.
- log.info("All used nodes on a non-empty wallet have changed history. Triggering a full wallet refresh.");
- throw new AllHistoryChangedException();
+ //If wallet was not empty, check if all used updated nodes have changed history
+ if(nodes == null && previousScriptHashes.values().stream().anyMatch(Objects::nonNull)) {
+ Set<WalletNode> changedNodes = updatedNodes.stream().filter(node -> {
+ String scriptHash = getScriptHash(node);
+ //A demoted node disagrees with the server because this pass demoted it, which is not the wallet having a different history
+ return !invalidatedScriptHashes.contains(scriptHash) && !demotedScriptHashes.contains(scriptHash);
+ }).collect(Collectors.toSet());
+ if(!changedNodes.isEmpty()
+ && changedNodes.equals(walletNodes.entrySet().stream().filter(entry -> !entry.getValue().isEmpty()).map(Map.Entry::getKey).collect(Collectors.toSet()))
+ && !sameHeightTxioScriptHashes.containsAll(changedNodes.stream().map(ElectrumServer::getScriptHash).collect(Collectors.toSet()))) {
+ //All used nodes on a non-empty wallet have changed history. Abort and trigger a full refresh.
+ log.info("All used nodes on a non-empty wallet have changed history. Triggering a full wallet refresh.");
+ throw new AllHistoryChangedException();
+ }
}
- }
- //The reorg exemption lasts for exactly one full fetch, and is cleared only once the check above has passed
- if(nodes == null && !invalidatedScriptHashes.isEmpty()) {
- reorgInvalidatedScriptHashes.removeAll(walletNodes.keySet().stream().map(ElectrumServer::getScriptHash).collect(Collectors.toSet()));
- }
+ //The exemption lasts for exactly one full fetch, and is cleared only once the check above has passed. Only what was invalidated
+ //before this pass fetched anything is cleared: the rest is for the refresh the reorg triggers, which has not run yet
+ if(nodes == null && !invalidatedBeforeFetch.isEmpty()) {
+ Set<String> walletScriptHashes = walletNodes.keySet().stream().map(ElectrumServer::getScriptHash).collect(Collectors.toSet());
+ reorgInvalidatedScriptHashes.removeAll(invalidatedBeforeFetch.stream().filter(walletScriptHashes::contains).collect(Collectors.toSet()));
+ }
- //Clear transaction outputs for nodes that have no history - this is useful when a transaction is replaced in the mempool
- if(nodes != null) {
- for(WalletNode node : nodes) {
- String scriptHash = getScriptHash(node);
- if(retrievedScriptHashes.get(scriptHash) == null && !node.getTransactionOutputs().isEmpty()) {
- log.debug("Clearing transaction history for " + node);
- node.getTransactionOutputs().clear();
+ //Clear transaction outputs for nodes that have no history - this is useful when a transaction is replaced in the mempool
+ if(nodes != null) {
+ for(WalletNode node : nodes) {
+ String scriptHash = getScriptHash(node);
+ if(retrievedScriptHashes.get(scriptHash) == null && !node.getTransactionOutputs().isEmpty()) {
+ log.debug("Clearing transaction history for " + node);
+ node.getTransactionOutputs().clear();
+ }
}
}
- }
- return true;
+ return true;
+ } finally {
+ //A finding is demoted, and may already have been written, before a later call in the pass can fail: it must be shown either way
+ postProofEvents(wallet);
+ }
}
return false;
@@ -771,8 +828,10 @@ public void subscribeWalletNodes(Wallet wallet, Collection<WalletNode> nodes, Ma
if(node != null) {
String scriptHash = getScriptHash(node);
- //Check if there is history for this script hash, and if the history has changed since last fetched
- if(status != null && !status.equals(retrievedScriptHashes.get(scriptHash))) {
+ //Check if there is history for this script hash, and if the history has changed since last fetched. The comparison against the
+ //node's own calculated status is what the already subscribed branch makes: subscriptions drop on every connect while retrieved
+ //statuses survive, so a node whose outputs disagree with the server would otherwise sit unfetched until its status changed
+ if(status != null && (!status.equals(retrievedScriptHashes.get(scriptHash)) || !status.equals(getScriptHashStatus(scriptHash, node)))) {
//Set the value for this node to be an empty set to mark it as requiring a get_history RPC call for this wallet
nodeTransactionMap.put(node, new TreeSet<>());
}
@@ -834,6 +893,10 @@ public List<Set<BlockTransactionHash>> getOutputTransactionReferences(Transactio
}
public void getReferencedTransactions(Wallet wallet, Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap) throws ServerException {
+ //The write boundary for every caller: the references below reach the wallet transactions here and the node transaction outputs through
+ //calculateNodeHistory, which consumes this same map after the unproven heights in it have been demoted in place
+ Map<BlockTransactionHash, BlockHeader> proven = isVerifyingTransactions() ? verifyConfirmedReferences(wallet, nodeTransactionMap) : Collections.emptyMap();
+
Map<BlockTransactionHash, Transaction> references = new TreeMap<>();
for(Set<BlockTransactionHash> nodeReferences : nodeTransactionMap.values()) {
for(BlockTransactionHash nodeReference : nodeReferences) {
@@ -846,7 +909,11 @@ public void getReferencedTransactions(Wallet wallet, Map<WalletNode, Set<BlockTr
BlockTransactionHash reference = entry.getKey();
BlockTransaction blockTransaction = wallet.getWalletTransaction(reference.getHash());
if(blockTransaction != null) {
- if(reference.getHeight() == blockTransaction.getHeight() && (reference.getFee() == null || blockTransaction.getFee() != null)) {
+ //A reference proven this pass is not removed even where its height is unchanged: the only way it was proven at an unchanged height is that
+ //its stored block was orphaned, so it must be rebuilt to carry the block it is now proven against. Its transaction comes from the wallet below,
+ //so nothing is fetched for it
+ if(reference.getHeight() == blockTransaction.getHeight() && (reference.getFee() == null || blockTransaction.getFee() != null)
+ && !proven.containsKey(reference)) {
iter.remove();
} else {
entry.setValue(blockTransaction.getTransaction());
@@ -859,27 +926,394 @@ public void getReferencedTransactions(Wallet wallet, Map<WalletNode, Set<BlockTr
Map<Sha256Hash, BlockTransaction> transactionMap = new HashMap<>();
if(!references.isEmpty()) {
Map<Integer, BlockHeader> blockHeaderMap = getBlockHeaders(wallet, references.keySet());
- transactionMap = getTransactions(wallet, references, blockHeaderMap);
+ transactionMap = getTransactions(wallet, references, blockHeaderMap, proven);
}
- if(!transactionMap.equals(wallet.getTransactions())) {
+ //A re-proof at an unchanged height leaves the map equal to what the wallet holds, since a block hash is not part of that comparison
+ if(!transactionMap.equals(wallet.getTransactions()) || !proven.isEmpty()) {
wallet.updateTransactions(transactionMap);
broadcastedTransactions.keySet().removeAll(transactionMap.entrySet().stream().filter(entry -> entry.getValue().getHeight() > 0)
.map(Map.Entry::getKey).collect(Collectors.toSet()));
}
}
+ /**
+ * Proves the confirmed references this pass introduces or changes, demoting in place those the server would not or could not prove, and returning
+ * the exact (transaction, height) pairs proven with the verified headers they were proven against.
+ * <p>
+ * Everything here is keyed by the pair rather than by the transaction: a server can report one transaction at two heights on two script hashes,
+ * and keyed by transaction alone the pair that did not prove would ride into the wallet on the back of the pair that did.
+ */
+ private Map<BlockTransactionHash, BlockHeader> verifyConfirmedReferences(Wallet wallet, Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap) throws ServerException {
+ Set<BlockTransactionHash> toProve = new LinkedHashSet<>();
+ Set<BlockTransactionHash> refusedNow = new HashSet<>();
+ for(Set<BlockTransactionHash> references : nodeTransactionMap.values()) {
+ for(BlockTransactionHash reference : references) {
+ if(reference.getHeight() > 0) {
+ if(refusedThisPass.contains(reference)) {
+ refusedNow.add(reference); //already refused earlier in this pass, so demote it again without spending the retries again
+ } else {
+ BlockTransaction existing = wallet.getWalletTransaction(reference.getHash());
+ if(existing == null || existing.getHeight() != reference.getHeight() || isProvenAgainstOrphanedHeader(existing)) {
+ toProve.add(reference);
+ }
+ }
+ }
+ }
+ }
+
+ if(!refusedNow.isEmpty()) {
+ demoteReferences(nodeTransactionMap, refusedNow);
+ }
+
+ if(toProve.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ Map<BlockTransactionHash, BlockHeader> proven;
+ try {
+ proven = verifyMerkleProofs(wallet, toProve);
+ } catch(UnsupportedMethodException e) {
+ disableVerification(e);
+ return Collections.emptyMap(); //nothing has been refused, so the history simply proceeds unverified
+ } catch(ProofsUnavailableException e) {
+ disableVerification(e);
+ return Collections.emptyMap();
+ }
+
+ Set<BlockTransactionHash> unproven = new LinkedHashSet<>(toProve);
+ unproven.removeAll(proven.keySet());
+ if(!unproven.isEmpty()) {
+ refusedThisPass.addAll(unproven);
+ //Exactly these pairs: a reference for the same transaction at another height is untouched and is judged on its own proof
+ demoteReferences(nodeTransactionMap, unproven);
+ }
+
+ return proven;
+ }
+
+ /**
+ * Proves the confirmed heights a silent payments batch introduces, demoting to unconfirmed in place those the connected server would not prove.
+ * These transactions are new to the wallet by construction, so unlike the history path there is nothing here already stored to compare against.
+ */
+ Map<BlockTransactionHash, BlockHeader> verifySilentPaymentReferences(Wallet wallet, Map<BlockTransactionHash, Transaction> referencesToFetch) throws ServerException {
+ Set<BlockTransactionHash> toProve = referencesToFetch.keySet().stream().filter(reference -> reference.getHeight() > 0)
+ .collect(Collectors.toCollection(LinkedHashSet::new));
+ if(toProve.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ Map<BlockTransactionHash, BlockHeader> proven;
+ try {
+ proven = verifyMerkleProofs(wallet, toProve);
+ } catch(UnsupportedMethodException e) {
+ disableVerification(e);
+ return Collections.emptyMap();
+ } catch(ProofsUnavailableException e) {
+ disableVerification(e);
+ return Collections.emptyMap();
+ }
+
+ for(BlockTransactionHash reference : toProve) {
+ if(!proven.containsKey(reference)) {
+ Transaction transaction = referencesToFetch.remove(reference);
+ referencesToFetch.put(new BlockTransaction(reference.getHash(), 0, null, reference.getFee(), null), transaction);
+ }
+ }
+
+ return proven;
+ }
+
+ /**
+ * Handles the connected server never answering a request for proofs at all, rather than failing to prove a particular transaction. Where
+ * verification is mandatory the history fails and the public server rotates; elsewhere the session proceeds unverified, since denying a wallet its
+ * history is worse than not verifying it against a server its owner chose. A lost connection arrives as the same exception and is not this, so the
+ * transport is asked: with the connection gone it is the ordinary reconnect's business.
+ */
+ private static void disableVerification(ProofsUnavailableException e) throws ServerException {
+ if(isVerificationMandatory() || !isConnected()) {
+ throw e;
+ }
+
+ log.warn("Server could not supply transaction proofs, disabling transaction verification for this session: " + e.getMessage());
+ serverCapability.withMerkleProofs(false);
+ }
+
+ /**
+ * Handles a server that turns out not to implement a call verification needs, on the same terms: the state the capability mapping would have
+ * produced had it known.
+ */
+ private static void disableVerification(UnsupportedMethodException e) throws ServerException {
+ if(isVerificationMandatory()) {
+ throw new ServerException("Server does not support transaction verification (" + e.getMethod() + ")", e);
+ }
+
+ log.warn("Server does not support " + e.getMethod() + ", disabling transaction verification for this session");
+ serverCapability.withMerkleProofs(false);
+ }
+
+ /**
+ * Replaces each of the given references with an unconfirmed one in every node that holds it, which is how an unproven height is kept out of the
+ * wallet: it flows on through both sinks as an ordinary mempool transaction rather than through a path of its own.
+ */
+ private void demoteReferences(Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap, Set<BlockTransactionHash> unproven) {
+ for(Map.Entry<WalletNode, Set<BlockTransactionHash>> entry : nodeTransactionMap.entrySet()) {
+ Set<BlockTransactionHash> references = entry.getValue();
+ if(!references.isEmpty()) {
+ for(BlockTransactionHash reference : unproven) {
+ if(references.remove(reference)) {
+ references.add(new BlockTransaction(reference.getHash(), 0, null, reference.getFee(), null));
+ //Recorded for the changed history check in fetchAndCalculateWalletHistory, which this node would otherwise trip
+ demotedScriptHashes.add(getScriptHash(entry.getKey()));
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Whether a stored transaction is held at a height that is still numerically what the server reports but was proven against a block the chain no
+ * longer holds, which is the form a reorg takes when a transaction is re-included at the same height. Above the last fork point this session only,
+ * so an ordinary pass reads neither the store nor the disk.
+ */
+ private boolean isProvenAgainstOrphanedHeader(BlockTransaction existing) throws ServerException {
+ if(existing.getBlockHash() == null || existing.getHeight() <= lastReorgForkHeight) {
+ return false; //history from before this feature, or a height no reorg this session has reached
+ }
+
+ try {
+ BlockHeader stored = getHeaderStore().getHeader(existing.getHeight());
+ return stored != null && !stored.getHash().equals(existing.getBlockHash());
+ } catch(IOException e) {
+ throw new ServerException("Could not read the block header store", e);
+ }
+ }
+
+ /**
+ * Verifies an inclusion proof for each given (transaction, height) pair, returning those proven with the header they were proven against. What is
+ * missing was refused or shown false, told apart by what the server did rather than said: one unable to keep up fails whole batches and recovers
+ * within the retries, while one that cannot substantiate a pair leaves it unanswered while its siblings succeed.
+ */
+ private Map<BlockTransactionHash, BlockHeader> verifyMerkleProofs(Wallet wallet, Set<BlockTransactionHash> toProve) throws ServerException {
+ Map<String, BlockTransactionHash> outstanding = new LinkedHashMap<>();
+ toProve.forEach(reference -> outstanding.put(reference.getHashAsString() + ":" + reference.getHeight(), reference));
+ Map<BlockTransactionHash, BlockHeader> proven = new HashMap<>();
+ //Accumulated here and merged out only on return. What is classified below is demoted by the caller on that return and on no other exit, so
+ //writing it as we go would let an exception mid-loop report a finding without the demotion it describes - telling the user a transaction is
+ //shown as unconfirmed while it is written confirmed
+ Set<BlockTransactionHash> failedProofs = new LinkedHashSet<>();
+ Set<BlockTransactionHash> refusedProofs = new LinkedHashSet<>();
+ prefetchVerifiedHeaders(toProve.stream().map(BlockTransactionHash::getHeight).toList());
+
+ ElectrumServerRpcException batchFailure = null;
+ boolean answered = false;
+ for(int attempt = 0; attempt < proofAttempts && !outstanding.isEmpty(); attempt++) {
+ if(attempt > 0) {
+ try {
+ //Jittered, so that many clients refused by one overloaded server do not retry in step
+ TimeUnit.MILLISECONDS.sleep(proofRetryDelayMillis + new Random().nextInt((int)proofRetryDelayMillis + 1));
+ } catch(InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new ServerException("Interrupted while verifying transactions", e); //the task was cancelled, so fail it rather than report refusals
+ }
+ }
+
+ Map<String, TransactionMerkleProof> proofs;
+ try {
+ proofs = electrumServerRpc.getTransactionMerkleProofs(getTransport(), wallet, outstanding.values());
+ batchFailure = null;
+ answered = true;
+ } catch(UnsupportedMethodException e) {
+ throw e; //before the catch below: whether the server implements the call is settled, and no retry will change it
+ } catch(ElectrumServerRpcException e) {
+ //The whole call failing says nothing about any one transaction, and a server momentarily unable to answer looks exactly like one
+ //that never will until the retries have been spent. Left to the loop, which is what tells the two apart
+ batchFailure = e;
+ continue;
+ }
+
+ for(Map.Entry<String, TransactionMerkleProof> entry : proofs.entrySet()) {
+ BlockTransactionHash reference = outstanding.get(entry.getKey());
+ TransactionMerkleProof proof = entry.getValue();
+ //An error, or a proof for some other block, leaves the pair outstanding: the server has not substantiated the height it reported
+ if(reference != null && proof != TransactionMerkleProof.ERROR_PROOF && proof.block_height == reference.getHeight()) {
+ outstanding.remove(entry.getKey());
+ BlockHeader header = getVerifiedHeader(reference.getHeight());
+ if(header == null) {
+ refusedProofs.add(reference); //the height itself cannot be substantiated
+ } else if(verifyProof(reference.getHash(), proof, header)) {
+ proven.put(reference, header);
+ } else {
+ failedProofs.add(reference); //the branch does not reconstruct against a verified header
+ }
+ }
+ }
+ }
+
+ if(batchFailure != null) {
+ //Every attempt failed as a whole rather than per transaction, so nothing here has been refused. Never answered at all is a server unable
+ //to serve the call, which the callers may work around; answered and then not is one that can, so it fails the pass like any other
+ if(answered) {
+ throw new ServerException(batchFailure.getMessage(), batchFailure.getCause());
+ }
+
+ throw new ProofsUnavailableException(batchFailure.getMessage(), batchFailure.getCause());
+ }
+
+ //Reported at this height, then not substantiated at it through every attempt
+ refusedProofs.addAll(outstanding.values());
+
+ if(!failedProofs.isEmpty()) {
+ failed.computeIfAbsent(wallet, w -> new LinkedHashSet<>()).addAll(failedProofs);
+ }
+ if(!refusedProofs.isEmpty()) {
+ refused.computeIfAbsent(wallet, w -> new LinkedHashSet<>()).addAll(refusedProofs);
+ }
+
+ return proven;
+ }
+
+ /**
+ * Whether the branch reconstructs the merkle root of the given verified header from the given transaction. Any malformed proof is a proof that
+ * does not verify rather than a broken session.
+ */
+ static boolean verifyProof(Sha256Hash txid, TransactionMerkleProof proof, BlockHeader header) {
+ if(proof.merkle == null || proof.merkle.size() > MerkleBranch.MAX_DEPTH) {
+ return false;
+ }
+
+ try {
+ MerkleBranch branch = new MerkleBranch(proof.pos, proof.merkle.stream().map(Sha256Hash::wrap).toList());
+ return branch.computeRoot(txid).equals(header.getMerkleRoot());
+ } catch(RuntimeException e) {
+ log.warn("Invalid merkle proof for " + txid + ": " + e.getMessage());
+ return false;
+ }
+ }
+
+ /**
+ * Fetches and verifies, in batched pages, the header ranges below the last pin the given heights need, so getVerifiedHeader serves them from the
+ * session cache. The one header path worth batching: a range averages a difficulty period, and a restore can touch dozens while the user waits.
+ */
+ void prefetchVerifiedHeaders(Collection<Integer> heights) throws ServerException {
+ HeaderCheckpoints checkpoints = Network.get().getHeaderCheckpoints();
+ Set<Integer> subCheckpointHeights = new TreeSet<>();
+ for(int height : heights) {
+ if(height > 0 && height <= checkpoints.getMaxHeight() && !verifiedHistoricalHeaders.containsKey(height)) {
+ subCheckpointHeights.add(height);
+ }
+ }
+
+ if(subCheckpointHeights.isEmpty()) {
+ return; //every height is above the last pin, and is served by the store the forward sync maintains
+ }
+
+ //Under the same lock as the forward sync and the single range fetch, so that two wallets restoring over the same periods fetch each once
+ synchronized(headerSyncLock) {
+ Map<Integer, Integer> ranges = new TreeMap<>();
+ for(int height : subCheckpointHeights) {
+ //A height already verified needs nothing, and one an added range covers is fetched by that range. Ranges reach up to the nearest
+ //verified header, so one usually covers a period - but not where part of it was verified already, hence asking rather than assuming
+ if(!verifiedHistoricalHeaders.containsKey(height) && ranges.entrySet().stream()
+ .noneMatch(range -> range.getKey() <= height && height < range.getKey() + range.getValue())) {
+ ranges.put(height, getVerifiedAnchorHeight(height, checkpoints.getPinnedHeightAtOrAbove(height)) - height + 1);
+ }
+ }
+
+ if(ranges.isEmpty()) {
+ return;
+ }
+
+ Map<Integer, BlockHeaders> chunks;
+ try {
+ chunks = electrumServerRpc.getBlockHeadersChunks(getTransport(), ranges);
+ } catch(UnsupportedMethodException e) {
+ throw e;
+ } catch(ElectrumServerRpcException e) {
+ throw new ServerException(e.getMessage(), e.getCause());
+ }
+
+ //A range the server errored on or answered malformed is absent, and one that fails linkage is not cached: either way getVerifiedHeader
+ //fetches it singly, and its heights resolve to refusals in the ordinary way if that fails too
+ for(Map.Entry<Integer, BlockHeaders> chunk : chunks.entrySet()) {
+ verifyAndCacheRange(chunk.getKey(), ranges.get(chunk.getKey()), chunk.getValue());
+ }
+ }
+ }
+
+ /**
+ * Raises one dialog per wallet for what this task could not prove, from a finally so that a later failure in the pass cannot bury a finding whose
+ * demotion has already been written. Each pair is reported once per session, since the passes that follow a refusal re-attempt it.
+ */
+ void postProofEvents(Wallet wallet) {
+ postProofEvents(wallet, wallet);
+ }
+
+ void postProofEvents(Wallet wallet, Wallet reportWallet) {
+ //Logged before the warned set is consulted, so that the log records a finding on every pass it recurs even where its dialog has been shown
+ //once already. Without it a refusal that is filtered leaves no trace at all, and cannot be told from no proof having been asked for
+ Set<BlockTransactionHash> walletFailed = failed.remove(wallet);
+ if(walletFailed != null && !walletFailed.isEmpty()) {
+ log.warn("Inclusion proofs from the connected server did not reconstruct the block they were supplied for: " + describePairs(walletFailed));
+ Set<BlockTransactionHash> unwarnedFailed = filterWarned(walletFailed, true);
+ if(!unwarnedFailed.isEmpty()) {
+ EventManager.get().post(new TransactionProofsFailedEvent(reportWallet, unwarnedFailed));
+ }
+ }
+
+ Set<BlockTransactionHash> walletRefused = refused.remove(wallet);
+ if(walletRefused != null && !walletRefused.isEmpty()) {
+ log.warn("The connected server would not prove the heights it reported for: " + describePairs(walletRefused));
+ Set<BlockTransactionHash> unwarnedRefused = filterWarned(walletRefused, false);
+ if(!unwarnedRefused.isEmpty()) {
+ EventManager.get().post(new TransactionProofsRefusedEvent(reportWallet, unwarnedRefused));
+ }
+ }
+ }
+
+ private static String describePairs(Set<BlockTransactionHash> references) {
+ return references.stream().map(reference -> reference.getHashAsString() + ":" + reference.getHeight()).collect(Collectors.joining(", "));
+ }
+
+ /**
+ * The findings not yet shown to the user, recording what it returns as shown. A pair shown false is raised even where it was refused on an earlier
+ * pass, the two being different claims; a refusal of a pair already shown false is not, saying less than what has been said. Within one pass the
+ * order of the two posts above already gets this right.
+ */
+ private static Set<BlockTransactionHash> filterWarned(Set<BlockTransactionHash> references, boolean shownFalse) {
+ return references.stream().filter(reference -> {
+ String pair = reference.getHashAsString() + ":" + reference.getHeight();
+ if(shownFalse) {
+ proofWarnedPairs.add(pair);
+ return proofsShownFalseWarnedPairs.add(pair);
+ }
+
+ return proofWarnedPairs.add(pair);
+ }).collect(Collectors.toCollection(LinkedHashSet::new));
+ }
+
public Map<Integer, BlockHeader> getBlockHeaders(Wallet wallet, Set<BlockTransactionHash> references) throws ServerException {
try {
Map<Integer, BlockHeader> blockHeaderMap = new TreeMap<>();
Set<Integer> blockHeights = new TreeSet<>();
for(BlockTransactionHash reference : references) {
if(reference.getHeight() > 0) {
- if(retrievedBlockHeaders.containsKey(reference.getHeight())) {
- blockHeaderMap.put(reference.getHeight(), retrievedBlockHeaders.get(reference.getHeight()));
- } else {
- blockHeights.add(reference.getHeight());
- }
+ blockHeights.add(reference.getHeight());
+ }
+ }
+
+ //Every height proven this pass already has its verified header in the store or the session cache, so serving timestamps from there is
+ //what leaves a confirmed wallet transaction emitting the same single call a recent transaction's confirmation does. Asked before
+ //retrievedBlockHeaders, and not copied into it: that cache holds every announced tip and is never rewound, so after a reorg it names
+ //the replaced block at any height that was one, while a header read from the store cannot be stale that way
+ putVerifiedHeaders(blockHeaderMap, blockHeights);
+
+ for(Iterator<Integer> iter = blockHeights.iterator(); iter.hasNext(); ) {
+ Integer blockHeight = iter.next();
+ BlockHeader retrievedHeader = retrievedBlockHeaders.get(blockHeight);
+ if(retrievedHeader != null) {
+ blockHeaderMap.put(blockHeight, retrievedHeader);
+ iter.remove();
}
}
@@ -911,6 +1345,32 @@ public Map<Integer, BlockHeader> getBlockHeaders(Wallet wallet, Set<BlockTransac
}
}
+ /**
+ * Serves what it can of the given heights from the store and from the session cache of headers linked to a pin, removing those it serves from the
+ * set of heights left to fetch. Nothing here fetches anything: a viewer looking up a date must not set a header range downloading, so the store is
+ * read only where it is already loaded and already covers the height, and a store that cannot be read is simply not used.
+ */
+ private static void putVerifiedHeaders(Map<Integer, BlockHeader> blockHeaderMap, Set<Integer> blockHeights) {
+ try {
+ HeaderStore store = headerStore;
+ HeaderStore readable = store != null && store.isIntact() ? store : null;
+ for(Iterator<Integer> iter = blockHeights.iterator(); iter.hasNext(); ) {
+ int height = iter.next();
+ BlockHeader header = verifiedHistoricalHeaders.get(height);
+ if(header == null && readable != null && height >= readable.getStartHeight() && height <= readable.getTipHeight()) {
+ header = readable.getHeader(height);
+ }
+
+ if(header != null) {
+ blockHeaderMap.put(height, header);
+ iter.remove();
+ }
+ }
+ } catch(IOException e) {
+ log.warn("Could not read the block header store: " + e.getMessage());
+ }
+ }
+
/**
* The header store for this network, loaded on first use from a background thread. It is not cleared when the server changes: headers are claims
* about the chain rather than about the server, and a new server announcing a different tip is handled as an ordinary reorg.
@@ -1064,9 +1524,13 @@ private void reconcile(HeaderStore store, int tipHeight) throws ServerException,
+ (store.getTipHeight() - forkHeight) + " verified headers it would replace");
}
- log.info("Reorganising the block header store at height " + forkHeight + ", replacing " + (store.getTipHeight() - forkHeight) + " headers with " + segment.size());
+ log.warn("Reorganising the block header store at height " + forkHeight + ", replacing " + (store.getTipHeight() - forkHeight) + " headers with " + segment.size());
store.truncate(forkHeight);
lastReorgForkHeight = Math.min(lastReorgForkHeight, forkHeight);
+ //Every announced tip is cached there and nothing else rewinds it, so above the fork it would name the replaced block at any height that was
+ //one - only the current tip is replaced by the next announcement. Dropped with the headers they came from
+ int reorganisedFrom = forkHeight;
+ retrievedBlockHeaders.keySet().removeIf(height -> height > reorganisedFrom);
try {
store.append(segment);
} finally {
@@ -1127,22 +1591,7 @@ public BlockHeader getVerifiedHeader(int height) throws ServerException {
return cached;
}
- //A header whose hash chain reaches a verified hash is that hash's ancestor at the corresponding depth, so linkage is the whole proof. The
- //anchor is the nearest header already verified this session, and the pin above the height where there is none, which keeps a second pass
- //over an already fetched period from downloading it again
- int pinnedHeight = checkpoints.getPinnedHeightAtOrAbove(height);
- int anchorHeight = pinnedHeight;
- Sha256Hash anchorHash = checkpoints.getHash(pinnedHeight);
- for(int above = height + 1; above < pinnedHeight; above++) {
- BlockHeader verified = verifiedHistoricalHeaders.get(above);
- if(verified != null) {
- anchorHeight = above;
- anchorHash = verified.getHash();
- break;
- }
- }
-
- int count = anchorHeight - height + 1;
+ int count = getVerifiedAnchorHeight(height, checkpoints.getPinnedHeightAtOrAbove(height)) - height + 1;
BlockHeaders chunk;
try {
chunk = electrumServerRpc.getBlockHeadersChunk(getTransport(), height, count);
@@ -1154,17 +1603,43 @@ public BlockHeader getVerifiedHeader(int height) throws ServerException {
throw new ServerException(e.getMessage(), e.getCause());
}
- List<BlockHeader> headers = getLinkedHeaders(chunk, count, anchorHash);
- if(headers == null) {
- return null;
- }
+ return verifyAndCacheRange(height, count, chunk);
+ }
+ }
- for(int i = 0; i < count; i++) {
- verifiedHistoricalHeaders.put(height + i, headers.get(i));
+ /**
+ * The height of the header nearest above the given one that has already been verified this session, and the pinned height where there is none.
+ * A header whose hash chain reaches a verified hash is that hash's ancestor at the corresponding depth, so linking to the nearest verified header
+ * rather than always to the pin is what keeps a later pass over an already fetched period from downloading it again.
+ */
+ private static int getVerifiedAnchorHeight(int height, int pinnedHeight) {
+ for(int above = height + 1; above < pinnedHeight; above++) {
+ if(verifiedHistoricalHeaders.containsKey(above)) {
+ return above;
}
+ }
+
+ return pinnedHeight;
+ }
+
+ /**
+ * Verifies a fetched range of headers below the last pin by hash linkage to its anchor and caches it for the session, returning the header the
+ * range starts at, or null where the range is short, malformed or does not reach the anchor. Called with headerSyncLock held.
+ */
+ private static BlockHeader verifyAndCacheRange(int startHeight, int count, BlockHeaders chunk) {
+ int anchorHeight = startHeight + count - 1;
+ BlockHeader anchor = verifiedHistoricalHeaders.get(anchorHeight);
+ Sha256Hash anchorHash = anchor != null ? anchor.getHash() : Network.get().getHeaderCheckpoints().getHash(anchorHeight);
+ List<BlockHeader> headers = getLinkedHeaders(chunk, count, anchorHash);
+ if(headers == null) {
+ return null;
+ }
- return headers.getFirst();
+ for(int i = 0; i < count; i++) {
+ verifiedHistoricalHeaders.put(startHeight + i, headers.get(i));
}
+
+ return headers.getFirst();
}
/**
@@ -1199,23 +1674,49 @@ static List<BlockHeader> getLinkedHeaders(BlockHeaders chunk, int count, Sha256H
}
/**
- * Whether transactions are being verified against the connected server, which turns on the header sync and the inclusion proofs alike.
+ * Whether transactions are being verified against the connected server, which turns on the header sync and the inclusion proofs alike. The config
+ * setting is asked here so one answer covers the sync, both write boundaries and the connect time enforcement.
+ * <p>
+ * A server below the last pinned header cannot substantiate any height, so asking would refuse every new confirmation and raise a dialog for it.
+ * The public tier rejects such a server at connect; a private one still catching up simply goes unverified until it arrives. A tip not yet
+ * announced is not evidence of lagging.
+ * <p>
+ * Not asked of a Bitcoin Core connection at all, whichever backend is fronting it: the node answering is the user's own, and a proof it built
+ * against headers it also supplied establishes nothing it has not already been trusted for. Cormorant declares as much in its capability, but
+ * bwt takes over where cormorant cannot start, and the same node should not verify or not according to which one did.
*/
static boolean isVerifyingTransactions() {
- return serverCapability != null && serverCapability.supportsMerkleProofs();
+ if(!Config.get().isVerifyTransactions() || Config.get().getServerType() == ServerType.BITCOIN_CORE
+ || serverCapability == null || !serverCapability.supportsMerkleProofs()) {
+ return false;
+ }
+
+ ChainTip announced = AppServices.getAnnouncedTip();
+ return announced == null || announced.height() >= Network.get().getHeaderCheckpoints().getMaxHeight();
}
/**
* Whether the connected server must support transaction verification to be used at all, which is the case for the public server tier on mainnet.
+ * Turned off with verification itself, or a public server would still be rejected for lacking a call nothing is going to make.
*/
static boolean isVerificationMandatory() {
- return Config.get().getServerType() == ServerType.PUBLIC_ELECTRUM_SERVER && Network.get() == Network.MAINNET;
+ return Config.get().isVerifyTransactions() && Config.get().getServerType() == ServerType.PUBLIC_ELECTRUM_SERVER && Network.get() == Network.MAINNET;
}
public Map<Sha256Hash, BlockTransaction> getTransactions(Wallet wallet, Map<BlockTransactionHash, Transaction> references, Map<Integer, BlockHeader> blockHeaderMap) throws ServerException {
+ return getTransactions(wallet, references, blockHeaderMap, Collections.emptyMap());
+ }
+
+ /**
+ * Builds the wallet transactions the given references name, recording on each pair proven this pass the hash of the block it was proven against,
+ * which is what identifies it later as held in a block the chain no longer has.
+ */
+ public Map<Sha256Hash, BlockTransaction> getTransactions(Wallet wallet, Map<BlockTransactionHash, Transaction> references, Map<Integer, BlockHeader> blockHeaderMap,
+ Map<BlockTransactionHash, BlockHeader> proven) throws ServerException {
try {
Map<Sha256Hash, BlockTransaction> transactionMap = new HashMap<>();
Set<BlockTransactionHash> checkReferences = new TreeSet<>(references.keySet());
+ Set<Sha256Hash> provenTxids = new HashSet<>();
Set<String> txids = new LinkedHashSet<>(references.size());
for(BlockTransactionHash reference : references.keySet()) {
@@ -1252,13 +1753,14 @@ public Map<Sha256Hash, BlockTransaction> getTransactions(Wallet wallet, Map<Bloc
throw new IllegalStateException("Server returned a transaction that does not match the requested txid " + hash);
}
- Optional<BlockTransactionHash> optionalReference = references.keySet().stream().filter(reference -> reference.getHash().equals(hash)).findFirst();
- if(optionalReference.isEmpty()) {
+ //One transaction can be referenced at more than one height, and each of those references needs it: left without it, the second
+ //would be built as unfetchable and would overwrite the first in the map below, which is keyed by transaction alone
+ List<BlockTransactionHash> matching = references.keySet().stream().filter(reference -> reference.getHash().equals(hash)).toList();
+ if(matching.isEmpty()) {
throw new IllegalStateException("Returned transaction " + hash.toString() + " that was not requested");
}
- BlockTransactionHash reference = optionalReference.get();
- references.put(reference, transaction);
+ matching.forEach(reference -> references.put(reference, transaction));
}
}
@@ -1281,16 +1783,27 @@ public Map<Sha256Hash, BlockTransaction> getTransactions(Wallet wallet, Map<Bloc
blockDate = blockHeader.getTimeAsDate();
}
+ BlockTransaction cached = wallet == null ? null : wallet.getWalletTransaction(reference.getHash());
Long fee = reference.getFee();
- if(fee == null && wallet != null) {
- BlockTransaction cached = wallet.getWalletTransaction(reference.getHash());
- if(cached != null && cached.getFee() != null) {
- fee = cached.getFee();
- }
+ if(fee == null && cached != null && cached.getFee() != null) {
+ fee = cached.getFee();
}
- BlockTransaction blockchainTransaction = new BlockTransaction(reference.getHash(), reference.getHeight(), blockDate, fee, transaction);
- transactionMap.put(reference.getHash(), blockchainTransaction);
+ //An existing block hash is carried over only while the height it was proven at is unchanged, since a height that has changed has just
+ //been proven against a different block or demoted to unconfirmed
+ BlockHeader provenHeader = proven.get(reference);
+ Sha256Hash blockHash = provenHeader != null ? provenHeader.getHash() :
+ (cached != null && cached.getHeight() == reference.getHeight() ? cached.getBlockHash() : null);
+ BlockTransaction blockchainTransaction = new BlockTransaction(reference.getHash(), reference.getHeight(), blockDate, fee, transaction, blockHash);
+
+ //Two references for one transaction collapse to one entry here, and the proven pair is the one that must survive: the other is a
+ //height this server would not prove, which the demotion has already replaced with an unconfirmed reference
+ if(provenHeader != null || !provenTxids.contains(reference.getHash())) {
+ transactionMap.put(reference.getHash(), blockchainTransaction);
+ }
+ if(provenHeader != null) {
+ provenTxids.add(reference.getHash());
+ }
checkReferences.remove(reference);
}
@@ -2019,10 +2532,16 @@ public Set<WalletNode> processSilentPaymentBatch(Wallet wallet, List<SilentPayme
}
if(!referencesToFetch.isEmpty()) {
- Map<Integer, BlockHeader> blockHeaderMap = getBlockHeaders(wallet, referencesToFetch.keySet());
- Map<Sha256Hash, BlockTransaction> fetched = getTransactions(wallet, referencesToFetch, blockHeaderMap);
- transactionMap.putAll(fetched);
- wallet.updateTransactions(fetched);
+ try {
+ //The second write boundary: these heights come from the subscription rather than from a history call, and reach the wallet just the same
+ Map<BlockTransactionHash, BlockHeader> proven = isVerifyingTransactions() ? verifySilentPaymentReferences(wallet, referencesToFetch) : Collections.emptyMap();
+ Map<Integer, BlockHeader> blockHeaderMap = getBlockHeaders(wallet, referencesToFetch.keySet());
+ Map<Sha256Hash, BlockTransaction> fetched = getTransactions(wallet, referencesToFetch, blockHeaderMap, proven);
+ transactionMap.putAll(fetched);
+ wallet.updateTransactions(fetched);
+ } finally {
+ postProofEvents(wallet);
+ }
}
ECKey scanPriv = wallet.getSilentPaymentScanAddress().getScanKey();
@@ -2668,6 +3187,28 @@ public void walletNodeHistoryChanged(WalletNodeHistoryChangedEvent event) {
log.debug("Error subscribing to recent mempool transaction outputs", e);
}
}
+
+ BlockTransactionHash reference = getProofReference(event);
+ if(reference != null) {
+ TransactionProofService transactionProofService = new TransactionProofService(reference);
+ transactionProofService.start();
+ }
+ }
+
+ static BlockTransactionHash getProofReference(WalletNodeHistoryChangedEvent event) {
+ BlockTransaction blkTx = confirmingRecent.get(event.getScriptHash());
+ Integer currentHeight = AppServices.getCurrentBlockHeight();
+ if(blkTx == null || currentHeight == null || !isVerifyingTransactions()) {
+ return null;
+ }
+
+ String confirmedStatus = getScriptHashStatus(List.of(new ScriptHashTx(currentHeight, blkTx.getHashAsString(), blkTx.getFee())));
+ if(!Objects.equals(confirmedStatus, event.getStatus())) {
+ return null;
+ }
+
+ confirmingRecent.remove(event.getScriptHash());
+ return new BlockTransaction(blkTx.getHash(), currentHeight, null, blkTx.getFee(), null);
}
}
@@ -3080,6 +3621,33 @@ protected List<BlockTransaction> call() throws ServerException {
}
}
+ public static class TransactionProofService extends Service<Map<String, TransactionMerkleProof>> {
+ private final BlockTransactionHash reference;
+
+ public TransactionProofService(BlockTransactionHash reference) {
+ this.reference = reference;
+ }
+
+ @Override
+ protected Task<Map<String, TransactionMerkleProof>> createTask() {
+ return new Task<>() {
+ @Override
+ protected Map<String, TransactionMerkleProof> call() {
+ try {
+ //getTransport() opens one where there is none, so a task running after the connection closed must not reach it
+ if(isConnected()) {
+ return electrumServerRpc.getTransactionMerkleProofs(getTransport(), null, List.of(reference));
+ }
+ } catch(Exception e) {
+ log.debug("Error retrieving proof for transaction", e);
+ }
+
+ return Collections.emptyMap();
+ }
+ };
+ }
+ }
+
public static class BroadcastTransactionService extends Service<Sha256Hash> {
private final Transaction transaction;
private final Long fee;
@@ -3180,15 +3748,18 @@ private void subscribeRecent(ElectrumServer electrumServer, int currentHeight) {
}
subscribedRecent.keySet().removeAll(unsubscribeScriptHashes);
broadcastRecent.keySet().removeAll(unsubscribeScriptHashes);
+ confirmingRecent.keySet().removeAll(unsubscribeScriptHashes);
Map<String, String> subscribeScriptHashes = new HashMap<>();
+ Map<String, BlockTransaction> confirming = new HashMap<>();
List<BlockTransaction> recentTransactions = electrumServer.getRecentMempoolTransactions();
for(BlockTransaction blkTx : recentTransactions) {
for(int i = 0; i < blkTx.getTransaction().getOutputs().size(); i++) {
TransactionOutput txOutput = blkTx.getTransaction().getOutputs().get(i);
String scriptHash = getScriptHash(txOutput);
if(!subscribedScriptHashes.containsKey(scriptHash)) {
subscribeScriptHashes.put("m/" + subscribeScriptHashes.size(), scriptHash);
+ confirming.put(scriptHash, blkTx);
}
if(Math.random() < 0.1d) {
break;
@@ -3211,6 +3782,7 @@ private void subscribeRecent(ElectrumServer electrumServer, int currentHeight) {
try {
electrumServerRpc.subscribeScriptHashes(transport, null, subscribeScriptHashes);
subscribeScriptHashes.values().forEach(scriptHash -> subscribedRecent.put(scriptHash, currentHeight));
+ confirmingRecent.putAll(confirming);
} catch(ElectrumServerRpcException e) {
log.debug("Error subscribing to recent mempool transactions", e);
}
@@ -3433,9 +4005,15 @@ protected List<Wallet> call() throws ServerException {
addCalculatedScriptHashes(notificationNode);
ElectrumServer electrumServer = new ElectrumServer();
- Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = electrumServer.getHistory(notificationWallet, List.of(notificationNode));
- electrumServer.getReferencedTransactions(notificationWallet, nodeTransactionMap);
- electrumServer.calculateNodeHistory(notificationWallet, nodeTransactionMap);
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap;
+ try {
+ nodeTransactionMap = electrumServer.getHistory(notificationWallet, List.of(notificationNode));
+ electrumServer.getReferencedTransactions(notificationWallet, nodeTransactionMap);
+ electrumServer.calculateNodeHistory(notificationWallet, nodeTransactionMap);
+ } finally {
+ //The notification wallet is derived rather than opened, so what it could not prove is reported against the wallet it belongs to
+ electrumServer.postProofEvents(notificationWallet, wallet);
+ }
List<Wallet> addedWallets = new ArrayList<>();
if(!nodeTransactionMap.isEmpty()) {
### src/main/java/com/sparrowwallet/sparrow/net/ProofsUnavailableException.java
@@ -0,0 +1,14 @@
+package com.sparrowwallet.sparrow.net;
+
+/**
+ * Thrown where the connected server did not answer a request for inclusion proofs at all, on any attempt, for a reason other than not implementing the
+ * call - a batch it will not accept, a size it will not return, an error of its own. Nothing has been proven or disproven, and nothing refused.
+ * <p>
+ * A server that answered and then stopped is deliberately not this: it has shown it can serve the call, so what followed is to be retried rather than
+ * worked around, and it may already have proven a transaction false - not a finding to trade away for an unverified session.
+ */
+public class ProofsUnavailableException extends ServerException {
+ public ProofsUnavailableException(String message, Throwable cause) {
+ super(message, cause);
+ }
+}
### src/main/java/com/sparrowwallet/sparrow/wallet/WalletForm.java
@@ -2,6 +2,7 @@
import com.google.common.eventbus.Subscribe;
import com.sparrowwallet.drongo.KeyPurpose;
+import com.sparrowwallet.drongo.protocol.Sha256Hash;
import com.sparrowwallet.drongo.policy.PolicyType;
import com.sparrowwallet.drongo.silentpayments.SilentPaymentScanAddress;
import com.sparrowwallet.drongo.wallet.*;
@@ -327,6 +328,7 @@ private List<WalletNode> notifyIfChanged(Integer blockHeight, Wallet currentWall
List<WalletNode> historyChangedNodes = new ArrayList<>();
historyChangedNodes.addAll(getHistoryChangedNodes(previousWallet.getNode(KeyPurpose.RECEIVE).getChildren(), currentWallet.getNode(KeyPurpose.RECEIVE).getChildren()));
historyChangedNodes.addAll(getHistoryChangedNodes(previousWallet.getNode(KeyPurpose.CHANGE).getChildren(), currentWallet.getNode(KeyPurpose.CHANGE).getChildren()));
+ addReprovenNodes(currentWallet, previousWallet, historyChangedNodes);
boolean changed = false;
if(!historyChangedNodes.isEmpty() || !nestedHistoryChangedNodes.isEmpty()) {
@@ -348,6 +350,35 @@ private List<WalletNode> notifyIfChanged(Integer blockHeight, Wallet currentWall
return historyChangedNodes;
}
+ /**
+ * Adds the nodes holding a transaction proven against a different block at the height it was already held at. A block replaced by another
+ * containing the same transaction changes neither its height nor any output, so the comparison above cannot see it, yet its block hash and its
+ * date - that block's timestamp - are both stale until the nodes holding it are written again.
+ * <p>
+ * The height being unchanged is required, not merely typical: a block hash follows the height, so it changes on every ordinary confirmation,
+ * demotion and unfetchable transaction too, and in each of those the node already has an output at a new height and has been reported.
+ */
+ static void addReprovenNodes(Wallet currentWallet, Wallet previousWallet, List<WalletNode> historyChangedNodes) {
+ Set<Sha256Hash> reproven = new HashSet<>();
+ for(Map.Entry<Sha256Hash, BlockTransaction> entry : currentWallet.getTransactions().entrySet()) {
+ BlockTransaction previousTransaction = previousWallet.getTransactions().get(entry.getKey());
+ if(previousTransaction != null && previousTransaction.getHeight() == entry.getValue().getHeight()
+ && !Objects.equals(previousTransaction.getBlockHash(), entry.getValue().getBlockHash())) {
+ reproven.add(entry.getKey());
+ }
+ }
+
+ if(!reproven.isEmpty()) {
+ Set<WalletNode> reportedNodes = new HashSet<>(historyChangedNodes);
+ for(Map.Entry<WalletNode, Set<BlockTransactionHashIndex>> entry : currentWallet.getWalletNodes().entrySet()) {
+ if(!reportedNodes.contains(entry.getKey()) && entry.getValue().stream()
+ .anyMatch(txo -> reproven.contains(txo.getHash()) || (txo.isSpent() && reproven.contains(txo.getSpentBy().getHash())))) {
+ historyChangedNodes.add(entry.getKey());
+ }
+ }
+ }
+ }
+
private List<WalletNode> getHistoryChangedNodes(Set<WalletNode> previousNodes, Set<WalletNode> currentNodes) {
Map<String, WalletNode> previousNodeMap = new HashMap<>(previousNodes.size());
previousNodes.forEach(walletNode -> previousNodeMap.put(walletNode.getDerivationPath(), walletNode));
@@ -575,6 +606,13 @@ public void chainReorg(ChainReorgEvent event) {
}
}
+ @Subscribe
+ public void requestWalletRefresh(RequestWalletRefreshEvent event) {
+ if(wallet.isValid() && !wallet.isNested() && wallet.equals(event.getWallet().resolveMasterWallet())) {
+ Platform.runLater(() -> refreshHistory(AppServices.getCurrentBlockHeight()));
+ }
+ }
+
@Subscribe
public void walletNodeHistoryChanged(WalletNodeHistoryChangedEvent event) {
if(wallet.isValid() && !wallet.isNested()) {
### src/test/java/com/sparrowwallet/sparrow/net/ElectrumServerRpcTest.java
@@ -1,10 +1,28 @@
package com.sparrowwallet.sparrow.net;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.github.arteam.simplejsonrpc.client.Transport;
import com.sparrowwallet.drongo.protocol.HeaderChainState;
+import com.sparrowwallet.drongo.protocol.Sha256Hash;
import com.sparrowwallet.drongo.protocol.VerificationException;
+import com.sparrowwallet.drongo.wallet.BlockTransaction;
+import com.sparrowwallet.drongo.wallet.BlockTransactionHash;
+import com.sparrowwallet.sparrow.SparrowWallet;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
/**
@@ -14,6 +32,65 @@
public class ElectrumServerRpcTest {
private static final int TIP = 900000;
+ @TempDir
+ private static Path tempHome;
+
+ @BeforeAll
+ public static void setUpAll() {
+ //Config.get() caches its instance statically for the life of the JVM, so keep this test from loading the developer's real config
+ System.setProperty(SparrowWallet.APP_HOME_PROPERTY, tempHome.toString());
+ }
+
+ @AfterAll
+ public static void tearDownAll() {
+ System.clearProperty(SparrowWallet.APP_HOME_PROPERTY);
+ }
+
+ /**
+ * A proof is small enough that its batch page is bounded by round trips rather than by bytes. Restoring a large wallet that has reused addresses
+ * asks for a hundred thousand of them, and the generic page of 100 spends a round trip on each hundred - four minutes of a nineteen minute load,
+ * measured, for about 130MB moved.
+ */
+ @Test
+ public void pagesMerkleProofsByTheirOwnPageSize() {
+ CountingTransport transport = new CountingTransport();
+ List<BlockTransactionHash> references = new ArrayList<>();
+ for(int i = 1; i <= BatchedElectrumServerRpc.MERKLE_BATCH_PAGE_SIZE + 1; i++) {
+ references.add(new BlockTransaction(Sha256Hash.wrap(String.format("%064x", i)), 800000, null, 0L, null));
+ }
+
+ Map<String, TransactionMerkleProof> proofs = new BatchedElectrumServerRpc(0, 25).getTransactionMerkleProofs(transport, null, references);
+
+ assertEquals(references.size(), proofs.size());
+ assertEquals(2, transport.requests, "one page beyond the page size must be two requests, not " + (references.size() / 100 + 1));
+ }
+
+ /**
+ * Answers every request in a batch with a proof, counting the batches it was sent.
+ */
+ private static class CountingTransport implements Transport {
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+
+ private int requests;
+
+ @Override
+ public String pass(String request) throws java.io.IOException {
+ requests++;
+ ArrayNode responses = MAPPER.createArrayNode();
+ for(JsonNode node : MAPPER.readTree(request)) {
+ ObjectNode response = responses.addObject();
+ response.put("jsonrpc", "2.0");
+ response.set("id", node.get("id"));
+ ObjectNode result = response.putObject("result");
+ result.put("block_height", 800000);
+ result.put("pos", 0);
+ result.putArray("merkle");
+ }
+
+ return MAPPER.writeValueAsString(responses);
+ }
+ }
+
@Test
public void acceptsAFullResponse() {
assertDoesNotThrow(() -> ElectrumServerRpc.checkBlockHeaders(headers(2016, 2016, 2016), 800000, 2016, TIP));
### src/test/java/com/sparrowwallet/sparrow/net/ElectrumServerTest.java
@@ -15,10 +15,19 @@
import com.sparrowwallet.drongo.protocol.BlockHeader;
import com.sparrowwallet.drongo.protocol.HeaderChainState;
import com.sparrowwallet.drongo.protocol.Sha256Hash;
+import com.sparrowwallet.sparrow.AppServices;
+import com.sparrowwallet.sparrow.ChainTip;
+import com.sparrowwallet.sparrow.SparrowWallet;
+import com.sparrowwallet.sparrow.io.Config;
+import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
@@ -31,6 +40,9 @@
import static org.junit.jupiter.api.Assertions.assertNull;
public class ElectrumServerTest {
+ @TempDir
+ private static Path tempHome;
+
//A plain BIP32 extended public key, since the wallet only needs to derive addresses to have script hashes
private static final String TEST_XPUB = "xpub6BosfCnifzxcFwrSzQiqu2DBVTshkCXacvNsWGYJVVhhawA7d4R5WSWGFNbi8Aw6ZRc1brxMyWMzG3DSSSSoekkudhUd9yLb6qx39T9nMdj";
@@ -40,6 +52,18 @@ public class ElectrumServerTest {
private static final String BLOCK_800000_HEADER_HEX = "00601d3455bb9fbd966b3ea2dc42d0c22722e4c0c1729fad17210100000000000000000055087fab0c8f3f89f8bcfd4df26c504d81b0a88e04907161838c0c53001af09135edbd64943805175e955e06";
private static final long BLOCK_800000_TIME_SECS = 1690168629L;
+ @BeforeAll
+ public static void setUpAll() {
+ //Config.get() caches its instance for the life of the JVM but resolves the file to write on each flush, so a test changing a setting must
+ //never be able to reach the developer's own Sparrow home. The test task sets this too; this is here for a run that bypasses it
+ System.setProperty(SparrowWallet.APP_HOME_PROPERTY, tempHome.toString());
+ }
+
+ @AfterAll
+ public static void tearDownAll() {
+ System.clearProperty(SparrowWallet.APP_HOME_PROPERTY);
+ }
+
@BeforeEach
public void setUp() {
Network.set(Network.MAINNET);
@@ -174,6 +198,90 @@ public void invalidatesANodeWhoseOutputWasSpentAboveTheFork() {
assertFalse(ElectrumServer.invalidateScriptHashesForReorg(wallet, 800000));
}
+ /**
+ * A server that has not reached the last pinned header cannot substantiate any height: the forward sync has nothing to advance from and every
+ * range below a pin comes back short. Verification therefore does not run against it at all until it catches up, rather than refusing every new
+ * confirmation and raising a dialog for each. The public tier rejects such a server at connect instead; this is what a private one gets.
+ */
+ @Test
+ public void doesNotVerifyAgainstAServerBelowTheLastPin() {
+ ServerCapability previousCapability = ElectrumServer.serverCapability;
+ try {
+ ElectrumServer.serverCapability = new ServerCapability(false, false, false);
+ int maxCheckpointHeight = Network.MAINNET.getHeaderCheckpoints().getMaxHeight();
+ BlockHeader header = Network.MAINNET.getGenesisHeader();
+
+ //A tip that has not been announced yet is not evidence of lagging, so the sync and the proofs are left to find out
+ AppServices.setAnnouncedTip(null);
+ assertTrue(ElectrumServer.isVerifyingTransactions());
+
+ AppServices.setAnnouncedTip(new ChainTip(maxCheckpointHeight - 1, header));
+ assertFalse(ElectrumServer.isVerifyingTransactions());
+
+ AppServices.setAnnouncedTip(new ChainTip(maxCheckpointHeight, header));
+ assertTrue(ElectrumServer.isVerifyingTransactions());
+
+ ElectrumServer.serverCapability.withMerkleProofs(false);
+ assertFalse(ElectrumServer.isVerifyingTransactions());
+ } finally {
+ ElectrumServer.serverCapability = previousCapability;
+ AppServices.setAnnouncedTip(null);
+ }
+ }
+
+ /**
+ * The escape hatch. It defaults on and has no user interface, and one answer has to cover the header sync, both write boundaries and the connect
+ * time enforcement - a public server rejected for lacking a call nothing is going to make would be no use.
+ */
+ @Test
+ public void stopsVerifyingWhereTheConfigTurnsItOff() {
+ ServerCapability previousCapability = ElectrumServer.serverCapability;
+ ServerType previousServerType = Config.get().getServerType();
+ try {
+ ElectrumServer.serverCapability = new ServerCapability(false, false, false);
+ Config.get().setServerType(ServerType.PUBLIC_ELECTRUM_SERVER);
+ assertTrue(Config.get().isVerifyTransactions());
+ assertTrue(ElectrumServer.isVerifyingTransactions());
+ assertTrue(ElectrumServer.isVerificationMandatory());
+
+ Config.get().setVerifyTransactions(false);
+ assertFalse(ElectrumServer.isVerifyingTransactions());
+ assertFalse(ElectrumServer.isVerificationMandatory());
+ } finally {
+ Config.get().setVerifyTransactions(true);
+ Config.get().setServerType(previousServerType);
+ ElectrumServer.serverCapability = previousCapability;
+ AppServices.setAnnouncedTip(null);
+ }
+ }
+
+ /**
+ * A Bitcoin Core connection is the user's own node, and a proof it built against headers it also supplied establishes nothing it has not already
+ * been trusted for. Asked of the server type rather than of the capability, because bwt takes over where cormorant cannot start - a legacy Core
+ * wallet, or an unsupported bitcoind - and it reaches getServerCapability under a version string of its own.
+ */
+ @Test
+ public void doesNotVerifyAgainstTheUsersOwnNode() {
+ ServerCapability previousCapability = ElectrumServer.serverCapability;
+ ServerType previousServerType = Config.get().getServerType();
+ try {
+ //The capability bwt falls through to, which unlike cormorant's says nothing about proofs
+ ElectrumServer.serverCapability = new ServerCapability(false, true, true);
+ assertTrue(ElectrumServer.serverCapability.supportsMerkleProofs());
+
+ Config.get().setServerType(ServerType.ELECTRUM_SERVER);
+ assertTrue(ElectrumServer.isVerifyingTransactions());
+
+ Config.get().setServerType(ServerType.BITCOIN_CORE);
+ assertFalse(ElectrumServer.isVerifyingTransactions());
+ assertFalse(ElectrumServer.isVerificationMandatory());
+ } finally {
+ Config.get().setServerType(previousServerType);
+ ElectrumServer.serverCapability = previousCapability;
+ AppServices.setAnnouncedTip(null);
+ }
+ }
+
private static Wallet testWallet() {
Wallet wallet = new Wallet();
wallet.setPolicyType(PolicyType.SINGLE_HD);
### src/test/java/com/sparrowwallet/sparrow/net/HeaderSyncTest.java
@@ -97,6 +97,7 @@ public void setUp() {
ElectrumServer.headerStore = null;
ElectrumServer.lastReorgForkHeight = Integer.MAX_VALUE;
ElectrumServer.verifiedHistoricalHeaders.clear();
+ ElectrumServer.retrievedBlockHeaders.clear();
previousElectrumServerRpc = ElectrumServer.electrumServerRpc;
previousTransport = ElectrumServer.transport;
//The fake answers without the transport, but getTransport() would otherwise build one from the configured server
@@ -113,10 +114,34 @@ public void tearDown() {
ElectrumServer.headerStore = null;
ElectrumServer.lastReorgForkHeight = Integer.MAX_VALUE;
ElectrumServer.verifiedHistoricalHeaders.clear();
+ ElectrumServer.retrievedBlockHeaders.clear();
AppServices.setAnnouncedTip(null);
Network.set(null);
}
+ /**
+ * The cache of headers the server has announced as its tip is not the store, and nothing else rewinds it: above the fork it names blocks the chain
+ * no longer has, and only the current tip is replaced by the next announcement. Left there it would serve the replaced block's timestamp at any
+ * height that was once a tip, which is the height a transaction re-proven against its replacement is looked up at.
+ */
+ @Test
+ public void dropsAnnouncedHeadersAboveTheForkOnReorganising() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 10, CHAIN_TIME);
+ List<BlockHeader> branch = fork(chain, 9, 1);
+ seedStore(chain, 10);
+ serve(branch);
+ ElectrumServer.updateRetrievedBlockHeaders(8, chain.get(7));
+ ElectrumServer.updateRetrievedBlockHeaders(9, chain.get(8));
+ ElectrumServer.updateRetrievedBlockHeaders(10, chain.get(9));
+
+ new ElectrumServer().syncHeaders(new ChainTip(10, branch.getLast()));
+
+ assertEquals(9, listener.getForkHeight());
+ assertEquals(chain.get(7), ElectrumServer.retrievedBlockHeaders.get(8)); //below the fork, still the chain the store keeps
+ assertEquals(chain.get(8), ElectrumServer.retrievedBlockHeaders.get(9));
+ assertNull(ElectrumServer.retrievedBlockHeaders.get(10)); //the replaced block, dropped with the header it came from
+ }
+
/**
* The tie: a stale block replaced at the store tip height. Equal work is accepted, because a client with one server can only verify against the
* chain that server serves - and because the loop that fetches headers would never look at a tip that is not above its own.
@@ -582,6 +607,81 @@ public void fetchesOnlyAsFarAsTheNearestVerifiedHeader() throws Exception {
assertEquals(2, fake.getChunkRequests());
}
+ /**
+ * The restore time prefetch: the heights a batch of proofs needs below the last pin are coalesced into one range per difficulty period, from the
+ * lowest height needed in it, so that every other height in the period is served from the cache without a request of its own.
+ */
+ @Test
+ public void prefetchesOneRangePerPeriodForTheHeightsBeingProven() throws Exception {
+ Network.set(Network.MAINNET);
+ FakeElectrumServerRpc fake = serveFrom(PERIOD_15_CLOSE, 32248);
+
+ new ElectrumServer().prefetchVerifiedHeaders(List.of(32252, 32249, 32249, 32255));
+
+ assertEquals(1, fake.getChunkRequests());
+ assertEquals(32249, fake.getLastStartHeight());
+ assertEquals(7, fake.getLastCount()); //from the lowest height needed up to the pin
+
+ assertEquals(PERIOD_15_CLOSE.get(4).getHash(), new ElectrumServer().getVerifiedHeader(32252).getHash());
+ assertEquals(PERIOD_15_CLOSE.get(1).getHash(), new ElectrumServer().getVerifiedHeader(32249).getHash());
+ assertEquals(1, fake.getChunkRequests());
+ }
+
+ /**
+ * A range reaches only as far as the nearest header already verified, so it does not necessarily cover the rest of its period: the heights above
+ * that header need a range of their own, which the coalescing must not skip because a lower range shares their period.
+ * <p>
+ * The cache is seeded directly, since nothing ordinary produces that gap - a range is cached whole and ends at a verified header or the pin, so a
+ * period's verified heights are always a run up to it. The coalescing should not have to rely on that.
+ */
+ @Test
+ public void prefetchesTheHeightsAboveAnAlreadyVerifiedHeaderInThePeriod() throws Exception {
+ Network.set(Network.MAINNET);
+ FakeElectrumServerRpc fake = serveFrom(PERIOD_15_CLOSE, 32248);
+ ElectrumServer.verifiedHistoricalHeaders.put(32253, PERIOD_15_CLOSE.get(5));
+
+ new ElectrumServer().prefetchVerifiedHeaders(List.of(32249, 32254));
+
+ //One range up to the verified header, and one for what it leaves above
+ assertEquals(2, fake.getChunkRequests());
+ assertEquals(PERIOD_15_CLOSE.get(1).getHash(), new ElectrumServer().getVerifiedHeader(32249).getHash());
+ assertEquals(PERIOD_15_CLOSE.get(6).getHash(), new ElectrumServer().getVerifiedHeader(32254).getHash());
+ assertEquals(2, fake.getChunkRequests());
+ }
+
+ /**
+ * A prefetched range that cannot be verified is simply not cached, which leaves the heights in it to be fetched singly and refused in the ordinary
+ * way. It is never a failure of the pass, and never a partial cache.
+ */
+ @Test
+ public void leavesAnUnverifiableRangeUncached() throws Exception {
+ Network.set(Network.MAINNET);
+ List<BlockHeader> tampered = new ArrayList<>(PERIOD_15_CLOSE);
+ BlockHeader original = tampered.get(4);
+ tampered.set(4, new BlockHeader(original.getVersion(), original.getPrevBlockHash(), original.getMerkleRoot(), null, original.getTime() + 1,
+ original.getDifficultyTarget(), original.getNonce()));
+ serveFrom(tampered, 32248);
+
+ new ElectrumServer().prefetchVerifiedHeaders(List.of(32250));
+
+ assertTrue(ElectrumServer.verifiedHistoricalHeaders.isEmpty());
+ }
+
+ /**
+ * Heights above the last pin are the store's business, so the prefetch leaves them alone rather than asking for ranges below a pin they do not sit
+ * under.
+ */
+ @Test
+ public void prefetchesNothingForHeightsAboveTheLastPin() throws Exception {
+ Network.set(Network.MAINNET);
+ FakeElectrumServerRpc fake = serveFrom(PERIOD_15_CLOSE, 32248);
+
+ int maxHeight = Network.MAINNET.getHeaderCheckpoints().getMaxHeight();
+ new ElectrumServer().prefetchVerifiedHeaders(List.of(0, maxHeight + 1, maxHeight + 5000));
+
+ assertEquals(0, fake.getChunkRequests());
+ }
+
@Test
public void refusesAHistoricalRangeThatDoesNotLinkToItsPin() throws Exception {
Network.set(Network.MAINNET);
### src/test/java/com/sparrowwallet/sparrow/net/TransactionProofTest.java
@@ -0,0 +1,1342 @@
+package com.sparrowwallet.sparrow.net;
+
+import com.github.arteam.simplejsonrpc.client.Transport;
+import com.google.common.eventbus.Subscribe;
+import com.google.common.net.HostAndPort;
+import com.sparrowwallet.drongo.ExtendedKey;
+import com.sparrowwallet.drongo.KeyDerivation;
+import com.sparrowwallet.drongo.KeyPurpose;
+import com.sparrowwallet.drongo.Network;
+import com.sparrowwallet.drongo.Utils;
+import com.sparrowwallet.drongo.policy.Policy;
+import com.sparrowwallet.drongo.policy.PolicyType;
+import com.sparrowwallet.drongo.protocol.BlockHeader;
+import com.sparrowwallet.drongo.protocol.HeaderChainState;
+import com.sparrowwallet.drongo.protocol.Script;
+import com.sparrowwallet.drongo.protocol.ScriptType;
+import com.sparrowwallet.drongo.protocol.Sha256Hash;
+import com.sparrowwallet.drongo.protocol.Transaction;
+import com.sparrowwallet.drongo.wallet.BlockTransaction;
+import com.sparrowwallet.drongo.wallet.BlockTransactionHash;
+import com.sparrowwallet.drongo.wallet.BlockTransactionHashIndex;
+import com.sparrowwallet.drongo.wallet.Keystore;
+import com.sparrowwallet.drongo.wallet.Wallet;
+import com.sparrowwallet.drongo.wallet.WalletNode;
+import com.sparrowwallet.sparrow.AppServices;
+import com.sparrowwallet.sparrow.ChainTip;
+import com.sparrowwallet.sparrow.EventManager;
+import com.sparrowwallet.sparrow.SparrowWallet;
+import com.sparrowwallet.sparrow.event.TransactionProofsFailedEvent;
+import com.sparrowwallet.sparrow.event.TransactionProofsRefusedEvent;
+import com.sparrowwallet.sparrow.event.WalletNodeHistoryChangedEvent;
+import com.sparrowwallet.sparrow.io.Config;
+import com.sparrowwallet.sparrow.io.Storage;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * The write boundary against a server that answers proofs as the test chooses: which confirmed heights are written, which are demoted to unconfirmed,
+ * and which of the two dialogs each outcome raises. Nothing here is exercised by an honest server, which is the reason it is covered so closely - a
+ * demotion that misfires costs a user their confirmed history, and a refusal reported as dishonesty accuses a server that did nothing wrong.
+ * <p>
+ * Regtest is the network whose trivial target a synthetic chain can be mined against, and its empty checkpoints anchor the store at genesis, so every
+ * height below is one the store itself serves.
+ */
+public class TransactionProofTest {
+ //The same extended key in each encoding, since the wallet only has to derive addresses to have script hashes
+ private static final String TEST_TPUB = "tpubDCBWBScQPGv4Xk3JSbhw6wYYpayMjb2eAYyArpbSqQTbLDpphHGAetB6VQgVeftLML8vDSUEWcC2xDi3qJJ3YCDChJDvqVzpgoYSuT52MhJ";
+
+ private static final String TEST_XPUB = "xpub6BosfCnifzxcFwrSzQiqu2DBVTshkCXacvNsWGYJVVhhawA7d4R5WSWGFNbi8Aw6ZRc1brxMyWMzG3DSSSSoekkudhUd9yLb6qx39T9nMdj";
+
+ private static final long CHAIN_TIME = 1600000000L;
+
+ private static final int CHAIN_LENGTH = 10;
+
+ //The block whose transactions are proven, with room above it for a height the store does not reach
+ private static final int PROVEN_HEIGHT = 5;
+
+ private static final long RECENT_FEE = 1000L;
+
+ @TempDir
+ private static Path tempHome;
+
+ private ElectrumServerRpc previousElectrumServerRpc;
+ private CloseableTransport previousTransport;
+ private ServerCapability previousServerCapability;
+ private int previousProofAttempts;
+ private long previousProofRetryDelayMillis;
+ private ProofListener listener;
+
+ private List<Transaction> blockTransactions;
+ private List<BlockHeader> chain;
+ private FakeProofServer server;
+
+ @BeforeAll
+ public static void setUpAll() {
+ System.setProperty(SparrowWallet.APP_HOME_PROPERTY, tempHome.toString());
+ }
+
+ @AfterAll
+ public static void tearDownAll() {
+ System.clearProperty(SparrowWallet.APP_HOME_PROPERTY);
+ }
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ Network.set(Network.REGTEST);
+ File[] files = Storage.getHeadersDir().listFiles();
+ if(files != null) {
+ for(File file : files) {
+ assertTrue(file.delete());
+ }
+ }
+
+ ElectrumServer.headerStore = null;
+ ElectrumServer.lastReorgForkHeight = Integer.MAX_VALUE;
+ ElectrumServer.verifiedHistoricalHeaders.clear();
+ ElectrumServer.clearPreviousServerState();
+ ElectrumServer.getSubscribedScriptHashes().clear();
+ previousElectrumServerRpc = ElectrumServer.electrumServerRpc;
+ previousTransport = ElectrumServer.transport;
+ previousServerCapability = ElectrumServer.serverCapability;
+ previousProofAttempts = ElectrumServer.proofAttempts;
+ previousProofRetryDelayMillis = ElectrumServer.proofRetryDelayMillis;
+ ElectrumServer.transport = new UnusedTransport();
+ ElectrumServer.serverCapability = new ServerCapability(false, false, false);
+ ElectrumServer.proofRetryDelayMillis = 0; //the retry budget is what is under test, not how long it takes to spend
+ listener = new ProofListener();
+ EventManager.get().register(listener);
+
+ //Four transactions in one block, so a branch has two levels and both concatenation orders are exercised
+ blockTransactions = List.of(transaction(0), transaction(1), transaction(2), transaction(3));
+ chain = mineChain(Network.REGTEST.getGenesisHeader(), CHAIN_LENGTH, merkleRoot(txids(blockTransactions)));
+ server = new FakeProofServer(chain);
+ ElectrumServer.electrumServerRpc = server;
+ seedStore(chain, CHAIN_LENGTH - 1); //one height short of the chain, so a proof above the store tip has somewhere to fail
+ }
+
+ @AfterEach
+ public void tearDown() {
+ EventManager.get().unregister(listener);
+ ElectrumServer.electrumServerRpc = previousElectrumServerRpc;
+ ElectrumServer.transport = previousTransport;
+ ElectrumServer.serverCapability = previousServerCapability;
+ ElectrumServer.proofAttempts = previousProofAttempts;
+ ElectrumServer.proofRetryDelayMillis = previousProofRetryDelayMillis;
+ ElectrumServer.headerStore = null;
+ ElectrumServer.lastReorgForkHeight = Integer.MAX_VALUE;
+ ElectrumServer.verifiedHistoricalHeaders.clear();
+ //Every wallet here derives from the same key, so one test's cached script hash state is the next one's, and the caches are keyed by script hash
+ ElectrumServer.clearPreviousServerState();
+ ElectrumServer.getSubscribedScriptHashes().clear();
+ ElectrumServer.confirmingRecent.clear();
+ AppServices.setAnnouncedTip(null);
+ Network.set(null);
+ }
+
+ /**
+ * The whole path for one newly confirmed transaction: the proof reconstructs the merkle root of a header the store already holds, the height is
+ * written, and the hash of the block it was proven against is written with it. No block header is fetched, which is what leaves a real
+ * confirmation making the same single call a recent transaction's confirmation makes.
+ */
+ @Test
+ public void provesAConfirmedTransactionAndRecordsItsBlock() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.serveTransaction(transaction);
+ server.serveProof(transaction, PROVEN_HEIGHT, 0);
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = references(wallet, reference(transaction, PROVEN_HEIGHT));
+
+ new ElectrumServer().getReferencedTransactions(wallet, nodeTransactionMap);
+
+ BlockTransaction written = wallet.getWalletTransaction(transaction.getTxId());
+ assertNotNull(written);
+ assertEquals(PROVEN_HEIGHT, written.getHeight());
+ assertEquals(chain.get(PROVEN_HEIGHT - 1).getHash(), written.getBlockHash());
+ assertEquals(1, server.getProofRequests());
+ assertEquals(0, server.getBlockHeaderRequests());
+ assertTrue(listener.isEmpty());
+ }
+
+ /**
+ * A branch that does not reconstruct the root of a verified header is the server proven wrong, which is the one outcome the dishonesty wording is
+ * reserved for. The transaction stays in the wallet as unconfirmed rather than being removed.
+ */
+ @Test
+ public void demotesATamperedProof() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.serveTransaction(transaction);
+ TransactionMerkleProof proof = server.serveProof(transaction, PROVEN_HEIGHT, 0);
+ proof.merkle.set(0, Sha256Hash.ZERO_HASH.toString());
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = references(wallet, reference(transaction, PROVEN_HEIGHT));
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ electrumServer.getReferencedTransactions(wallet, nodeTransactionMap);
+ electrumServer.postProofEvents(wallet);
+
+ assertEquals(0, wallet.getWalletTransaction(transaction.getTxId()).getHeight());
+ assertNull(wallet.getWalletTransaction(transaction.getTxId()).getBlockHash());
+ assertEquals(Set.of(reference(transaction, 0)), nodeTransactionMap.values().iterator().next());
+ assertEquals(Set.of(reference(transaction, PROVEN_HEIGHT)), listener.getFailed());
+ assertTrue(listener.getRefused().isEmpty());
+ }
+
+ /**
+ * The CVE-2017-12842 forgery: a fake transaction whose txid is one half of a genuine 64 byte transaction mined in the block, with the other half
+ * presented as its sibling. The verifier rejects it because that concatenation deserializes as a transaction, which no inner node of a real tree
+ * can. A rejected branch is a proof that does not verify rather than an exception escaping the pass.
+ */
+ @Test
+ public void rejectsABranchWhoseInnerNodeIsATransaction() {
+ Transaction sixtyFourByteTransaction = new Transaction();
+ sixtyFourByteTransaction.addInput(Sha256Hash.ZERO_HASH, 0, new Script(new byte[0]));
+ sixtyFourByteTransaction.addOutput(0, new Script(new byte[] {0x51, 0x51, 0x51, 0x51}));
+ byte[] serialized = sixtyFourByteTransaction.bitcoinSerialize();
+ assertEquals(64, serialized.length);
+
+ Sha256Hash forgedLeaf = Sha256Hash.wrapReversed(Arrays.copyOfRange(serialized, 0, 32));
+ TransactionMerkleProof proof = new TransactionMerkleProof();
+ proof.block_height = PROVEN_HEIGHT;
+ proof.pos = 0;
+ proof.merkle = List.of(Sha256Hash.wrapReversed(Arrays.copyOfRange(serialized, 32, 64)).toString());
+
+ assertFalse(ElectrumServer.verifyProof(forgedLeaf, proof, chain.get(PROVEN_HEIGHT - 1)));
+ }
+
+ /**
+ * Malformed proofs are proofs that do not verify, never exceptions escaping the pass.
+ */
+ @Test
+ public void rejectsAMalformedBranch() {
+ BlockHeader header = chain.get(PROVEN_HEIGHT - 1);
+ TransactionMerkleProof proof = new TransactionMerkleProof();
+ proof.block_height = PROVEN_HEIGHT;
+ proof.pos = 0;
+
+ proof.merkle = null;
+ assertFalse(ElectrumServer.verifyProof(blockTransactions.getFirst().getTxId(), proof, header));
+
+ proof.merkle = Collections.singletonList(null);
+ assertFalse(ElectrumServer.verifyProof(blockTransactions.getFirst().getTxId(), proof, header));
+
+ proof.merkle = List.of("cafebabe");
+ assertFalse(ElectrumServer.verifyProof(blockTransactions.getFirst().getTxId(), proof, header));
+ }
+
+ /**
+ * A proof answered for another block is the server declining to substantiate the height it reported, not the server caught lying about it: Electrum
+ * treats the requested height as a hint, and the wording the user is shown has to reflect what has actually been shown.
+ */
+ @Test
+ public void refusesAProofForAnotherBlock() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.serveTransaction(transaction);
+ TransactionMerkleProof proof = server.serveProof(transaction, PROVEN_HEIGHT, 0);
+ proof.block_height = PROVEN_HEIGHT + 1;
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = references(wallet, reference(transaction, PROVEN_HEIGHT));
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ electrumServer.getReferencedTransactions(wallet, nodeTransactionMap);
+ electrumServer.postProofEvents(wallet);
+
+ assertEquals(0, wallet.getWalletTransaction(transaction.getTxId()).getHeight());
+ assertEquals(Set.of(reference(transaction, PROVEN_HEIGHT)), listener.getRefused());
+ assertTrue(listener.getFailed().isEmpty());
+ assertEquals(ElectrumServer.proofAttempts, server.getProofRequests()); //it stays outstanding, so the whole retry budget is spent on it
+ }
+
+ /**
+ * A branch deeper than a block can hold is refused by the verifier's own bound before any hashing is done.
+ */
+ @Test
+ public void demotesAnOverDeepBranch() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.serveTransaction(transaction);
+ TransactionMerkleProof proof = server.serveProof(transaction, PROVEN_HEIGHT, 0);
+ proof.merkle = new ArrayList<>(Collections.nCopies(16, Sha256Hash.ZERO_HASH.toString()));
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = references(wallet, reference(transaction, PROVEN_HEIGHT));
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ electrumServer.getReferencedTransactions(wallet, nodeTransactionMap);
+ electrumServer.postProofEvents(wallet);
+
+ assertEquals(0, wallet.getWalletTransaction(transaction.getTxId()).getHeight());
+ assertEquals(1, listener.getFailed().size());
+ }
+
+ /**
+ * Where Sparrow cannot reach a verified header at the height, nothing has been shown to be false: the server may be honest and simply unable to
+ * serve the range. That is refusal class, however good the proof it supplied looks.
+ */
+ @Test
+ public void refusesAHeightTheStoreCannotReach() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.serveTransaction(transaction);
+ server.serveProof(transaction, CHAIN_LENGTH, 0);
+ //The store stops one short of the height, and the server will not serve the header that would advance it
+ server.setServedChainLength(CHAIN_LENGTH - 1);
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = references(wallet, reference(transaction, CHAIN_LENGTH));
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ electrumServer.getReferencedTransactions(wallet, nodeTransactionMap);
+ electrumServer.postProofEvents(wallet);
+
+ assertEquals(0, wallet.getWalletTransaction(transaction.getTxId()).getHeight());
+ assertEquals(Set.of(reference(transaction, CHAIN_LENGTH)), listener.getRefused());
+ assertTrue(listener.getFailed().isEmpty());
+ }
+
+ /**
+ * The discrimination is behavioural: a server refusing for capacity fails whole batches and recovers, while one that cannot substantiate a
+ * particular pair leaves that pair unanswered while its siblings succeed. Only the second raises a dialog, and only for the pair it concerns.
+ */
+ @Test
+ public void reportsAPersistentRefusalAmongSucceedingSiblings() throws Exception {
+ Wallet wallet = testWallet();
+ List<Transaction> transactions = blockTransactions.subList(0, 3);
+ for(int i = 0; i < transactions.size(); i++) {
+ server.serveTransaction(transactions.get(i));
+ if(i > 0) {
+ server.serveProof(transactions.get(i), PROVEN_HEIGHT, i);
+ }
+ }
+
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = references(wallet,
+ transactions.stream().map(transaction -> reference(transaction, PROVEN_HEIGHT)).toArray(BlockTransactionHash[]::new));
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ electrumServer.getReferencedTransactions(wallet, nodeTransactionMap);
+ electrumServer.postProofEvents(wallet);
+
+ assertEquals(PROVEN_HEIGHT, wallet.getWalletTransaction(transactions.get(1).getTxId()).getHeight());
+ assertEquals(PROVEN_HEIGHT, wallet.getWalletTransaction(transactions.get(2).getTxId()).getHeight());
+ assertEquals(0, wallet.getWalletTransaction(transactions.getFirst().getTxId()).getHeight());
+ assertEquals(Set.of(reference(transactions.getFirst(), PROVEN_HEIGHT)), listener.getRefused());
+ assertTrue(listener.getFailed().isEmpty());
+ //The siblings are answered on the first attempt, and only the pair that stays outstanding is asked for again
+ assertEquals(ElectrumServer.proofAttempts, server.getProofRequests());
+ assertEquals(3, server.getProofRequestKeys().getFirst().size());
+ assertEquals(1, server.getProofRequestKeys().getLast().size());
+ }
+
+ /**
+ * A refusal that clears on a retry is the shape a server under load has, and it must cost nothing but the wait.
+ */
+ @Test
+ public void retriesATransientRefusal() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.serveTransaction(transaction);
+ server.serveProof(transaction, PROVEN_HEIGHT, 0);
+ server.refuseFirstAttempts(transaction, PROVEN_HEIGHT, 1);
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = references(wallet, reference(transaction, PROVEN_HEIGHT));
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ electrumServer.getReferencedTransactions(wallet, nodeTransactionMap);
+ electrumServer.postProofEvents(wallet);
+
+ assertEquals(PROVEN_HEIGHT, wallet.getWalletTransaction(transaction.getTxId()).getHeight());
+ assertEquals(2, server.getProofRequests());
+ assertTrue(listener.isEmpty());
+ }
+
+ /**
+ * A server momentarily unable to answer the call at all looks exactly like one that never will, until the retries have been spent. The first
+ * attempt failing as a whole therefore costs a retry and nothing else.
+ */
+ @Test
+ public void retriesAWholeCallThatFailsOnce() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.serveTransaction(transaction);
+ server.serveProof(transaction, PROVEN_HEIGHT, 0);
+ server.failFirstRequests(1, new ElectrumServerRpcException("Server busy"));
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = references(wallet, reference(transaction, PROVEN_HEIGHT));
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ electrumServer.getReferencedTransactions(wallet, nodeTransactionMap);
+ electrumServer.postProofEvents(wallet);
+
+ assertEquals(PROVEN_HEIGHT, wallet.getWalletTransaction(transaction.getTxId()).getHeight());
+ assertEquals(2, server.getProofRequests());
+ assertTrue(ElectrumServer.isVerifyingTransactions());
+ assertTrue(listener.isEmpty());
+ }
+
+ /**
+ * A private server that cannot serve the call at all, for any reason other than not implementing it, must not cost the user their wallet history:
+ * it is a server they chose and already trust for everything else it reports. The session goes unverified from here, and nothing is demoted, since
+ * a call that failed as a whole has refused nothing.
+ */
+ @Test
+ public void proceedsUnverifiedWhereAPrivateServerCannotSupplyProofs() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.serveTransaction(transaction);
+ server.setProofFailure(new ElectrumServerRpcException("Batch too large"));
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = references(wallet, reference(transaction, PROVEN_HEIGHT));
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ electrumServer.getReferencedTransactions(wallet, nodeTransactionMap);
+ electrumServer.postProofEvents(wallet);
+
+ assertFalse(ElectrumServer.isVerifyingTransactions());
+ assertEquals(ElectrumServer.proofAttempts, server.getProofRequests()); //only after the retries are spent
+ assertEquals(PROVEN_HEIGHT, wallet.getWalletTransaction(transaction.getTxId()).getHeight());
+ assertEquals(Set.of(reference(transaction, PROVEN_HEIGHT)), nodeTransactionMap.values().iterator().next());
+ assertTrue(listener.isEmpty());
+ }
+
+ /**
+ * The public tier does not get that leniency: verification is what the server is there for, so the history fails and another server is tried.
+ */
+ @Test
+ public void failsTheHistoryWhereAPublicServerCannotSupplyProofs() throws Exception {
+ Network.set(Network.MAINNET);
+ ServerType previousServerType = Config.get().getServerType();
+ Config.get().setServerType(ServerType.PUBLIC_ELECTRUM_SERVER);
+ try {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.setProofFailure(new ElectrumServerRpcException("Batch too large"));
+ int height = Network.MAINNET.getHeaderCheckpoints().getMaxHeight() + 1;
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = references(wallet, reference(transaction, height));
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ assertThrows(ServerException.class, () -> electrumServer.getReferencedTransactions(wallet, nodeTransactionMap));
+
+ assertTrue(ElectrumServer.isVerifyingTransactions());
+ assertNull(wallet.getWalletTransaction(transaction.getTxId()));
+ } finally {
+ Config.get().setServerType(previousServerType);
+ }
+ }
+
+ /**
+ * A lost connection reaches this as the same exception and must not be read as the server being unable to serve the call: it is the ordinary
+ * reconnect's business, and downgrading the session for it would let one network failure leave a wallet unverified until the next connect.
+ */
+ @Test
+ public void failsThePassWhereTheConnectionIsGoneRatherThanDisablingVerification() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.setProofFailure(new ElectrumServerRpcException("Connection closed"));
+ ElectrumServer.transport = new UnusedTransport(false);
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = references(wallet, reference(transaction, PROVEN_HEIGHT));
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ assertThrows(ServerException.class, () -> electrumServer.getReferencedTransactions(wallet, nodeTransactionMap));
+
+ assertTrue(ElectrumServer.isVerifyingTransactions());
+ assertNull(wallet.getWalletTransaction(transaction.getTxId()));
+ }
+
+ /**
+ * A stored height carries its verdict: it was proven when it was written, or it predates the feature, and either way it is never proven again.
+ * This is what keeps a refresh, a restart and a server switch from re-proving a whole wallet.
+ */
+ @Test
+ public void doesNotReproveAStoredHeight() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ wallet.updateTransactions(Map.of(transaction.getTxId(),
+ new BlockTransaction(transaction.getTxId(), PROVEN_HEIGHT, null, 0L, transaction, chain.get(PROVEN_HEIGHT - 1).getHash())));
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = references(wallet, reference(transaction, PROVEN_HEIGHT));
+
+ new ElectrumServer().getReferencedTransactions(wallet, nodeTransactionMap);
+
+ assertEquals(0, server.getProofRequests());
+ assertEquals(PROVEN_HEIGHT, wallet.getWalletTransaction(transaction.getTxId()).getHeight());
+ }
+
+ /**
+ * The input a lying server would craft: one transaction reported at two heights on two script hashes. Keyed by transaction alone the pair that did
+ * not prove rides into the wallet on the back of the pair that did, so each is proven and demoted on its own, and the proven one wins the collapse
+ * to a single wallet transaction.
+ */
+ @Test
+ public void demotesTheUnprovenPairOfATransactionReportedAtTwoHeights() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.serveTransaction(transaction);
+ server.serveProof(transaction, PROVEN_HEIGHT, 0); //the other height is not answered at all
+
+ List<WalletNode> nodes = new ArrayList<>(wallet.getNode(KeyPurpose.RECEIVE).getChildren());
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = new LinkedHashMap<>();
+ nodeTransactionMap.put(nodes.get(0), new TreeSet<>(Set.of(reference(transaction, PROVEN_HEIGHT))));
+ nodeTransactionMap.put(nodes.get(1), new TreeSet<>(Set.of(reference(transaction, PROVEN_HEIGHT + 1))));
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ electrumServer.getReferencedTransactions(wallet, nodeTransactionMap);
+ electrumServer.postProofEvents(wallet);
+
+ assertEquals(PROVEN_HEIGHT, wallet.getWalletTransaction(transaction.getTxId()).getHeight());
+ assertEquals(chain.get(PROVEN_HEIGHT - 1).getHash(), wallet.getWalletTransaction(transaction.getTxId()).getBlockHash());
+ assertEquals(Set.of(reference(transaction, PROVEN_HEIGHT)), nodeTransactionMap.get(nodes.get(0)));
+ assertEquals(Set.of(reference(transaction, 0)), nodeTransactionMap.get(nodes.get(1)));
+ assertEquals(Set.of(reference(transaction, PROVEN_HEIGHT + 1)), listener.getRefused());
+ }
+
+ /**
+ * The gate runs many times in one pass over overlapping reference sets, and a demoted height reads as changed to every later call. Without the memo
+ * each of those calls would spend the retry budget on it again; with it the later calls demote and move on, and the next pass re-attempts it.
+ */
+ @Test
+ public void doesNotReproveAPairAlreadyRefusedInThePass() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.serveTransaction(transaction);
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = references(wallet, reference(transaction, PROVEN_HEIGHT));
+ electrumServer.getReferencedTransactions(wallet, nodeTransactionMap);
+ int afterFirstCall = server.getProofRequests();
+
+ //The server reports the same height again, as it does on every later call in the pass
+ nodeTransactionMap = references(wallet, reference(transaction, PROVEN_HEIGHT));
+ electrumServer.getReferencedTransactions(wallet, nodeTransactionMap);
+ assertEquals(afterFirstCall, server.getProofRequests());
+ assertEquals(Set.of(reference(transaction, 0)), nodeTransactionMap.values().iterator().next());
+
+ //A new pass constructs its own ElectrumServer, which starts clean
+ nodeTransactionMap = references(wallet, reference(transaction, PROVEN_HEIGHT));
+ new ElectrumServer().getReferencedTransactions(wallet, nodeTransactionMap);
+ assertEquals(afterFirstCall * 2, server.getProofRequests());
+ }
+
+ /**
+ * The same height form of a reorg, which is the one the server reports nothing about: the height is still what it was, but the block it was proven
+ * against is no longer on the chain. The recorded block hash is what selects it, and only above the deepest fork this session has seen.
+ */
+ @Test
+ public void reprovesATransactionProvenAgainstAnOrphanedHeader() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.serveProof(transaction, PROVEN_HEIGHT, 0);
+ wallet.updateTransactions(Map.of(transaction.getTxId(),
+ new BlockTransaction(transaction.getTxId(), PROVEN_HEIGHT, null, 0L, transaction, Sha256Hash.ZERO_HASH)));
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = references(wallet, reference(transaction, PROVEN_HEIGHT));
+
+ //Below the deepest fork accepted this session, so a stale block hash is not looked for at all
+ new ElectrumServer().getReferencedTransactions(wallet, nodeTransactionMap);
+ assertEquals(0, server.getProofRequests());
+
+ ElectrumServer.lastReorgForkHeight = PROVEN_HEIGHT - 1;
+ new ElectrumServer().getReferencedTransactions(wallet, nodeTransactionMap);
+
+ assertEquals(1, server.getProofRequests());
+ assertEquals(PROVEN_HEIGHT, wallet.getWalletTransaction(transaction.getTxId()).getHeight());
+ assertEquals(chain.get(PROVEN_HEIGHT - 1).getHash(), wallet.getWalletTransaction(transaction.getTxId()).getBlockHash());
+ }
+
+ /**
+ * One dialog per wallet per task, however many gated calls the pass makes, and however many transactions each surfaces. A pair already shown is
+ * not shown again by the passes that re-attempt it.
+ */
+ @Test
+ public void raisesOneEventPerWalletForTheWholeTask() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction first = blockTransactions.get(0);
+ Transaction second = blockTransactions.get(1);
+ server.serveTransaction(first);
+ server.serveTransaction(second);
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ electrumServer.getReferencedTransactions(wallet, references(wallet, reference(first, PROVEN_HEIGHT)));
+ electrumServer.getReferencedTransactions(wallet, references(wallet, reference(second, PROVEN_HEIGHT)));
+ electrumServer.postProofEvents(wallet);
+
+ assertEquals(1, listener.getRefusedEvents());
+ assertEquals(Set.of(reference(first, PROVEN_HEIGHT), reference(second, PROVEN_HEIGHT)), listener.getRefused());
+
+ //The next task re-attempts both, and raises nothing for what has already been shown
+ listener.reset();
+ ElectrumServer nextTask = new ElectrumServer();
+ nextTask.getReferencedTransactions(wallet, references(wallet, reference(first, PROVEN_HEIGHT)));
+ nextTask.postProofEvents(wallet);
+ assertEquals(0, listener.getRefusedEvents());
+ }
+
+ /**
+ * A refusal and a proof shown false are different claims, and the weaker one must not filter out the stronger. A server that leaves a pair
+ * unanswered for a pass and then supplies a branch that does not reconstruct has been proven wrong, which is the only thing the dishonesty wording
+ * is for - and the user has to be told, even though the same pair already produced a refusal dialog. The converse is not true.
+ */
+ @Test
+ public void showsAProofShownFalseAfterTheSamePairWasRefused() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.serveTransaction(transaction);
+
+ //The server will not answer for it at all
+ ElectrumServer refusingPass = new ElectrumServer();
+ refusingPass.getReferencedTransactions(wallet, references(wallet, reference(transaction, PROVEN_HEIGHT)));
+ refusingPass.postProofEvents(wallet);
+ assertEquals(1, listener.getRefusedEvents());
+ assertEquals(0, listener.getFailedEvents());
+
+ //It answers on the next pass, and the branch does not reconstruct
+ listener.reset();
+ server.serveProof(transaction, PROVEN_HEIGHT, 0).merkle.set(0, Sha256Hash.ZERO_HASH.toString());
+ ElectrumServer failingPass = new ElectrumServer();
+ failingPass.getReferencedTransactions(wallet, references(wallet, reference(transaction, PROVEN_HEIGHT)));
+ failingPass.postProofEvents(wallet);
+
+ assertEquals(1, listener.getFailedEvents());
+ assertEquals(Set.of(reference(transaction, PROVEN_HEIGHT)), listener.getFailed());
+
+ //Going quiet about it afterwards says less than what has been said, so it raises nothing
+ listener.reset();
+ server.refuseFirstAttempts(transaction, PROVEN_HEIGHT, Integer.MAX_VALUE);
+ ElectrumServer quietPass = new ElectrumServer();
+ quietPass.getReferencedTransactions(wallet, references(wallet, reference(transaction, PROVEN_HEIGHT)));
+ quietPass.postProofEvents(wallet);
+
+ assertTrue(listener.isEmpty());
+ }
+
+ /**
+ * The dialogs ask the user to switch servers, so what one server was shown to have got wrong must not filter the next one's answer for the same
+ * transaction. Suppression is scoped to the connected server, and getTransport clears it on connecting to a different one.
+ */
+ @Test
+ public void showsAFindingAgainAfterTheServerChanges() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.serveTransaction(transaction);
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ electrumServer.getReferencedTransactions(wallet, references(wallet, reference(transaction, PROVEN_HEIGHT)));
+ electrumServer.postProofEvents(wallet);
+ assertEquals(1, listener.getRefusedEvents());
+
+ listener.reset();
+ ElectrumServer.clearPreviousServerState(); //what getTransport does on connecting to a server other than the previous one
+
+ ElectrumServer afterSwitch = new ElectrumServer();
+ afterSwitch.getReferencedTransactions(wallet, references(wallet, reference(transaction, PROVEN_HEIGHT)));
+ afterSwitch.postProofEvents(wallet);
+
+ assertEquals(1, listener.getRefusedEvents());
+ assertEquals(Set.of(reference(transaction, PROVEN_HEIGHT)), listener.getRefused());
+ }
+
+ /**
+ * A private server without the call is a configuration rather than a fault: the session goes unverified from here, and nothing is demoted, since
+ * nothing has been refused.
+ */
+ @Test
+ public void disablesVerificationOnAServerWithoutTheCall() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.serveTransaction(transaction);
+ server.setProofFailure(new UnsupportedMethodException("blockchain.transaction.get_merkle", null));
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = references(wallet, reference(transaction, PROVEN_HEIGHT));
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ electrumServer.getReferencedTransactions(wallet, nodeTransactionMap);
+ electrumServer.postProofEvents(wallet);
+
+ assertTrue(!ElectrumServer.isVerifyingTransactions());
+ assertEquals(PROVEN_HEIGHT, wallet.getWalletTransaction(transaction.getTxId()).getHeight());
+ assertEquals(Set.of(reference(transaction, PROVEN_HEIGHT)), nodeTransactionMap.values().iterator().next());
+ assertTrue(listener.isEmpty());
+ }
+
+ /**
+ * The other half of the same finding: on the public tier verification is not optional, so a server without the call fails the history instead,
+ * which is what carries it to WalletHistoryFailedEvent and rotates to another server. The capability is left on, since the next server has it.
+ */
+ @Test
+ public void failsTheHistoryOnAPublicServerWithoutTheCall() throws Exception {
+ Network.set(Network.MAINNET);
+ ServerType previousServerType = Config.get().getServerType();
+ Config.get().setServerType(ServerType.PUBLIC_ELECTRUM_SERVER);
+ try {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.setProofFailure(new UnsupportedMethodException("blockchain.transaction.get_merkle", null));
+ //Above the last pin, so nothing is fetched ahead of the call that is missing
+ int height = Network.MAINNET.getHeaderCheckpoints().getMaxHeight() + 1;
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = references(wallet, reference(transaction, height));
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ ServerException e = assertThrows(ServerException.class, () -> electrumServer.getReferencedTransactions(wallet, nodeTransactionMap));
+ assertTrue(e.getMessage().contains("blockchain.transaction.get_merkle"));
+ electrumServer.postProofEvents(wallet);
+
+ assertTrue(ElectrumServer.isVerifyingTransactions());
+ assertNull(wallet.getWalletTransaction(transaction.getTxId()));
+ assertEquals(Set.of(reference(transaction, height)), nodeTransactionMap.values().iterator().next());
+ assertTrue(listener.isEmpty());
+ } finally {
+ Config.get().setServerType(previousServerType);
+ }
+ }
+
+ /**
+ * The store is asked before the cache of headers the server has announced. That cache holds every tip it has ever reported and is not rewound when
+ * the chain reorganises, so at a height that was once a tip it can name the replaced block - and serving that would put the orphaned block's
+ * timestamp on a transaction the store has just re-proven against its replacement.
+ */
+ @Test
+ public void servesTimestampsFromTheStoreRatherThanAnAnnouncedHeader() throws Exception {
+ Wallet wallet = testWallet();
+ //A different header at the proven height, as an announcement made before the block at that height was replaced would have left behind
+ BlockHeader replaced = mineHeader(chain.get(PROVEN_HEIGHT - 2), Sha256Hash.ZERO_HASH, CHAIN_TIME + 900);
+ assertNotEquals(chain.get(PROVEN_HEIGHT - 1).getHash(), replaced.getHash());
+ ElectrumServer.updateRetrievedBlockHeaders(PROVEN_HEIGHT, replaced);
+
+ Map<Integer, BlockHeader> blockHeaderMap = new ElectrumServer().getBlockHeaders(wallet,
+ Set.of(reference(blockTransactions.getFirst(), PROVEN_HEIGHT)));
+
+ assertEquals(chain.get(PROVEN_HEIGHT - 1).getHash(), blockHeaderMap.get(PROVEN_HEIGHT).getHash());
+ assertEquals(0, server.getBlockHeaderRequests());
+ }
+
+ /**
+ * Timestamps for a proven height come from the store the proof already used, so a confirmed wallet transaction issues no header request of its own.
+ */
+ @Test
+ public void servesTimestampsFromTheVerifiedHeaders() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction transaction = blockTransactions.getFirst();
+ server.serveTransaction(transaction);
+ server.serveProof(transaction, PROVEN_HEIGHT, 0);
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ electrumServer.getReferencedTransactions(wallet, references(wallet, reference(transaction, PROVEN_HEIGHT)));
+
+ assertEquals(chain.get(PROVEN_HEIGHT - 1).getTimeAsDate(), wallet.getWalletTransaction(transaction.getTxId()).getDate());
+ assertEquals(0, server.getBlockHeaderRequests());
+ }
+
+ /**
+ * The silent payments batch is the second write path, and it writes heights the subscription reported without going through a history call at all.
+ * The same proof applies, and the unproven reference is replaced in place by an unconfirmed one carrying whatever the batch had already found.
+ */
+ @Test
+ public void demotesAnUnprovenSilentPaymentReference() throws Exception {
+ Wallet wallet = testWallet();
+ Transaction proven = blockTransactions.get(0);
+ Transaction refused = blockTransactions.get(1);
+ server.serveProof(proven, PROVEN_HEIGHT, 0);
+
+ Map<BlockTransactionHash, Transaction> referencesToFetch = new TreeMap<>();
+ referencesToFetch.put(reference(proven, PROVEN_HEIGHT), null);
+ referencesToFetch.put(reference(refused, PROVEN_HEIGHT), refused);
+ referencesToFetch.put(reference(blockTransactions.get(2), 0), null); //unconfirmed, so nothing to prove
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ Map<BlockTransactionHash, BlockHeader> result = electrumServer.verifySilentPaymentReferences(wallet, referencesToFetch);
+ electrumServer.postProofEvents(wallet);
+
+ assertEquals(Set.of(reference(proven, PROVEN_HEIGHT)), result.keySet());
+ assertEquals(chain.get(PROVEN_HEIGHT - 1).getHash(), result.values().iterator().next().getHash());
+ assertEquals(Set.of(reference(proven, PROVEN_HEIGHT), reference(refused, 0), reference(blockTransactions.get(2), 0)), referencesToFetch.keySet());
+ assertEquals(refused, referencesToFetch.get(reference(refused, 0))); //the demotion carries over what the batch already had
+ assertEquals(Set.of(reference(refused, PROVEN_HEIGHT)), listener.getRefused());
+ }
+
+ /**
+ * The whole pass on a wallet reopened with a demoted transaction already stored. The stored output sits at height 0 while the server still reports
+ * it confirmed, so every used node holding it counts as changed - and where they all are, one address with one refused transaction is enough for
+ * the check to read the whole history as having changed and abort into a backup and full refresh.
+ * <p>
+ * It would do so on every open, and the refresh would not help: the transaction is refused, demoted and stored again. For a passphrase wallet it
+ * is the "incorrect passphrase" dialog, every open, until the server changes.
+ */
+ @Test
+ public void doesNotReadAStoredDemotionAsTheWholeHistoryChanging() throws Exception {
+ Wallet wallet = testWallet();
+ WalletNode node = wallet.getNode(KeyPurpose.RECEIVE).getChildren().iterator().next();
+
+ //A real payment to the wallet's one used address, so the node keeps the output the pass recalculates
+ Transaction transaction = new Transaction();
+ transaction.addInput(Sha256Hash.ZERO_HASH, 0, new Script(new byte[0]));
+ transaction.addOutput(10000L, wallet.getAddress(node));
+ server.serveTransaction(transaction);
+
+ //The wallet as the previous session left it: demoted to unconfirmed and persisted that way
+ wallet.updateTransactions(Map.of(transaction.getTxId(), new BlockTransaction(transaction.getTxId(), 0, null, 0L, transaction)));
+ node.getTransactionOutputs().add(new BlockTransactionHashIndex(transaction.getTxId(), 0, null, 0L, 0, 10000));
+
+ //The server has not changed its mind, and still will not prove it
+ String scriptHash = ElectrumServer.getScriptHash(node);
+ server.serveHistory(node.getDerivationPath(), new ScriptHashTx(PROVEN_HEIGHT, transaction.getTxId().toString(), 0));
+ server.serveScriptHashStatus(scriptHash, ElectrumServer.getScriptHashStatus(List.of(new ScriptHashTx(PROVEN_HEIGHT, transaction.getTxId().toString(), 0))));
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ assertTrue(electrumServer.fetchAndCalculateHistory(wallet, null, null));
+
+ //Still the one used node, still unconfirmed, and nothing cleared
+ assertEquals(0, wallet.getWalletTransaction(transaction.getTxId()).getHeight());
+ assertEquals(1, node.getTransactionOutputs().size());
+ assertEquals(0, node.getTransactionOutputs().iterator().next().getHeight());
+ //The pass posts its own findings from the finally that closes it
+ assertEquals(1, listener.getRefusedEvents());
+ }
+
+ /**
+ * A reorg detected by the pass itself - which is what happens when a history thread reconciles from inside its own gate - invalidates nodes whose
+ * data that pass has already fetched. Its status must stay cleared, or the refresh the reorg triggers finds the node looking up to date and never
+ * fetches it again, leaving the transaction held against the block it was proven in rather than the one that replaced it.
+ */
+ @Test
+ public void leavesANodeInvalidatedMidPassForTheRefreshThatFollows() throws Exception {
+ Wallet wallet = testWallet();
+ WalletNode node = wallet.getNode(KeyPurpose.RECEIVE).getChildren().iterator().next();
+ String scriptHash = ElectrumServer.getScriptHash(node);
+ Transaction transaction = confirmedPayment(wallet, node);
+ server.invalidateDuringPass(PROVEN_HEIGHT - 1);
+
+ new ElectrumServer().fetchAndCalculateHistory(wallet, null, null);
+
+ assertNull(ElectrumServer.retrievedScriptHashes.get(scriptHash), "the status must not be restored over an invalidation this pass caused");
+ assertTrue(ElectrumServer.reorgInvalidatedScriptHashes.contains(scriptHash), "the invalidation must survive for the refresh the reorg triggers");
+ }
+
+ /**
+ * The refresh the reorg triggers is the pass that acts on the invalidation, so it does restore the status and clear it - otherwise no pass ever
+ * would, and the node would be fetched again on every refresh for the rest of the session.
+ */
+ @Test
+ public void clearsAnInvalidationTheRefreshItTriggeredHasActedOn() throws Exception {
+ Wallet wallet = testWallet();
+ WalletNode node = wallet.getNode(KeyPurpose.RECEIVE).getChildren().iterator().next();
+ String scriptHash = ElectrumServer.getScriptHash(node);
+ confirmedPayment(wallet, node);
+ assertTrue(ElectrumServer.invalidateScriptHashesForReorg(wallet, PROVEN_HEIGHT - 1));
+
+ new ElectrumServer().fetchAndCalculateHistory(wallet, null, null);
+
+ assertNotNull(ElectrumServer.retrievedScriptHashes.get(scriptHash), "a pass acting on the invalidation records what it fetched");
+ assertFalse(ElectrumServer.reorgInvalidatedScriptHashes.contains(scriptHash), "and the exemption lasts for exactly that one fetch");
+ }
+
+ /**
+ * A payment to the node, stored as confirmed and reported so by the server, as a wallet is when a reorg reaches it.
+ */
+ private Transaction confirmedPayment(Wallet wallet, WalletNode node) throws Exception {
+ Transaction transaction = new Transaction();
+ transaction.addInput(Sha256Hash.ZERO_HASH, 0, new Script(new byte[0]));
+ transaction.addOutput(10000L, wallet.getAddress(node));
+ server.serveTransaction(transaction);
+ server.serveProof(transaction, PROVEN_HEIGHT, 0);
+
+ wallet.updateTransactions(Map.of(transaction.getTxId(), new BlockTransaction(transaction.getTxId(), PROVEN_HEIGHT, null, 0L, transaction,
+ chain.get(PROVEN_HEIGHT - 1).getHash())));
+ node.getTransactionOutputs().add(new BlockTransactionHashIndex(transaction.getTxId(), PROVEN_HEIGHT, null, 0L, 0, 10000));
+
+ ScriptHashTx confirmed = new ScriptHashTx(PROVEN_HEIGHT, transaction.getTxId().toString(), 0);
+ server.serveHistory(node.getDerivationPath(), confirmed);
+ server.serveScriptHashStatus(ElectrumServer.getScriptHash(node), ElectrumServer.getScriptHashStatus(List.of(confirmed)));
+
+ return transaction;
+ }
+
+ /**
+ * Demotion leaves the retrieved script hash statuses alone, since clearing or withholding one wipes the node's outputs or trips a full history
+ * change. What brings the demoted node back is the comparison against its own calculated status, which the fresh subscribe branch has to make as
+ * well as the already subscribed one, or a reconnect leaves the transaction unconfirmed until its status changes.
+ */
+ @Test
+ public void refetchesADemotedNodeWhenSubscribingAfresh() throws Exception {
+ Wallet wallet = testWallet();
+ List<WalletNode> nodes = new ArrayList<>(wallet.getNode(KeyPurpose.RECEIVE).getChildren());
+ WalletNode synced = nodes.get(0);
+ WalletNode demoted = nodes.get(1);
+ Sha256Hash txid = blockTransactions.getFirst().getTxId();
+ synced.getTransactionOutputs().add(new BlockTransactionHashIndex(txid, PROVEN_HEIGHT, null, 0L, 0, 10000));
+ demoted.getTransactionOutputs().add(new BlockTransactionHashIndex(txid, 0, null, 0L, 0, 10000));
+
+ //A status is a digest of the transaction heights alone, so what the synced node calculates is what the server reports for both of them
+ String confirmedStatus = ElectrumServer.getScriptHashStatus(ElectrumServer.getScriptHash(synced), synced);
+ for(WalletNode node : List.of(synced, demoted)) {
+ String scriptHash = ElectrumServer.getScriptHash(node);
+ server.serveScriptHashStatus(scriptHash, confirmedStatus);
+ //The put after a demotion runs unchanged, so the demoted node's retrieved status is the server's like any other
+ ElectrumServer.retrievedScriptHashes.put(scriptHash, confirmedStatus);
+ }
+
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = new LinkedHashMap<>();
+ new ElectrumServer().subscribeWalletNodes(wallet, List.of(synced, demoted), nodeTransactionMap, 0);
+
+ assertEquals(Set.of(demoted), nodeTransactionMap.keySet());
+ assertEquals(1, demoted.getTransactionOutputs().size()); //cache state only: the node's own outputs are untouched
+ }
+
+ private static Map<WalletNode, Set<BlockTransactionHash>> references(Wallet wallet, BlockTransactionHash... references) {
+ WalletNode node = wallet.getNode(KeyPurpose.RECEIVE).getChildren().iterator().next();
+ Map<WalletNode, Set<BlockTransactionHash>> nodeTransactionMap = new LinkedHashMap<>();
+ nodeTransactionMap.put(node, new TreeSet<>(List.of(references)));
+
+ return nodeTransactionMap;
+ }
+
+ private static BlockTransactionHash reference(Transaction transaction, int height) {
+ return new BlockTransaction(transaction.getTxId(), height, null, 0L, null);
+ }
+
+ private static void seedStore(List<BlockHeader> chain, int toHeight) throws Exception {
+ HeaderStore store = ElectrumServer.getHeaderStore();
+ for(int height = 1; height <= toHeight; height++) {
+ store.append(chain.get(height - 1));
+ }
+ }
+
+ /**
+ * A chain whose block at PROVEN_HEIGHT carries the given merkle root, the rest being empty blocks that only have to link.
+ */
+ private static List<BlockHeader> mineChain(BlockHeader previous, int count, Sha256Hash merkleRoot) {
+ List<BlockHeader> chain = new ArrayList<>();
+ for(int i = 0; i < count; i++) {
+ previous = mineHeader(previous, i + 1 == PROVEN_HEIGHT ? merkleRoot : Sha256Hash.ZERO_HASH, CHAIN_TIME + i);
+ chain.add(previous);
+ }
+
+ return chain;
+ }
+
+ private static BlockHeader mineHeader(BlockHeader previous, Sha256Hash merkleRoot, long time) {
+ for(long nonce = 0; nonce < 1000; nonce++) {
+ BlockHeader header = new BlockHeader(1, previous.getHash(), merkleRoot, null, time, 0x207fffffL, nonce);
+ if(header.verifyProofOfWork()) {
+ return header;
+ }
+ }
+
+ throw new IllegalStateException("Could not mine a regtest header at time " + time);
+ }
+
+ private static Transaction transaction(int index) {
+ Transaction transaction = new Transaction();
+ transaction.addInput(Sha256Hash.ZERO_HASH, index, new Script(new byte[0]));
+ transaction.addOutput(10000L + index, new Script(new byte[] {0x51}));
+
+ return transaction;
+ }
+
+ private static List<Sha256Hash> txids(List<Transaction> transactions) {
+ return transactions.stream().map(Transaction::getTxId).toList();
+ }
+
+ private static Sha256Hash merkleRoot(List<Sha256Hash> txids) {
+ List<Sha256Hash> level = new ArrayList<>(txids);
+ while(level.size() > 1) {
+ level = nextLevel(level);
+ }
+
+ return level.getFirst();
+ }
+
+ /**
+ * The sibling path from the leaf at the given position to the root, deepest level first, which is the shape the server returns.
+ */
+ private static List<String> merkleBranch(List<Sha256Hash> txids, int position) {
+ List<String> branch = new ArrayList<>();
+ List<Sha256Hash> level = new ArrayList<>(txids);
+ int index = position;
+ while(level.size() > 1) {
+ List<Sha256Hash> padded = new ArrayList<>(level);
+ if(padded.size() % 2 == 1) {
+ padded.add(padded.getLast());
+ }
+ branch.add(padded.get(index ^ 1).toString());
+ level = nextLevel(level);
+ index >>= 1;
+ }
+
+ return branch;
+ }
+
+ private static List<Sha256Hash> nextLevel(List<Sha256Hash> level) {
+ List<Sha256Hash> padded = new ArrayList<>(level);
+ if(padded.size() % 2 == 1) {
+ padded.add(padded.getLast());
+ }
+
+ List<Sha256Hash> next = new ArrayList<>();
+ for(int i = 0; i < padded.size(); i += 2) {
+ next.add(Sha256Hash.wrapReversed(Sha256Hash.hashTwice(Utils.concat(padded.get(i).getReversedBytes(), padded.get(i + 1).getReversedBytes()))));
+ }
+
+ return next;
+ }
+
+ /**
+ * A recent mempool transaction confirming yields the pair a wallet transaction confirming would, so the request that follows is the same one and
+ * the traffic after a new block is the same shape whether or not the wallet has anything in it. The height comes from the status guess.
+ */
+ @Test
+ public void provesARecentTransactionAsItConfirms() {
+ Transaction transaction = blockTransactions.getFirst();
+ AppServices.setAnnouncedTip(new ChainTip(PROVEN_HEIGHT, chain.get(PROVEN_HEIGHT)));
+
+ String scriptHash = "aa".repeat(32);
+ ElectrumServer.confirmingRecent.put(scriptHash, new BlockTransaction(transaction.getTxId(), 0, null, RECENT_FEE, transaction));
+
+ String confirmedStatus = ElectrumServer.getScriptHashStatus(List.of(new ScriptHashTx(PROVEN_HEIGHT, transaction.getTxId().toString(), RECENT_FEE)));
+ BlockTransactionHash reference = ElectrumServer.ConnectionService.getProofReference(new WalletNodeHistoryChangedEvent(scriptHash, confirmedStatus));
+
+ assertNotNull(reference);
+ assertEquals(transaction.getTxId(), reference.getHash());
+ assertEquals(PROVEN_HEIGHT, reference.getHeight());
+ //Consumed, so a further notification on the same script hash does not ask a second time
+ assertFalse(ElectrumServer.confirmingRecent.containsKey(scriptHash));
+ }
+
+ /**
+ * The guess is the entire height mechanism here, so a status it does not explain contributes nothing for that block. There is deliberately no
+ * fallback: asking the server for the history is the one call a wallet node whose history is already known does not make.
+ */
+ @Test
+ public void makesNoRequestWhereTheRecentTransactionHasNotConfirmed() {
+ Transaction transaction = blockTransactions.getFirst();
+ AppServices.setAnnouncedTip(new ChainTip(PROVEN_HEIGHT, chain.get(PROVEN_HEIGHT)));
+
+ String scriptHash = "bb".repeat(32);
+ ElectrumServer.confirmingRecent.put(scriptHash, new BlockTransaction(transaction.getTxId(), 0, null, RECENT_FEE, transaction));
+
+ //The status the script hash carries while its transaction is still in the mempool
+ String unconfirmedStatus = ElectrumServer.getScriptHashStatus(List.of(new ScriptHashTx(0, transaction.getTxId().toString(), RECENT_FEE)));
+ assertNull(ElectrumServer.ConnectionService.getProofReference(new WalletNodeHistoryChangedEvent(scriptHash, unconfirmedStatus)));
+
+ //Left in place, so the notification for its confirmation is still matched
+ assertTrue(ElectrumServer.confirmingRecent.containsKey(scriptHash));
+ }
+
+ /**
+ * Verification being off leaves the wallet making no proof requests at all, so making them for anything else would be traffic with nothing to
+ * cover. The surrounding gates already exclude Bitcoin Core and Tor by never reaching this code.
+ */
+ @Test
+ public void makesNoRequestWhereTransactionsAreNotBeingVerified() {
+ Transaction transaction = blockTransactions.getFirst();
+ AppServices.setAnnouncedTip(new ChainTip(PROVEN_HEIGHT, chain.get(PROVEN_HEIGHT)));
+
+ String scriptHash = "cc".repeat(32);
+ ElectrumServer.confirmingRecent.put(scriptHash, new BlockTransaction(transaction.getTxId(), 0, null, RECENT_FEE, transaction));
+ String confirmedStatus = ElectrumServer.getScriptHashStatus(List.of(new ScriptHashTx(PROVEN_HEIGHT, transaction.getTxId().toString(), RECENT_FEE)));
+
+ ElectrumServer.serverCapability.withMerkleProofs(false);
+ assertFalse(ElectrumServer.isVerifyingTransactions());
+ assertNull(ElectrumServer.ConnectionService.getProofReference(new WalletNodeHistoryChangedEvent(scriptHash, confirmedStatus)));
+ //Left in place rather than consumed, so turning verification back on within the retention window still covers the block
+ assertTrue(ElectrumServer.confirmingRecent.containsKey(scriptHash));
+ }
+
+ private static Wallet testWallet() {
+ Wallet wallet = new Wallet();
+ wallet.setPolicyType(PolicyType.SINGLE_HD);
+ wallet.setScriptType(ScriptType.P2WPKH);
+ Keystore keystore = new Keystore();
+ keystore.setKeyDerivation(new KeyDerivation("00000000", "m/84'/0'/0'"));
+ keystore.setExtendedPublicKey(ExtendedKey.fromDescriptor(Network.get() == Network.MAINNET ? TEST_XPUB : TEST_TPUB));
+ wallet.getKeystores().add(keystore);
+ wallet.setDefaultPolicy(Policy.getPolicy(PolicyType.SINGLE_HD, ScriptType.P2WPKH, wallet.getKeystores(), 1));
+ wallet.getNode(KeyPurpose.RECEIVE).fillToIndex(wallet, 1);
+
+ return wallet;
+ }
+
+ /**
+ * Answers proofs, headers and transactions from what the test has served it, recording what it was asked for. A pair the test has not served a
+ * proof for is answered with the error sentinel every attempt, which is a server that will not substantiate what it reported.
+ */
+ private class FakeProofServer extends SimpleElectrumServerRpc {
+ private final List<BlockHeader> chain;
+ private final Map<String, TransactionMerkleProof> proofs = new HashMap<>();
+ private final Map<String, Integer> refusals = new HashMap<>();
+ private final Map<String, String> rawTransactions = new HashMap<>();
+ private final Map<String, String> scriptHashStatuses = new HashMap<>();
+ private final Map<String, ScriptHashTx[]> histories = new HashMap<>();
+ private volatile Integer invalidateAtForkHeight;
+ private final AtomicInteger proofRequests = new AtomicInteger();
+ private final AtomicInteger blockHeaderRequests = new AtomicInteger();
+ private final List<Set<String>> proofRequestKeys = new ArrayList<>();
+ private volatile int servedChainLength;
+ private volatile RuntimeException proofFailure;
+ private volatile int proofFailureAfter;
+ private volatile int proofFailureUntil = Integer.MAX_VALUE;
+ private volatile RuntimeException headersFailure;
+
+ public FakeProofServer(List<BlockHeader> chain) {
+ this.chain = List.copyOf(chain);
+ this.servedChainLength = chain.size();
+ }
+
+ @Override
+ public Map<String, TransactionMerkleProof> getTransactionMerkleProofs(Transport transport, Wallet wallet, Collection<BlockTransactionHash> references) {
+ int request = proofRequests.incrementAndGet();
+ proofRequestKeys.add(references.stream().map(FakeProofServer::key).collect(Collectors.toCollection(LinkedHashSet::new)));
+ if(proofFailure != null && request > proofFailureAfter && request <= proofFailureUntil) {
+ throw proofFailure;
+ }
+
+ Map<String, TransactionMerkleProof> result = new LinkedHashMap<>();
+ for(BlockTransactionHash reference : references) {
+ String key = key(reference);
+ Integer remaining = refusals.get(key);
+ if(remaining != null && remaining > 0) {
+ refusals.put(key, remaining - 1);
+ result.put(key, TransactionMerkleProof.ERROR_PROOF);
+ } else {
+ result.put(key, proofs.getOrDefault(key, TransactionMerkleProof.ERROR_PROOF));
+ }
+ }
+
+ return result;
+ }
+
+ @Override
+ public BlockHeaders getBlockHeadersChunk(Transport transport, int startHeight, int count) {
+ if(headersFailure != null) {
+ throw headersFailure;
+ }
+
+ int index = startHeight - 1;
+ int available = Math.max(0, Math.min(count, servedChainLength - index));
+ List<BlockHeader> headers = available == 0 ? Collections.emptyList() : chain.subList(index, index + available);
+ BlockHeaders blockHeaders = new BlockHeaders();
+ blockHeaders.count = headers.size();
+ blockHeaders.max = HeaderChainState.RETARGET_INTERVAL;
+ blockHeaders.hex = headers.stream().map(header -> Utils.bytesToHex(header.bitcoinSerialize())).collect(Collectors.joining());
+
+ return ElectrumServerRpc.checkBlockHeaders(blockHeaders, startHeight, count, servedChainLength);
+ }
+
+ @Override
+ public Map<Integer, String> getBlockHeaders(Transport transport, Wallet wallet, Set<Integer> blockHeights) {
+ blockHeaderRequests.incrementAndGet();
+ Map<Integer, String> result = new TreeMap<>();
+ for(Integer height : blockHeights) {
+ if(height >= 1 && height <= chain.size()) {
+ result.put(height, Utils.bytesToHex(chain.get(height - 1).bitcoinSerialize()));
+ }
+ }
+
+ return result;
+ }
+
+ @Override
+ public Map<String, String> getTransactions(Transport transport, Wallet wallet, Set<String> txids) {
+ Map<String, String> result = new LinkedHashMap<>();
+ for(String txid : txids) {
+ String hex = rawTransactions.get(txid);
+ result.put(txid, hex == null ? Sha256Hash.ZERO_HASH.toString() : hex);
+ }
+
+ return result;
+ }
+
+ @Override
+ public Map<String, ScriptHashTx[]> getScriptHashHistory(Transport transport, Wallet wallet, Map<String, String> pathScriptHashes, boolean failOnError) {
+ Map<String, ScriptHashTx[]> result = new LinkedHashMap<>();
+ pathScriptHashes.keySet().forEach(path -> result.put(path, histories.getOrDefault(path, new ScriptHashTx[0])));
+
+ return result;
+ }
+
+ @Override
+ public Map<String, String> subscribeScriptHashes(Transport transport, Wallet wallet, Map<String, String> pathScriptHashes) {
+ //Stands in for a reorg reconciled while this pass is in flight, whether by the sync service or by a history thread's own gate. The real
+ //invalidation is used rather than an imitation of it, since what it clears is half of what the pass must not put back
+ if(invalidateAtForkHeight != null) {
+ ElectrumServer.invalidateScriptHashesForReorg(wallet, invalidateAtForkHeight);
+ invalidateAtForkHeight = null;
+ }
+
+ Map<String, String> result = new LinkedHashMap<>();
+ pathScriptHashes.forEach((path, scriptHash) -> result.put(path, scriptHashStatuses.get(scriptHash)));
+
+ return result;
+ }
+
+ public void invalidateDuringPass(int forkHeight) {
+ this.invalidateAtForkHeight = forkHeight;
+ }
+
+ public void serveHistory(String derivationPath, ScriptHashTx... history) {
+ histories.put(derivationPath, history);
+ }
+
+ public void serveScriptHashStatus(String scriptHash, String status) {
+ scriptHashStatuses.put(scriptHash, status);
+ }
+
+ public void serveTransaction(Transaction transaction) {
+ rawTransactions.put(transaction.getTxId().toString(), Utils.bytesToHex(transaction.bitcoinSerialize()));
+ }
+
+ public TransactionMerkleProof serveProof(Transaction transaction, int height, int position) {
+ TransactionMerkleProof proof = new TransactionMerkleProof();
+ proof.block_height = height;
+ proof.pos = position;
+ proof.merkle = merkleBranch(txids(blockTransactions), position);
+ proofs.put(transaction.getTxId() + ":" + height, proof);
+
+ return proof;
+ }
+
+ public void serveProof(Sha256Hash txid, int height, TransactionMerkleProof proof) {
+ proofs.put(txid + ":" + height, proof);
+ }
+
+ public void refuseFirstAttempts(Transaction transaction, int height, int attempts) {
+ refusals.put(transaction.getTxId() + ":" + height, attempts);
+ }
+
+ public void setProofFailure(RuntimeException proofFailure) {
+ this.proofFailure = proofFailure;
+ }
+
+ /**
+ * Answers the given number of proof requests before failing every one after them, which is a connection lost partway through a verification.
+ */
+ public void failAfterRequests(int requests, RuntimeException failure) {
+ this.proofFailureAfter = requests;
+ this.proofFailure = failure;
+ }
+
+ /**
+ * Fails the given number of proof requests before answering the rest, which is a server momentarily unable to answer the call at all.
+ */
+ public void failFirstRequests(int requests, RuntimeException failure) {
+ this.proofFailureUntil = requests;
+ this.proofFailure = failure;
+ }
+
+ public void setHeadersFailure(RuntimeException headersFailure) {
+ this.headersFailure = headersFailure;
+ }
+
+ public void setServedChainLength(int servedChainLength) {
+ this.servedChainLength = servedChainLength;
+ }
+
+ public int getProofRequests() {
+ return proofRequests.get();
+ }
+
+ public int getBlockHeaderRequests() {
+ return blockHeaderRequests.get();
+ }
+
+ public List<Set<String>> getProofRequestKeys() {
+ return proofRequestKeys;
+ }
+
+ private static String key(BlockTransactionHash reference) {
+ return reference.getHashAsString() + ":" + reference.getHeight();
+ }
+ }
+
+ /**
+ * The events are dispatched on the thread that verified, so they are captured by the time the call returns.
+ */
+ public static class ProofListener {
+ private final Set<BlockTransactionHash> failed = new LinkedHashSet<>();
+ private final Set<BlockTransactionHash> refused = new LinkedHashSet<>();
+ private int failedEvents;
+ private int refusedEvents;
+
+ @Subscribe
+ public void transactionProofsFailed(TransactionProofsFailedEvent event) {
+ failedEvents++;
+ failed.addAll(event.getReferences());
+ }
+
+ @Subscribe
+ public void transactionProofsRefused(TransactionProofsRefusedEvent event) {
+ refusedEvents++;
+ refused.addAll(event.getReferences());
+ }
+
+ public Set<BlockTransactionHash> getFailed() {
+ return failed;
+ }
+
+ public Set<BlockTransactionHash> getRefused() {
+ return refused;
+ }
+
+ public int getFailedEvents() {
+ return failedEvents;
+ }
+
+ public int getRefusedEvents() {
+ return refusedEvents;
+ }
+
+ public boolean isEmpty() {
+ return failedEvents == 0 && refusedEvents == 0;
+ }
+
+ public void reset() {
+ failed.clear();
+ refused.clear();
+ failedEvents = 0;
+ refusedEvents = 0;
+ }
+ }
+
+ /**
+ * A transport that reports itself connected without opening a socket, since the fake answers without it.
+ */
+ private static class UnusedTransport extends TcpTransport {
+ private final boolean connected;
+
+ public UnusedTransport() {
+ this(true);
+ }
+
+ public UnusedTransport(boolean connected) {
+ super(HostAndPort.fromParts("localhost", 1));
+ this.connected = connected;
+ }
+
+ @Override
+ public String pass(String request) {
+ throw new UnsupportedOperationException("The fake server answers without the transport");
+ }
+
+ @Override
+ public boolean isConnected() {
+ return connected;
+ }
+ }
+}
### src/test/java/com/sparrowwallet/sparrow/wallet/WalletFormTest.java
@@ -0,0 +1,188 @@
+package com.sparrowwallet.sparrow.wallet;
+
+import com.sparrowwallet.drongo.ExtendedKey;
+import com.sparrowwallet.drongo.KeyDerivation;
+import com.sparrowwallet.drongo.KeyPurpose;
+import com.sparrowwallet.drongo.Network;
+import com.sparrowwallet.drongo.policy.Policy;
+import com.sparrowwallet.drongo.policy.PolicyType;
+import com.sparrowwallet.drongo.protocol.ScriptType;
+import com.sparrowwallet.drongo.protocol.Sha256Hash;
+import com.sparrowwallet.drongo.wallet.BlockTransaction;
+import com.sparrowwallet.drongo.wallet.BlockTransactionHashIndex;
+import com.sparrowwallet.drongo.wallet.Keystore;
+import com.sparrowwallet.drongo.wallet.Wallet;
+import com.sparrowwallet.drongo.wallet.WalletNode;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Which nodes a wallet update reports as changed, and so which rows the persistence layer rewrites. The case covered here is the one the ordinary
+ * comparison cannot see: a transaction re-proven against a different block at an unchanged height, whose block hash and date are stale on disk until
+ * the nodes holding it are written again.
+ */
+public class WalletFormTest {
+ private static final String TEST_XPUB = "xpub6BosfCnifzxcFwrSzQiqu2DBVTshkCXacvNsWGYJVVhhawA7d4R5WSWGFNbi8Aw6ZRc1brxMyWMzG3DSSSSoekkudhUd9yLb6qx39T9nMdj";
+
+ private static final Sha256Hash TXID = Sha256Hash.wrap("f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16");
+ private static final Sha256Hash ORPHANED_BLOCK = Sha256Hash.wrap("00000000000000000001a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f6");
+ private static final Sha256Hash REPLACING_BLOCK = Sha256Hash.wrap("00000000000000000002b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f607");
+
+ private static final int HEIGHT = 800000;
+
+ @BeforeEach
+ public void setUp() {
+ Network.set(Network.MAINNET);
+ }
+
+ @AfterEach
+ public void tearDown() {
+ Network.set(null);
+ }
+
+ /**
+ * The same-height reorg: the height, the outputs and the wallet's own view of them are all unchanged, and only the block the transaction was
+ * proven against differs. The node has to be reported so that its outputs and its transaction row are written with the replacing block's hash and
+ * timestamp, which are otherwise corrected in memory and lost on restart.
+ */
+ @Test
+ public void reportsTheNodeHoldingATransactionProvenAgainstAnotherBlock() {
+ Wallet wallet = testWallet();
+ WalletNode node = receiveNode(wallet, 0);
+ node.getTransactionOutputs().add(new BlockTransactionHashIndex(TXID, HEIGHT, new Date(1600000000000L), 0L, 0, 10000));
+ wallet.updateTransactions(Map.of(TXID, blockTransaction(new Date(1600000000000L), ORPHANED_BLOCK)));
+
+ Wallet previousWallet = wallet.copy();
+ wallet.updateTransactions(Map.of(TXID, blockTransaction(new Date(1600000600000L), REPLACING_BLOCK)));
+
+ List<WalletNode> changedNodes = new ArrayList<>();
+ WalletForm.addReprovenNodes(wallet, previousWallet, changedNodes);
+
+ assertEquals(List.of(node), changedNodes);
+ }
+
+ /**
+ * A node whose output was spent in the replaced block is the same case, since the spend carries the stale height's date too.
+ */
+ @Test
+ public void reportsTheNodeWhoseSpendWasProvenAgainstAnotherBlock() {
+ Wallet wallet = testWallet();
+ WalletNode node = receiveNode(wallet, 0);
+ BlockTransactionHashIndex output = new BlockTransactionHashIndex(Sha256Hash.ZERO_HASH, 700000, new Date(1500000000000L), 0L, 0, 10000);
+ output.setSpentBy(new BlockTransactionHashIndex(TXID, HEIGHT, new Date(1600000000000L), 0L, 0, 10000));
+ node.getTransactionOutputs().add(output);
+ wallet.updateTransactions(Map.of(TXID, blockTransaction(new Date(1600000000000L), ORPHANED_BLOCK)));
+
+ Wallet previousWallet = wallet.copy();
+ wallet.updateTransactions(Map.of(TXID, blockTransaction(new Date(1600000600000L), REPLACING_BLOCK)));
+
+ List<WalletNode> changedNodes = new ArrayList<>();
+ WalletForm.addReprovenNodes(wallet, previousWallet, changedNodes);
+
+ assertEquals(List.of(node), changedNodes);
+ }
+
+ /**
+ * The non-regression that matters most: a wallet whose transactions predate the feature carries no block hash at all, and an ordinary pass must
+ * report nothing here, or every node would be rewritten on every update.
+ */
+ @Test
+ public void reportsNothingWhereNoBlockWasProvenAgainstAnother() {
+ Wallet wallet = testWallet();
+ WalletNode node = receiveNode(wallet, 0);
+ node.getTransactionOutputs().add(new BlockTransactionHashIndex(TXID, HEIGHT, new Date(1600000000000L), 0L, 0, 10000));
+
+ //No block hash at all, as everything written before this feature has
+ wallet.updateTransactions(Map.of(TXID, blockTransaction(new Date(1600000000000L), null)));
+ Wallet previousWallet = wallet.copy();
+ List<WalletNode> changedNodes = new ArrayList<>();
+ WalletForm.addReprovenNodes(wallet, previousWallet, changedNodes);
+ assertTrue(changedNodes.isEmpty());
+
+ //And a pass that rewrites the transaction without changing the block it was proven against
+ wallet.updateTransactions(Map.of(TXID, blockTransaction(new Date(1600000000000L), null)));
+ WalletForm.addReprovenNodes(wallet, previousWallet, changedNodes);
+ assertTrue(changedNodes.isEmpty());
+
+ wallet.updateTransactions(Map.of(TXID, blockTransaction(new Date(1600000000000L), ORPHANED_BLOCK)));
+ Wallet provenWallet = wallet.copy();
+ wallet.updateTransactions(Map.of(TXID, blockTransaction(new Date(1600000000000L), ORPHANED_BLOCK)));
+ WalletForm.addReprovenNodes(wallet, provenWallet, changedNodes);
+ assertTrue(changedNodes.isEmpty());
+ }
+
+ /**
+ * A block hash follows the height, so it changes on every ordinary confirmation, demotion and unfetchable transaction too. In each of those the
+ * node has an output at a new height and the ordinary comparison has already reported it, so this must not walk the wallet again to reach it.
+ */
+ @Test
+ public void reportsNothingWhereTheHeightChangedWithTheBlock() {
+ Wallet wallet = testWallet();
+ WalletNode node = receiveNode(wallet, 0);
+ node.getTransactionOutputs().add(new BlockTransactionHashIndex(TXID, 0, null, 0L, 0, 10000));
+ wallet.updateTransactions(Map.of(TXID, new BlockTransaction(TXID, 0, null, 0L, null, null)));
+
+ //Confirmed: a height and a block hash where there was neither
+ Wallet previousWallet = wallet.copy();
+ wallet.updateTransactions(Map.of(TXID, blockTransaction(new Date(1600000000000L), ORPHANED_BLOCK)));
+ List<WalletNode> changedNodes = new ArrayList<>();
+ WalletForm.addReprovenNodes(wallet, previousWallet, changedNodes);
+ assertTrue(changedNodes.isEmpty());
+
+ //Demoted: the block hash goes away with the height
+ previousWallet = wallet.copy();
+ wallet.updateTransactions(Map.of(TXID, new BlockTransaction(TXID, 0, null, 0L, null, null)));
+ WalletForm.addReprovenNodes(wallet, previousWallet, changedNodes);
+ assertTrue(changedNodes.isEmpty());
+ }
+
+ /**
+ * A node already reported by the ordinary comparison is not reported twice.
+ */
+ @Test
+ public void doesNotReportANodeTwice() {
+ Wallet wallet = testWallet();
+ WalletNode node = receiveNode(wallet, 0);
+ node.getTransactionOutputs().add(new BlockTransactionHashIndex(TXID, HEIGHT, new Date(1600000000000L), 0L, 0, 10000));
+ wallet.updateTransactions(Map.of(TXID, blockTransaction(new Date(1600000000000L), ORPHANED_BLOCK)));
+
+ Wallet previousWallet = wallet.copy();
+ wallet.updateTransactions(Map.of(TXID, blockTransaction(new Date(1600000600000L), REPLACING_BLOCK)));
+
+ List<WalletNode> changedNodes = new ArrayList<>(List.of(node));
+ WalletForm.addReprovenNodes(wallet, previousWallet, changedNodes);
+
+ assertEquals(List.of(node), changedNodes);
+ }
+
+ private static BlockTransaction blockTransaction(Date date, Sha256Hash blockHash) {
+ return new BlockTransaction(TXID, HEIGHT, date, 0L, null, blockHash);
+ }
+
+ private static WalletNode receiveNode(Wallet wallet, int index) {
+ return wallet.getNode(KeyPurpose.RECEIVE).getChildren().stream().filter(node -> node.getIndex() == index).findFirst().orElseThrow();
+ }
+
+ private static Wallet testWallet() {
+ Wallet wallet = new Wallet();
+ wallet.setPolicyType(PolicyType.SINGLE_HD);
+ wallet.setScriptType(ScriptType.P2WPKH);
+ Keystore keystore = new Keystore();
+ keystore.setKeyDerivation(new KeyDerivation("00000000", "m/84'/0'/0'"));
+ keystore.setExtendedPublicKey(ExtendedKey.fromDescriptor(TEST_XPUB));
+ wallet.getKeystores().add(keystore);
+ wallet.setDefaultPolicy(Policy.getPolicy(PolicyType.SINGLE_HD, ScriptType.P2WPKH, wallet.getKeystores(), 1));
+ wallet.getNode(KeyPurpose.RECEIVE).fillToIndex(wallet, 1);
+
+ return wallet;
+ }
+}Why this scored 68/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.