maintain a verified block header store anchored at the pinned checkpoints, refreshing affected wallets on a reorg
What changed, and why it matters
This commit adds a verified block-header store to the Sparrow Bitcoin wallet. It keeps a local copy of block headers above the built-in checkpoints, checks that each new header really links to the previous one, and automatically rewinds and refreshes wallet history if the Bitcoin chain reorganizes (a 'reorg'). The goal is to stop a malicious or buggy Electrum server from fooling the wallet with fake transaction confirmations.
Review the new HeaderStore and ElectrumServer header-sync paths for concurrency and error-handling correctness; exercise the reorg and deep-fork test scenarios; ensure the MAX_REORG_DEPTH choice and the 'equal work is adopted' policy are acceptable for the threat model. No immediate patch is indicated by the diff itself.
Security signals we found
Adds local verification of block headers against compiled-in checkpoints and proof-of-work rules
Handles chain reorgs by rewinding the header store and refreshing wallet history above the fork
Pairs block height and header together to prevent race conditions where height and header come from different blocks
Serializes header sync with headerSyncLock to prevent concurrent fetches from an honest server being misread as a fork
Distinguishes server refusal (VerificationException -> disable verification) from transport failure (ServerException) and unsupported method (UnsupportedMethodException)
Stores header cache in a network-specific, owner-only directory under the application cache
Evidence from the diff
The change introduces HeaderStore (flat file of 80-byte headers above pinned checkpoints), HeaderSyncService (event-driven sync to the announced chain tip), and ChainTip (height+header pair to avoid torn reads). ElectrumServer now verifies header linkage, proof-of-work/difficulty, and reorgs up to MAX_REORG_DEPTH (100). On reorg it dispatches ChainReorgEvent; WalletForm invalidates affected script-hash caches and refreshes. Historical headers below the last pin are verified by hash linkage to a pin. Extensive unit tests cover reorgs, torn writes, superseded stores, and concurrency.
Changed components
com.sparrowwallet.sparrow.AppServicescom.sparrowwallet.sparrow.ChainTipcom.sparrowwallet.sparrow.event.ChainReorgEventcom.sparrowwallet.sparrow.io.Storagecom.sparrowwallet.sparrow.net.ElectrumServercom.sparrowwallet.sparrow.net.HeaderStorecom.sparrowwallet.sparrow.wallet.WalletFormInspect captured patch +2192 / −23
### src/main/java/com/sparrowwallet/sparrow/AppServices.java
@@ -129,9 +129,7 @@ public class AppServices {
private ScheduledService<Void> preventSleepService;
- private static Integer currentBlockHeight;
-
- private static BlockHeader latestBlockHeader;
+ private static volatile ChainTip announcedTip;
private static final Map<Integer, BlockSummary> blockSummaries = new ConcurrentHashMap<>();
@@ -207,6 +205,7 @@ private AppServices(Application application, InteractionServices interactionServ
public void start() {
Config config = Config.get();
connectionService = createConnectionService();
+ registerHeaderSyncService();
feeRatesService = createFeeRatesService();
ratesService = createRatesService(config.getExchangeSource(), config.getFiatCurrency());
versionCheckService = createVersionCheckService();
@@ -395,6 +394,25 @@ private ElectrumServer.FeeRatesService createFeeRatesService() {
return feeRatesService;
}
+ /**
+ * The header sync service is driven entirely by the events it subscribes to - started by an announced tip, cancelled on disconnection - so nothing
+ * here holds it or restarts it. Registering it with the event bus is what keeps it reachable for the life of the session.
+ */
+ private void registerHeaderSyncService() {
+ ElectrumServer.HeaderSyncService headerSyncService = new ElectrumServer.HeaderSyncService();
+ headerSyncService.setPeriod(Duration.seconds(ElectrumServer.HeaderSyncService.RETRY_PERIOD_SECS));
+ headerSyncService.setRestartOnFailure(true);
+ EventManager.get().register(headerSyncService);
+
+ //The service is started by the tip it is told about, so a successful run has nothing left to do until the next one
+ headerSyncService.setOnSucceeded(successEvent -> {
+ headerSyncService.cancel();
+ });
+ headerSyncService.setOnFailed(failEvent -> {
+ log.warn("Failed to sync block headers, retrying in " + ElectrumServer.HeaderSyncService.RETRY_PERIOD_SECS + "s", failEvent.getSource().getException());
+ });
+ }
+
private ExchangeSource.RatesService createRatesService(ExchangeSource exchangeSource, Currency currency) {
ExchangeSource.RatesService ratesService = new ExchangeSource.RatesService(
exchangeSource == null ? DEFAULT_EXCHANGE_SOURCE : exchangeSource,
@@ -738,11 +756,25 @@ public static BooleanProperty onlineProperty() {
}
public static Integer getCurrentBlockHeight() {
- return currentBlockHeight;
+ ChainTip tip = announcedTip;
+ return tip == null ? null : tip.height();
}
public static BlockHeader getLatestBlockHeader() {
- return latestBlockHeader;
+ ChainTip tip = announcedTip;
+ return tip == null ? null : tip.header();
+ }
+
+ /**
+ * The chain tip as the connected server last announced it, whose height and header are written together. A reader needing both must take them from
+ * one of these, since the two accessors above read the tip separately and can straddle a new block, pairing a new height with the previous header.
+ */
+ public static ChainTip getAnnouncedTip() {
+ return announcedTip;
+ }
+
+ public static void setAnnouncedTip(ChainTip announcedTip) {
+ AppServices.announcedTip = announcedTip;
}
public static Map<Integer, BlockSummary> getBlockSummaries() {
@@ -1281,13 +1313,12 @@ public static boolean isOnWayland() {
@Subscribe
public void newConnection(ConnectionEvent event) {
- currentBlockHeight = event.getBlockHeight();
- System.setProperty(Network.BLOCK_HEIGHT_PROPERTY, Integer.toString(currentBlockHeight));
+ setAnnouncedTip(new ChainTip(event.getBlockHeight(), event.getBlockHeader()));
+ System.setProperty(Network.BLOCK_HEIGHT_PROPERTY, Integer.toString(event.getBlockHeight()));
if(getConfiguredMinimumRelayFeeRate(Config.get()) == null) {
minimumRelayFeeRate = event.getMinimumRelayFeeRate() == null ? Transaction.DEFAULT_MIN_RELAY_FEE : event.getMinimumRelayFeeRate();
}
serverMinimumRelayFeeRate = event.getMinimumRelayFeeRate();
- latestBlockHeader = event.getBlockHeader();
Config.get().addRecentServer();
FeeRatesSource feeRatesSource = Config.get().getFeeRatesSource();
@@ -1296,7 +1327,7 @@ public void newConnection(ConnectionEvent event) {
fetchFeeRates();
}
- if(!blockSummaries.containsKey(currentBlockHeight)) {
+ if(!blockSummaries.containsKey(getCurrentBlockHeight())) {
fetchBlockSummaries(Collections.emptyList());
}
}
@@ -1308,9 +1339,8 @@ public void usbDevicesFound(UsbDeviceEvent event) {
@Subscribe
public void newBlock(NewBlockEvent event) {
- currentBlockHeight = event.getHeight();
- System.setProperty(Network.BLOCK_HEIGHT_PROPERTY, Integer.toString(currentBlockHeight));
- latestBlockHeader = event.getBlockHeader();
+ setAnnouncedTip(new ChainTip(event.getHeight(), event.getBlockHeader()));
+ System.setProperty(Network.BLOCK_HEIGHT_PROPERTY, Integer.toString(event.getHeight()));
String status = "Updating to new block height " + event.getHeight();
EventManager.get().post(new StatusEvent(status));
newBlockSubject.onNext(event);
@@ -1319,8 +1349,9 @@ public void newBlock(NewBlockEvent event) {
@Subscribe
public void blockSummary(BlockSummaryEvent event) {
blockSummaries.putAll(event.getBlockSummaryMap());
- if(AppServices.currentBlockHeight != null) {
- blockSummaries.keySet().removeIf(height -> AppServices.currentBlockHeight - height > 5);
+ Integer currentBlockHeight = getCurrentBlockHeight();
+ if(currentBlockHeight != null) {
+ blockSummaries.keySet().removeIf(height -> currentBlockHeight - height > 5);
}
nextBlockMedianFeeRate = event.getNextBlockMedianFeeRate();
}
### src/main/java/com/sparrowwallet/sparrow/ChainTip.java
@@ -0,0 +1,9 @@
+package com.sparrowwallet.sparrow;
+
+import com.sparrowwallet.drongo.protocol.BlockHeader;
+
+/**
+ * The height and header of a chain tip, carried together so that a reader cannot take the height of one block with the header of another.
+ * Whether a tip is one a server has announced or one that has been verified is said by the accessor or the field holding it, not by this type.
+ */
+public record ChainTip(int height, BlockHeader header) {}
### src/main/java/com/sparrowwallet/sparrow/event/ChainReorgEvent.java
@@ -0,0 +1,21 @@
+package com.sparrowwallet.sparrow.event;
+
+/**
+ * Posted once the verified header chain has been rewound to a fork point and the connected server's headers adopted above it. Every open wallet
+ * invalidates the cached status of the nodes the fork affects and refreshes, so that heights proven against a header that is no longer on the chain
+ * are proven again.
+ * <p>
+ * This is dispatched on the thread that synced the headers - a wallet history thread as often as the header sync service - while it holds the sync
+ * lock, so a handler must hop to the application thread itself rather than doing anything lengthy here.
+ */
+public class ChainReorgEvent {
+ private final int forkHeight;
+
+ public ChainReorgEvent(int forkHeight) {
+ this.forkHeight = forkHeight;
+ }
+
+ public int getForkHeight() {
+ return forkHeight;
+ }
+}
### src/main/java/com/sparrowwallet/sparrow/io/Storage.java
@@ -44,6 +44,7 @@ public class Storage {
public static final String WALLETS_DIR = "wallets";
public static final String WALLETS_BACKUP_DIR = "backup";
public static final String CERTS_DIR = "certs";
+ public static final String HEADERS_DIR = "headers";
public static final List<String> RESERVED_WALLET_NAMES = List.of("temp");
private Persistence persistence;
@@ -549,6 +550,18 @@ static File getCertsDir() {
return certsDir;
}
+ /**
+ * Returns the network specific directory containing the verified block headers, which are regenerable and so kept with the other caches.
+ */
+ public static File getHeadersDir() {
+ File headersDir = new File(getCacheDir(), HEADERS_DIR);
+ if(!headersDir.exists()) {
+ createOwnerOnlyDirectory(headersDir);
+ }
+
+ return headersDir;
+ }
+
/**
* Returns the network specific directory containing the configuration file.
*/
### src/main/java/com/sparrowwallet/sparrow/net/ElectrumServer.java
@@ -18,6 +18,7 @@
import com.sparrowwallet.drongo.wallet.*;
import com.sparrowwallet.sparrow.AppServices;
import com.sparrowwallet.sparrow.BlockSummary;
+import com.sparrowwallet.sparrow.ChainTip;
import com.sparrowwallet.sparrow.EventManager;
import com.sparrowwallet.sparrow.event.*;
import com.sparrowwallet.sparrow.io.Config;
@@ -69,7 +70,7 @@ public class ElectrumServer {
public static final BlockTransaction UNFETCHABLE_BLOCK_TRANSACTION = new BlockTransaction(Sha256Hash.ZERO_HASH, 0, null, null, null);
- private static CloseableTransport transport;
+ static CloseableTransport transport;
private static final Map<String, List<String>> subscribedScriptHashes = new ConcurrentHashMap<>();
@@ -85,12 +86,34 @@ public class ElectrumServer {
private static final Set<String> sameHeightTxioScriptHashes = ConcurrentHashMap.newKeySet();
+ static final Set<String> reorgInvalidatedScriptHashes = ConcurrentHashMap.newKeySet();
+
+ static volatile HeaderStore headerStore;
+
+ //Not the ElectrumServer class monitor that getTransport() uses: a store load or re-walk would then block every transport acquisition for its duration
+ private static final Object headerStoreLock = new Object();
+
+ //Serialises fetch and append across the header sync service and the wallet history threads, which sync concurrently by design
+ private static final Object headerSyncLock = new Object();
+
+ //Session cache of headers below the last pin, verified by hash linkage to it and deliberately never persisted
+ static final Map<Integer, BlockHeader> verifiedHistoricalHeaders = new ConcurrentHashMap<>();
+
+ //The deepest fork point the store has been rewound to this session, at or above which a stored height may have been proven against an orphaned
+ //header. Written only under headerSyncLock, which is what makes the min in reconcile atomic; volatile is for the readers that do not take it
+ static volatile int lastReorgForkHeight = Integer.MAX_VALUE;
+
private static final Map<Integer, WalletSyncLock> walletSyncLocks = Collections.synchronizedMap(new HashMap<>());
private static final Map<String, SilentPaymentsScanCache> spScanCaches = new ConcurrentHashMap<>();
private static final int TAPROOT_ACTIVATION_HEIGHT = 709632;
+ private static final int HEADERS_CHUNK_SIZE = HeaderChainState.RETARGET_INTERVAL;
+
+ //A reorg deeper than this is a global event rather than a client concern, and the store keeps the heavier chain it already has
+ private static final int MAX_REORG_DEPTH = 100;
+
//Consensus rejects blocks timestamped more than 2 hours in the future, extended here to allow for local clock skew
private static final long MAXIMUM_FUTURE_TIP_TIME_SECS = 4 * 60 * 60;
@@ -111,7 +134,7 @@ public class ElectrumServer {
private final static Map<String, String> broadcastRecent = new ConcurrentHashMap<>();
- private static ElectrumServerRpc electrumServerRpc = new SimpleElectrumServerRpc();
+ static ElectrumServerRpc electrumServerRpc = new SimpleElectrumServerRpc();
private static Cormorant cormorant;
@@ -160,6 +183,7 @@ private static synchronized CloseableTransport getTransport() throws ServerExcep
retrievedScriptHashes.clear();
retrievedTransactions.clear();
retrievedBlockHeaders.clear();
+ reorgInvalidatedScriptHashes.clear();
walletSyncLocks.values().forEach(syncLock -> syncLock.scriptHashesInitialized = false);
}
previousServer = electrumServer;
@@ -329,6 +353,39 @@ private static void clearRetrievedScriptHash(String scriptHash) {
sameHeightTxioScriptHashes.remove(scriptHash);
}
+ /**
+ * Invalidates the cached status of every node holding a transaction output, or a spend of one, above the given fork point, so that the reorganised
+ * history is fetched again. A transaction re-included at the same height leaves the server reporting an unchanged status, and without this the
+ * node would never be revisited. This touches cache state only, never wallet data.
+ * <p>
+ * Returns whether any node was invalidated, which is false for a wallet holding nothing above the fork: it can have proven nothing against a
+ * header that was discarded, so there is nothing for it to fetch again.
+ */
+ public static boolean invalidateScriptHashesForReorg(Wallet wallet, int forkHeight) {
+ boolean invalidated = invalidateWalletScriptHashesForReorg(wallet, forkHeight);
+ for(Wallet childWallet : new ArrayList<>(wallet.getChildWallets())) {
+ if(childWallet.isNested()) {
+ invalidated |= invalidateWalletScriptHashesForReorg(childWallet, forkHeight);
+ }
+ }
+
+ return invalidated;
+ }
+
+ private static boolean invalidateWalletScriptHashesForReorg(Wallet wallet, int forkHeight) {
+ boolean invalidated = false;
+ for(Map.Entry<WalletNode, Set<BlockTransactionHashIndex>> entry : wallet.getWalletNodes().entrySet()) {
+ if(entry.getValue().stream().anyMatch(txo -> txo.getHeight() > forkHeight || (txo.getSpentBy() != null && txo.getSpentBy().getHeight() > forkHeight))) {
+ String scriptHash = getScriptHash(entry.getKey());
+ clearRetrievedScriptHash(scriptHash);
+ reorgInvalidatedScriptHashes.add(scriptHash);
+ invalidated = true;
+ }
+ }
+
+ return invalidated;
+ }
+
public boolean fetchAndCalculateHistory(Wallet mainWallet, List<Wallet> filterToWallets, Set<WalletNode> filterToNodes) throws ServerException {
boolean historyFetched = fetchAndCalculateWalletHistory(mainWallet, filterToWallets, filterToNodes);
for(Wallet childWallet : new ArrayList<>(mainWallet.getChildWallets())) {
@@ -363,6 +420,9 @@ private boolean fetchAndCalculateWalletHistory(Wallet wallet, List<Wallet> filte
getReferencedTransactions(wallet, nodeTransactionMap);
calculateNodeHistory(wallet, nodeTransactionMap);
+ //A node invalidated by a reorg has no retrieved status left to compare against, so it must not count as changed history below
+ Set<String> invalidatedScriptHashes = Set.copyOf(reorgInvalidatedScriptHashes);
+
//Add all of the script hashes we have now fetched the history for so we don't need to fetch again until the script hash status changes
Set<WalletNode> updatedNodes = new HashSet<>();
Map<WalletNode, Set<BlockTransactionHashIndex>> walletNodes = wallet.getWalletNodes();
@@ -377,15 +437,21 @@ private boolean fetchAndCalculateWalletHistory(Wallet wallet, List<Wallet> filte
//If wallet was not empty, check if all used updated nodes have changed history
if(nodes == null && previousScriptHashes.values().stream().anyMatch(Objects::nonNull)) {
- if(!updatedNodes.isEmpty()
- && updatedNodes.equals(walletNodes.entrySet().stream().filter(entry -> !entry.getValue().isEmpty()).map(Map.Entry::getKey).collect(Collectors.toSet()))
- && !sameHeightTxioScriptHashes.containsAll(updatedNodes.stream().map(ElectrumServer::getScriptHash).collect(Collectors.toSet()))) {
+ Set<WalletNode> changedNodes = updatedNodes.stream().filter(node -> !invalidatedScriptHashes.contains(getScriptHash(node))).collect(Collectors.toSet());
+ if(!changedNodes.isEmpty()
+ && changedNodes.equals(walletNodes.entrySet().stream().filter(entry -> !entry.getValue().isEmpty()).map(Map.Entry::getKey).collect(Collectors.toSet()))
+ && !sameHeightTxioScriptHashes.containsAll(changedNodes.stream().map(ElectrumServer::getScriptHash).collect(Collectors.toSet()))) {
//All used nodes on a non-empty wallet have changed history. Abort and trigger a full refresh.
log.info("All used nodes on a non-empty wallet have changed history. Triggering a full wallet refresh.");
throw new AllHistoryChangedException();
}
}
+ //The reorg exemption lasts for exactly one full fetch, and is cleared only once the check above has passed
+ if(nodes == null && !invalidatedScriptHashes.isEmpty()) {
+ reorgInvalidatedScriptHashes.removeAll(walletNodes.keySet().stream().map(ElectrumServer::getScriptHash).collect(Collectors.toSet()));
+ }
+
//Clear transaction outputs for nodes that have no history - this is useful when a transaction is replaced in the mempool
if(nodes != null) {
for(WalletNode node : nodes) {
@@ -845,6 +911,307 @@ public Map<Integer, BlockHeader> getBlockHeaders(Wallet wallet, Set<BlockTransac
}
}
+ /**
+ * The header store for this network, loaded on first use from a background thread. It is not cleared when the server changes: headers are claims
+ * about the chain rather than about the server, and a new server announcing a different tip is handled as an ordinary reorg.
+ */
+ static HeaderStore getHeaderStore() throws ServerException {
+ try {
+ HeaderStore store = headerStore;
+ if(store != null && store.isIntact()) {
+ return store;
+ }
+
+ synchronized(headerStoreLock) {
+ //A loaded store outlives the connection, so its file may have been removed or truncated underneath it since it was read
+ if(headerStore == null || !headerStore.isIntact()) {
+ headerStore = HeaderStore.load(Network.get().getHeaderCheckpoints());
+ }
+
+ return headerStore;
+ }
+ } catch(IOException e) {
+ throw new ServerException("Could not load the block header store", e);
+ }
+ }
+
+ /**
+ * Advances the store to the tip the server has announced, taken as one value so that a height and a header from either side of a new block cannot
+ * be mixed. Every chain problem - a linkage or difficulty failure, a short, empty or malformed chunk, or a reorg candidate that cannot
+ * be accepted - is thrown as a VerificationException, meaning this server cannot substantiate these heights; transport problems propagate as they
+ * are, meaning the session is broken.
+ */
+ void syncHeaders(ChainTip tip) throws ServerException {
+ synchronized(headerSyncLock) {
+ HeaderStore store = getHeaderStore();
+ try {
+ if(tip == null || tip.height() < store.getStartHeight()) {
+ return; //no tip yet, or a server that has not caught up to the last pinned header
+ }
+
+ if(tip.height() <= store.getTipHeight() && !tip.header().getHash().equals(store.getHash(tip.height()))) {
+ reconcile(store, tip.height()); //the tie form of a divergence, which the loop below would never examine
+ }
+
+ syncTo(store, tip.height(), tip);
+ } catch(IOException e) {
+ throw new ServerException("Could not write to the block header store", e);
+ }
+ }
+ }
+
+ /**
+ * Advances the store to the given height on the calling thread, for a wallet history that has reached a height the sync service has not, under the
+ * exception contract of syncHeaders above. Both are internal to the header sync: it is getVerifiedHeader that turns what they throw into the
+ * refusal, failure and unsupported outcomes its callers act on.
+ */
+ void syncHeadersTo(int height) throws ServerException {
+ synchronized(headerSyncLock) {
+ HeaderStore store = getHeaderStore();
+ try {
+ syncTo(store, height, AppServices.getAnnouncedTip());
+ } catch(IOException e) {
+ throw new ServerException("Could not write to the block header store", e);
+ }
+ }
+ }
+
+ private void syncTo(HeaderStore store, int targetHeight, ChainTip tip) throws ServerException, IOException {
+ while(store.getTipHeight() < targetHeight) {
+ if(tip != null && tip.height() == store.getTipHeight() + 1 && tip.header().getPrevBlockHash().equals(store.getTipHash())) {
+ store.append(tip.header()); //the steady state: the announced header extends the store, so there is nothing to fetch
+ continue;
+ }
+
+ int startHeight = store.getTipHeight() + 1;
+ BlockHeaders chunk = electrumServerRpc.getBlockHeadersChunk(getTransport(), startHeight, HEADERS_CHUNK_SIZE);
+ if(chunk.count == 0) {
+ throw new VerificationException("Server returned no headers from height " + startHeight + " while the store must reach height " + targetHeight);
+ }
+
+ List<BlockHeader> headers;
+ try {
+ //Parsed whole here rather than one at a time below, so that a chunk carrying fewer headers than it claims is a refusal like any other
+ //malformed response rather than an exception escaping the sync
+ byte[] bytes = Utils.hexToBytes(chunk.hex);
+ headers = IntStream.range(0, chunk.count).mapToObj(i -> new BlockHeader(bytes, i * HeaderStore.HEADER_LENGTH)).toList();
+ } catch(ProtocolException | IllegalArgumentException e) {
+ //Refusal class, as a short chunk is: the server could not substantiate the range
+ throw new VerificationException("Server returned a malformed header chunk from height " + startHeight, e);
+ }
+
+ if(!headers.getFirst().getPrevBlockHash().equals(store.getTipHash())) {
+ //The server's chain diverges from the store, so reconcile here, then re-read the tip and fetch again
+ reconcile(store, targetHeight);
+ continue;
+ }
+
+ store.append(headers);
+ }
+ }
+
+ /**
+ * Rewinds the store to the fork point it shares with the server's chain and adopts the server's headers above it, where they carry at least as much
+ * work. Equal work is the same height tie of a stale block and must be adopted: a client with one server can only verify against the chain that
+ * server serves, and staying on the replaced block would report every proof from the replacing one as dishonest.
+ */
+ private void reconcile(HeaderStore store, int tipHeight) throws ServerException, IOException {
+ //The fork point is at or below the store tip whatever the server has announced, so the window sits there rather than at the announced tip: a
+ //server hundreds of blocks ahead of a diverged store would otherwise be searched over a range holding no height the store has
+ int endHeight = Math.min(tipHeight, store.getTipHeight() + 1);
+ int startHeight = Math.max(endHeight - MAX_REORG_DEPTH + 1, store.getStartHeight());
+ int count = endHeight - startHeight + 1;
+ BlockHeaders chunk = electrumServerRpc.getBlockHeadersChunk(getTransport(), startHeight, count);
+ if(chunk.count != count) {
+ throw new VerificationException("Server returned " + chunk.count + " of " + count + " headers when reconciling to height " + endHeight);
+ }
+
+ List<BlockHeader> candidate;
+ try {
+ byte[] bytes = Utils.hexToBytes(chunk.hex);
+ candidate = IntStream.range(0, count).mapToObj(i -> new BlockHeader(bytes, i * HeaderStore.HEADER_LENGTH)).toList();
+ } catch(ProtocolException | IllegalArgumentException e) {
+ throw new VerificationException("Server returned a malformed header chunk when reconciling to height " + endHeight, e);
+ }
+
+ //Walk back from the announced tip, checking each header against the one below it, until one descends from a header the store already holds
+ int forkHeight = -1;
+ for(int i = count - 1; i >= 0; i--) {
+ if(candidate.get(i).getPrevBlockHash().equals(store.getHash(startHeight + i - 1))) {
+ forkHeight = startHeight + i - 1;
+ break;
+ }
+ if(i == 0 || !candidate.get(i).getPrevBlockHash().equals(candidate.get(i - 1).getHash())) {
+ break;
+ }
+ }
+
+ if(forkHeight < 0) {
+ throw new VerificationException("Server's chain at height " + endHeight + " shares no fork point with the last " + count + " verified headers");
+ }
+
+ List<BlockHeader> segment = candidate.subList(forkHeight - startHeight + 1, count);
+ HeaderChainState candidateState = store.chainStateAt(forkHeight);
+ for(BlockHeader header : segment) {
+ candidateState.add(header);
+ }
+
+ //Both chains measured from the same pinned anchor, so the work they share below the fork cancels and what remains is what would be adopted
+ //against what would be discarded. In a forward sync the candidate reaches one header past the store tip, which is why it is heavier: the
+ //server holding a header there is what exposed the divergence
+ if(candidateState.getChainWork().compareTo(store.getChainWork()) < 0) {
+ throw new VerificationException("Server's chain from height " + (forkHeight + 1) + " carries less work than the "
+ + (store.getTipHeight() - forkHeight) + " verified headers it would replace");
+ }
+
+ log.info("Reorganising the block header store at height " + forkHeight + ", replacing " + (store.getTipHeight() - forkHeight) + " headers with " + segment.size());
+ store.truncate(forkHeight);
+ lastReorgForkHeight = Math.min(lastReorgForkHeight, forkHeight);
+ try {
+ store.append(segment);
+ } finally {
+ //The truncation is what the wallets have to hear about, whether or not the replacement was written: a height above the fork was proven
+ //against a header the store no longer holds either way. Dispatched on this thread, which is a wallet history thread as often as it is the
+ //sync service: the wallet handler hops to the FX thread itself
+ EventManager.get().post(new ChainReorgEvent(forkHeight));
+ }
+ }
+
+ /**
+ * The header at the given height verified against the compiled-in checkpoints, or null where the connected server cannot substantiate it, which is
+ * reported as a refusal. Heights above the last pin are served from the store, and those below it by hash linkage to a pin.
+ */
+ public BlockHeader getVerifiedHeader(int height) throws ServerException {
+ HeaderCheckpoints checkpoints = Network.get().getHeaderCheckpoints();
+ if(height > checkpoints.getMaxHeight()) {
+ HeaderStore store = getHeaderStore();
+ try {
+ //One object written once per event: the height and the header read separately can straddle a new block
+ ChainTip announced = AppServices.getAnnouncedTip();
+ if(announced != null && announced.height() >= store.getStartHeight() && announced.height() <= store.getTipHeight()
+ && !announced.header().getHash().equals(store.getHash(announced.height()))) {
+ //Never serve a stored header while the store tip disagrees with the announced tip: the tie form of a reorg
+ syncHeaders(announced);
+ } else if(height > store.getTipHeight()) {
+ syncHeadersTo(height); //the sync service has not caught up, so fetch on this thread
+ }
+
+ return store.getHeader(height);
+ } catch(UnsupportedMethodException e) {
+ throw e; //before the catch below, so the caller can disable verification for the session rather than reading a refusal
+ } catch(VerificationException e) {
+ log.warn("Could not verify the header chain to height " + height + ": " + e.getMessage());
+ return null;
+ } catch(ElectrumServerRpcException e) {
+ throw new ServerException(e.getMessage(), e.getCause()); //the server said nothing about this height, so it is a failed call rather than a refusal
+ } catch(IOException e) {
+ throw new ServerException("Could not read the block header store", e);
+ }
+ }
+
+ if(height == 0) {
+ return Network.get().getGenesisHeader();
+ }
+
+ BlockHeader cached = verifiedHistoricalHeaders.get(height);
+ if(cached != null) {
+ return cached;
+ }
+
+ //One call at a time crosses the transport, so this lock defers no fetch the connection would not have deferred anyway. What it buys is dedup: a
+ //concurrent wallet wanting this range finds it cached, and one wanting an overlapping range fetches only the part below what is now cached.
+ //Were the transport ever to carry calls concurrently, this would become the limiter and would want an in flight map keyed by range instead
+ synchronized(headerSyncLock) {
+ cached = verifiedHistoricalHeaders.get(height);
+ if(cached != null) {
+ return cached;
+ }
+
+ //A header whose hash chain reaches a verified hash is that hash's ancestor at the corresponding depth, so linkage is the whole proof. The
+ //anchor is the nearest header already verified this session, and the pin above the height where there is none, which keeps a second pass
+ //over an already fetched period from downloading it again
+ int pinnedHeight = checkpoints.getPinnedHeightAtOrAbove(height);
+ int anchorHeight = pinnedHeight;
+ Sha256Hash anchorHash = checkpoints.getHash(pinnedHeight);
+ for(int above = height + 1; above < pinnedHeight; above++) {
+ BlockHeader verified = verifiedHistoricalHeaders.get(above);
+ if(verified != null) {
+ anchorHeight = above;
+ anchorHash = verified.getHash();
+ break;
+ }
+ }
+
+ int count = anchorHeight - height + 1;
+ BlockHeaders chunk;
+ try {
+ chunk = electrumServerRpc.getBlockHeadersChunk(getTransport(), height, count);
+ } catch(UnsupportedMethodException e) {
+ throw e; //before the catch below, so the caller can disable verification for the session rather than reading a refusal
+ } catch(VerificationException e) {
+ return null; //a short or malformed response is refusal class here as in the forward sync, never a session failure
+ } catch(ElectrumServerRpcException e) {
+ throw new ServerException(e.getMessage(), e.getCause());
+ }
+
+ List<BlockHeader> headers = getLinkedHeaders(chunk, count, anchorHash);
+ if(headers == null) {
+ return null;
+ }
+
+ for(int i = 0; i < count; i++) {
+ verifiedHistoricalHeaders.put(height + i, headers.get(i));
+ }
+
+ return headers.getFirst();
+ }
+ }
+
+ /**
+ * The headers of a requested range verified by hash linkage to the given anchor hash, which is the hash the last header of the range must have, or
+ * null where the response is short or malformed or the chain does not reach the anchor. No proof of work, difficulty or timestamp check is needed
+ * below a pinned header: descent from the pin is what places a header at its height.
+ */
+ static List<BlockHeader> getLinkedHeaders(BlockHeaders chunk, int count, Sha256Hash anchorHash) {
+ if(chunk.count != count) {
+ return null;
+ }
+
+ List<BlockHeader> headers;
+ try {
+ byte[] bytes = Utils.hexToBytes(chunk.hex);
+ headers = IntStream.range(0, count).mapToObj(i -> new BlockHeader(bytes, i * HeaderStore.HEADER_LENGTH)).toList();
+ } catch(ProtocolException | IllegalArgumentException e) {
+ return null;
+ }
+
+ if(!headers.getLast().getHash().equals(anchorHash)) {
+ return null;
+ }
+
+ for(int i = count - 2; i >= 0; i--) {
+ if(!headers.get(i + 1).getPrevBlockHash().equals(headers.get(i).getHash())) {
+ return null;
+ }
+ }
+
+ return headers;
+ }
+
+ /**
+ * Whether transactions are being verified against the connected server, which turns on the header sync and the inclusion proofs alike.
+ */
+ static boolean isVerifyingTransactions() {
+ return serverCapability != null && serverCapability.supportsMerkleProofs();
+ }
+
+ /**
+ * Whether the connected server must support transaction verification to be used at all, which is the case for the public server tier on mainnet.
+ */
+ static boolean isVerificationMandatory() {
+ return Config.get().getServerType() == ServerType.PUBLIC_ELECTRUM_SERVER && Network.get() == Network.MAINNET;
+ }
+
public Map<Sha256Hash, BlockTransaction> getTransactions(Wallet wallet, Map<BlockTransactionHash, Transaction> references, Map<Integer, BlockHeader> blockHeaderMap) throws ServerException {
try {
Map<Sha256Hash, BlockTransaction> transactionMap = new HashMap<>();
@@ -2153,10 +2520,6 @@ protected FeeRatesUpdatedEvent call() throws ServerException {
};
}
- private boolean isVerificationMandatory() {
- return Config.get().getServerType() == ServerType.PUBLIC_ELECTRUM_SERVER && Network.get() == Network.MAINNET;
- }
-
private void checkTipStaleness() {
if(subscribe && Network.get() == Network.MAINNET && lastTipReceivedAt > 0 && !staleTipWarned && System.currentTimeMillis() - lastTipReceivedAt > STALE_TIP_WARNING_AGE_MILLIS) {
staleTipWarned = true;
@@ -2308,6 +2671,81 @@ public void walletNodeHistoryChanged(WalletNodeHistoryChangedEvent event) {
}
}
+ /**
+ * Keeps the verified header store level with the chain tip. There is nothing to poll - the tip subscription pushes every new block - so this
+ * service is event driven: it is restarted whenever a tip is announced, cancels itself once a run succeeds, and is cancelled on disconnection so
+ * that a retry cannot open a transport of its own. Its period is therefore only the interval at which a failed run is retried.
+ */
+ public static class HeaderSyncService extends ScheduledService<Void> {
+ public static final int RETRY_PERIOD_SECS = 60;
+
+ //The pair from the event that last restarted this service: the height and the header of one announcement, never of two
+ private volatile ChainTip announcedTip;
+
+ @Override
+ protected Task<Void> createTask() {
+ return new Task<>() {
+ @Override
+ protected Void call() throws Exception {
+ syncAnnouncedHeaders(announcedTip);
+ return null;
+ }
+ };
+ }
+
+ /**
+ * The body of a run, which happens on a background thread: the connection check is therefore the transport level one, since AppServices reads
+ * a JavaFX Service and may only be called from the application thread. Without the check a retry firing after the connection closed would
+ * have getTransport() open a transport of its own, outside the connection lifecycle.
+ */
+ static void syncAnnouncedHeaders(ChainTip tip) throws ServerException {
+ if(!isConnected()) {
+ return;
+ }
+
+ ElectrumServer electrumServer = new ElectrumServer();
+ try {
+ electrumServer.syncHeaders(tip);
+ } catch(UnsupportedMethodException e) {
+ //Without this call the store can never advance, so verification would refuse every new confirmation for the rest of the session
+ if(isVerificationMandatory()) {
+ //Leaving the capability on is what lets the next wallet history thread raise this and rotate the server, which this service cannot do
+ log.warn("Server does not support " + e.getMethod() + ", which is required to verify transactions");
+ } else {
+ log.warn("Server does not support " + e.getMethod() + ", disabling transaction verification for this session");
+ serverCapability.withMerkleProofs(false);
+ }
+ }
+ }
+
+ @Subscribe
+ public void connected(ConnectionEvent event) {
+ if(!isVerifyingTransactions()) {
+ return;
+ }
+
+ announcedTip = new ChainTip(event.getBlockHeight(), event.getBlockHeader());
+ restart();
+ }
+
+ @Subscribe
+ public void newBlock(NewBlockEvent event) {
+ if(!isVerifyingTransactions()) {
+ return;
+ }
+
+ //A header that extends the store, one that leaves a gap, and one that conflicts with it are all handled by the sync itself, which appends
+ //an extending header without fetching anything, so there is nothing to classify here
+ announcedTip = new ChainTip(event.getHeight(), event.getBlockHeader());
+ restart();
+ }
+
+ @Subscribe
+ public void disconnection(DisconnectionEvent event) {
+ cancel();
+ }
+ }
+
public static class ReadRunnable implements Runnable {
@Override
public void run() {
### src/main/java/com/sparrowwallet/sparrow/net/HeaderStore.java
@@ -0,0 +1,307 @@
+package com.sparrowwallet.sparrow.net;
+
+import com.sparrowwallet.drongo.protocol.BlockHeader;
+import com.sparrowwallet.drongo.protocol.HeaderChainState;
+import com.sparrowwallet.drongo.protocol.HeaderCheckpoints;
+import com.sparrowwallet.drongo.protocol.ProtocolException;
+import com.sparrowwallet.drongo.protocol.Sha256Hash;
+import com.sparrowwallet.drongo.protocol.VerificationException;
+import com.sparrowwallet.sparrow.io.Storage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.math.BigInteger;
+import java.util.List;
+
+/**
+ * The block headers above the last pinned checkpoint, held as a flat file of consecutive raw 80 byte records named for the height of its first record,
+ * which is the height following the last pin. The height h is stored at offset (h - startHeight) * 80, and the first record must descend from the pin,
+ * so that the compiled-in checkpoints and the file agree on where each height lies.
+ * <p>
+ * Nothing in the file is trusted: the whole chain is re-verified from the pinned anchor on every load, which costs a few tens of milliseconds per year
+ * of headers. A torn, truncated or tampered file therefore costs a re-download of the headers above the damage rather than admitting an unverified
+ * header. A file that does not descend from the last pin at all - another network's, or one copied from another machine - is discarded.
+ * <p>
+ * A store written against an earlier checkpoint set is superseded rather than read: everything below the new anchor is unreachable and what is above it
+ * is a small re-download, so deleting it keeps the file to exactly what this release's checkpoints verify, from its first record.
+ */
+public class HeaderStore {
+ private static final Logger log = LoggerFactory.getLogger(HeaderStore.class);
+
+ public static final int HEADER_LENGTH = 80;
+
+ private final File file;
+ private final int startHeight; //the height of the record at offset zero, being the header immediately above the last pin
+ private final HeaderCheckpoints checkpoints;
+ private HeaderChainState chainState; //the live state at the store tip, rebuilt from the anchor whenever the file is truncated
+
+ private HeaderStore(File file, HeaderCheckpoints checkpoints) {
+ this.file = file;
+ this.startHeight = checkpoints.getMaxHeight() + 1;
+ this.checkpoints = checkpoints;
+ }
+
+ /**
+ * Loads (or creates) the store for the given checkpoints, fully re-verifying the chain from the pinned anchor.
+ */
+ public static synchronized HeaderStore load(HeaderCheckpoints checkpoints) throws IOException {
+ int startHeight = checkpoints.getMaxHeight() + 1;
+ File headersDir = Storage.getHeadersDir();
+ deleteSupersededStores(headersDir, startHeight);
+
+ HeaderStore store = new HeaderStore(new File(headersDir, Integer.toString(startHeight)), checkpoints);
+ if(!store.isAnchored()) {
+ log.warn("Discarding the block header store at " + store.file.getName() + ": it does not descend from the checkpoint at height " + checkpoints.getMaxHeight());
+ if(!store.file.delete()) {
+ //Only the clearer message is lost: the walk below refuses the first record and empties the file to the same end
+ log.debug("Could not delete the block header store at " + store.file.getAbsolutePath());
+ }
+ }
+
+ store.open();
+
+ return store;
+ }
+
+ /**
+ * Deletes the stores written against earlier checkpoint sets, whose records this one cannot use. A store starting above this anchor is from a later
+ * release and is left alone, so that a downgrade does not cost the upgrade its headers.
+ */
+ private static void deleteSupersededStores(File headersDir, int startHeight) {
+ File[] files = headersDir.listFiles();
+ if(files == null) {
+ return;
+ }
+
+ for(File file : files) {
+ try {
+ int base = Integer.parseInt(file.getName());
+ if(base >= 0 && base < startHeight && !file.delete()) {
+ //Nothing reads a superseded store, so leaving one behind costs the space it holds and nothing else
+ log.warn("Could not delete the superseded block header store at " + file.getAbsolutePath());
+ }
+ } catch(NumberFormatException e) {
+ //Not a header store file
+ }
+ }
+ }
+
+ /**
+ * Verifies and appends a header extending the store tip, writing it only once the chain state has accepted it.
+ */
+ public synchronized void append(BlockHeader header) throws IOException {
+ append(List.of(header));
+ }
+
+ /**
+ * Verifies and appends a run of headers extending the store tip, writing in one pass what the chain state accepted. A run carrying a header that
+ * the chain state rejects appends the headers below it and throws, leaving the store exactly where appending them one at a time would.
+ */
+ public synchronized void append(List<BlockHeader> headers) throws IOException {
+ long offset = getOffset(chainState.getHeight() + 1);
+ ByteArrayOutputStream accepted = new ByteArrayOutputStream(headers.size() * HEADER_LENGTH);
+ VerificationException rejected = null;
+ try {
+ for(BlockHeader header : headers) {
+ chainState.add(header);
+ accepted.writeBytes(header.bitcoinSerialize());
+ }
+ } catch(VerificationException e) {
+ rejected = e;
+ }
+
+ if(accepted.size() > 0) {
+ try(RandomAccessFile randomAccessFile = new RandomAccessFile(file, "rw")) {
+ randomAccessFile.seek(offset);
+ randomAccessFile.write(accepted.toByteArray());
+ } catch(IOException e) {
+ rebuild(); //the chain state and the file must agree on the tip, whatever of the run was written
+ throw e;
+ }
+ }
+
+ if(rejected != null) {
+ throw rejected;
+ }
+ }
+
+ /**
+ * Drops every header above the given height, which becomes the new tip.
+ */
+ public synchronized void truncate(int newTipHeight) throws IOException {
+ if(newTipHeight >= chainState.getHeight()) {
+ return;
+ }
+
+ setLength(Math.max(0, getOffset(newTipHeight + 1)));
+ rebuild();
+ }
+
+ /**
+ * The header at the given height, or null where it is below the anchor or above the tip.
+ */
+ public synchronized BlockHeader getHeader(int height) throws IOException {
+ if(height < startHeight || height > chainState.getHeight()) {
+ return null;
+ }
+
+ byte[] record = readRecord(getOffset(height));
+ return record == null ? null : new BlockHeader(record, 0);
+ }
+
+ /**
+ * The hash at the given height, which at the anchor itself is the pinned hash.
+ */
+ public synchronized Sha256Hash getHash(int height) throws IOException {
+ if(height == startHeight - 1) {
+ return checkpoints.getHash(checkpoints.getMaxHeight());
+ }
+
+ BlockHeader header = getHeader(height);
+ return header == null ? null : header.getHash();
+ }
+
+ /**
+ * Whether the file still holds exactly what the chain state says it does. A file removed or truncated underneath a loaded store would otherwise
+ * have every height read as absent, which surfaces as a server unable to substantiate a chain it served rather than as the local fault it is.
+ */
+ public synchronized boolean isIntact() throws IOException {
+ return file.length() == getOffset(chainState.getHeight() + 1);
+ }
+
+ /**
+ * The lowest height this store serves, being the header immediately above the last pin.
+ */
+ public synchronized int getStartHeight() {
+ return startHeight;
+ }
+
+ /**
+ * The height of the last verified header, which for an empty store is the anchor itself.
+ */
+ public synchronized int getTipHeight() {
+ return chainState.getHeight();
+ }
+
+ public synchronized Sha256Hash getTipHash() {
+ return chainState.getHash();
+ }
+
+ /**
+ * The chain work accumulated above the pinned anchor, measured under the network's own rule. A candidate chain validated on a state re-walked to a
+ * fork point shares that anchor, so the two are directly comparable and the shared work below the fork cancels.
+ */
+ public synchronized BigInteger getChainWork() {
+ return chainState.getChainWork();
+ }
+
+ /**
+ * A throwaway chain state re-walked from the anchor to the given height, on which a reorg candidate can be validated. A rolling state cannot be
+ * rewound, so this walk is what a fork point costs.
+ */
+ public synchronized HeaderChainState chainStateAt(int height) throws IOException {
+ if(height < startHeight - 1 || height > chainState.getHeight()) {
+ throw new IllegalArgumentException("Height " + height + " is outside the store range " + (startHeight - 1) + " to " + chainState.getHeight());
+ }
+
+ HeaderChainState state = walkTo(getOffset(height + 1));
+ if(state.getHeight() != height) {
+ throw new VerificationException("The block header store could not be re-verified to height " + height + ", reaching only height " + state.getHeight());
+ }
+
+ return state;
+ }
+
+ private void open() throws IOException {
+ if(!file.exists()) {
+ if(!file.createNewFile()) {
+ throw new IOException("Could not create the block header store at " + file.getAbsolutePath());
+ }
+
+ chainState = checkpoints.newChainState();
+ return;
+ }
+
+ //A header interrupted mid write costs only itself: the tip append is deliberately not synced, and what is lost is fetched again
+ long records = file.length() / HEADER_LENGTH;
+ if(records * HEADER_LENGTH != file.length()) {
+ setLength(records * HEADER_LENGTH);
+ }
+
+ rebuild();
+ }
+
+ private void rebuild() throws IOException {
+ HeaderChainState state = walkTo(file.length());
+ long verifiedLength = getOffset(state.getHeight() + 1);
+ if(file.length() > verifiedLength) {
+ setLength(verifiedLength);
+ }
+
+ chainState = state;
+ }
+
+ /**
+ * Re-verifies the records below the given offset from the pinned anchor, stopping at the first that does not extend the chain.
+ */
+ private HeaderChainState walkTo(long endOffset) throws IOException {
+ HeaderChainState state = checkpoints.newChainState();
+ if(endOffset < HEADER_LENGTH) {
+ return state;
+ }
+
+ try(DataInputStream inputStream = new DataInputStream(new BufferedInputStream(new FileInputStream(file)))) {
+ byte[] record = new byte[HEADER_LENGTH];
+ for(long offset = 0; offset + HEADER_LENGTH <= endOffset; offset += HEADER_LENGTH) {
+ inputStream.readFully(record);
+ try {
+ state.add(new BlockHeader(record, 0));
+ } catch(VerificationException | ProtocolException e) {
+ log.warn("Dropping the block header store above height " + state.getHeight() + ": " + e.getMessage());
+ break;
+ }
+ }
+ }
+
+ return state;
+ }
+
+ /**
+ * Whether the first record descends from the last pin. A store that holds no records yet cannot contradict it.
+ */
+ private boolean isAnchored() throws IOException {
+ byte[] record = readRecord(0);
+ return record == null || new BlockHeader(record, 0).getPrevBlockHash().equals(checkpoints.getHash(checkpoints.getMaxHeight()));
+ }
+
+ private byte[] readRecord(long offset) throws IOException {
+ if(offset < 0 || file.length() < offset + HEADER_LENGTH) {
+ return null;
+ }
+
+ try(RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r")) {
+ randomAccessFile.seek(offset);
+ byte[] record = new byte[HEADER_LENGTH];
+ randomAccessFile.readFully(record);
+
+ return record;
+ }
+ }
+
+ private void setLength(long length) throws IOException {
+ try(RandomAccessFile randomAccessFile = new RandomAccessFile(file, "rw")) {
+ randomAccessFile.setLength(length);
+ }
+ }
+
+ private long getOffset(int height) {
+ return (long)(height - startHeight) * HEADER_LENGTH;
+ }
+}
### src/main/java/com/sparrowwallet/sparrow/wallet/WalletForm.java
@@ -561,6 +561,20 @@ public void connected(ConnectionEvent event) {
refreshHistory(event.getBlockHeight());
}
+ @Subscribe
+ public void chainReorg(ChainReorgEvent event) {
+ if(wallet.isValid() && !wallet.isNested()) {
+ //Posted on the syncing thread, and invalidating must precede the refresh: a transaction re-included at the same height leaves the server
+ //reporting an unchanged status, and the node would not otherwise be revisited. A wallet holding nothing above the fork is left alone,
+ //since anything new to it still arrives on its script hash subscriptions
+ Platform.runLater(() -> {
+ if(ElectrumServer.invalidateScriptHashesForReorg(wallet, event.getForkHeight())) {
+ refreshHistory(AppServices.getCurrentBlockHeight());
+ }
+ });
+ }
+ }
+
@Subscribe
public void walletNodeHistoryChanged(WalletNodeHistoryChangedEvent event) {
if(wallet.isValid() && !wallet.isNested()) {
### src/test/java/com/sparrowwallet/sparrow/net/ElectrumServerTest.java
@@ -1,15 +1,39 @@
package com.sparrowwallet.sparrow.net;
+import com.sparrowwallet.drongo.ExtendedKey;
+import com.sparrowwallet.drongo.KeyDerivation;
+import com.sparrowwallet.drongo.KeyPurpose;
import com.sparrowwallet.drongo.Network;
import com.sparrowwallet.drongo.Utils;
+import com.sparrowwallet.drongo.policy.Policy;
+import com.sparrowwallet.drongo.policy.PolicyType;
+import com.sparrowwallet.drongo.protocol.ScriptType;
+import com.sparrowwallet.drongo.wallet.BlockTransactionHashIndex;
+import com.sparrowwallet.drongo.wallet.Keystore;
+import com.sparrowwallet.drongo.wallet.Wallet;
+import com.sparrowwallet.drongo.wallet.WalletNode;
+import com.sparrowwallet.drongo.protocol.BlockHeader;
+import com.sparrowwallet.drongo.protocol.HeaderChainState;
+import com.sparrowwallet.drongo.protocol.Sha256Hash;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
public class ElectrumServerTest {
+ //A plain BIP32 extended public key, since the wallet only needs to derive addresses to have script hashes
+ private static final String TEST_XPUB = "xpub6BosfCnifzxcFwrSzQiqu2DBVTshkCXacvNsWGYJVVhhawA7d4R5WSWGFNbi8Aw6ZRc1brxMyWMzG3DSSSSoekkudhUd9yLb6qx39T9nMdj";
+
private static final String GENESIS_HEADER_HEX = "0100000000000000000000000000000000000000000000000000000000000000000000003ba3edfd7a7b12b27ac72c3e67768f617fc81bc3888a51323a9fb8aa4b1e5e4a29ab5f49ffff001d1dac2b7c";
private static final long GENESIS_TIME_SECS = 1231006505L;
@@ -56,6 +80,114 @@ public void rejectsMalformedTips() {
assertNotNull(ElectrumServer.getTipValidationError(tip(800000, "cafebabe"), now));
}
+ /**
+ * A header below a pinned checkpoint is placed at its height by descent from the pin, so the hash chain up to the pinned hash is the whole proof:
+ * no proof of work, difficulty or timestamp check is applied to a range that ends in a hash already known to be on the chain.
+ */
+ @Test
+ public void verifiesHeadersLinkedToTheAnchor() {
+ List<BlockHeader> headers = chain(5);
+ List<BlockHeader> linked = ElectrumServer.getLinkedHeaders(headersChunk(headers), 5, headers.getLast().getHash());
+ assertNotNull(linked);
+ assertEquals(headers.getFirst().getHash(), linked.getFirst().getHash());
+ assertEquals(headers.getLast().getHash(), linked.getLast().getHash());
+ }
+
+ @Test
+ public void rejectsHeadersThatDoNotReachTheAnchor() {
+ List<BlockHeader> headers = chain(5);
+
+ //The range is internally consistent, but ends somewhere other than the pinned hash
+ assertNull(ElectrumServer.getLinkedHeaders(headersChunk(headers), 5, headers.get(3).getHash()));
+ }
+
+ @Test
+ public void rejectsATamperedHeaderWithinTheRange() {
+ List<BlockHeader> headers = new ArrayList<>(chain(5));
+ BlockHeader tampered = headers.get(2);
+ headers.set(2, new BlockHeader(tampered.getVersion(), tampered.getPrevBlockHash(), Sha256Hash.ZERO_HASH, null, tampered.getTime() + 1,
+ tampered.getDifficultyTarget(), tampered.getNonce()));
+
+ //The header above it still carries the original hash, so the chain to the anchor is broken at the substitution
+ assertNull(ElectrumServer.getLinkedHeaders(headersChunk(headers), 5, headers.getLast().getHash()));
+ }
+
+ @Test
+ public void rejectsAShortOrMalformedRange() {
+ List<BlockHeader> headers = chain(5);
+ assertNull(ElectrumServer.getLinkedHeaders(headersChunk(headers.subList(0, 4)), 5, headers.getLast().getHash()));
+
+ BlockHeaders malformed = headersChunk(headers);
+ malformed.hex = "cafebabe";
+ assertNull(ElectrumServer.getLinkedHeaders(malformed, 5, headers.getLast().getHash()));
+ }
+
+ private static List<BlockHeader> chain(int count) {
+ List<BlockHeader> headers = new ArrayList<>();
+ Sha256Hash previousHash = Sha256Hash.ZERO_HASH;
+ for(int i = 0; i < count; i++) {
+ BlockHeader header = new BlockHeader(1, previousHash, Sha256Hash.ZERO_HASH, null, 1600000000L + i, 0x1d00ffffL, i);
+ headers.add(header);
+ previousHash = header.getHash();
+ }
+
+ return headers;
+ }
+
+ private static BlockHeaders headersChunk(List<BlockHeader> headers) {
+ BlockHeaders blockHeaders = new BlockHeaders();
+ blockHeaders.count = headers.size();
+ blockHeaders.hex = headers.stream().map(header -> Utils.bytesToHex(header.bitcoinSerialize())).collect(Collectors.joining());
+ blockHeaders.max = HeaderChainState.RETARGET_INTERVAL;
+
+ return blockHeaders;
+ }
+
+ /**
+ * A reorg invalidates only the nodes holding something above the fork point. A wallet with nothing there has proven nothing against a header that
+ * was discarded, so it reports no invalidation and its handler leaves it alone rather than joining a refresh of every open wallet.
+ */
+ @Test
+ public void invalidatesOnlyTheNodesHoldingSomethingAboveTheFork() {
+ Wallet wallet = testWallet();
+ WalletNode receiveNode = wallet.getNode(KeyPurpose.RECEIVE).getChildren().iterator().next();
+ receiveNode.getTransactionOutputs().add(new BlockTransactionHashIndex(Sha256Hash.ZERO_HASH, 800000, new Date(), 0L, 0, 10000));
+
+ assertFalse(ElectrumServer.invalidateScriptHashesForReorg(wallet, 800000));
+ assertFalse(ElectrumServer.invalidateScriptHashesForReorg(wallet, 900000));
+ assertTrue(ElectrumServer.invalidateScriptHashesForReorg(wallet, 799999));
+ }
+
+ /**
+ * A spend confirmed in the orphaned block is the same case as an output received in it: the node holding the output it spent has to be refetched.
+ */
+ @Test
+ public void invalidatesANodeWhoseOutputWasSpentAboveTheFork() {
+ Wallet wallet = testWallet();
+ WalletNode receiveNode = wallet.getNode(KeyPurpose.RECEIVE).getChildren().iterator().next();
+ BlockTransactionHashIndex output = new BlockTransactionHashIndex(Sha256Hash.ZERO_HASH, 700000, new Date(), 0L, 0, 10000);
+ output.setSpentBy(new BlockTransactionHashIndex(Sha256Hash.ZERO_HASH, 800000, new Date(), 0L, 0, 10000));
+ receiveNode.getTransactionOutputs().add(output);
+
+ //The output itself is far below the fork, but the spend of it is not
+ assertTrue(ElectrumServer.invalidateScriptHashesForReorg(wallet, 799999));
+ assertFalse(ElectrumServer.invalidateScriptHashesForReorg(wallet, 800000));
+ }
+
+ private static Wallet testWallet() {
+ Wallet wallet = new Wallet();
+ wallet.setPolicyType(PolicyType.SINGLE_HD);
+ wallet.setScriptType(ScriptType.P2WPKH);
+ Keystore keystore = new Keystore();
+ keystore.setKeyDerivation(new KeyDerivation("00000000", "m/84'/0'/0'"));
+ keystore.setExtendedPublicKey(ExtendedKey.fromDescriptor(TEST_XPUB));
+ wallet.getKeystores().add(keystore);
+ wallet.setDefaultPolicy(Policy.getPolicy(PolicyType.SINGLE_HD, ScriptType.P2WPKH, wallet.getKeystores(), 1));
+ wallet.getNode(KeyPurpose.RECEIVE).fillToIndex(wallet, 1);
+
+ return wallet;
+ }
+
private BlockHeaderTip tip(int height, String hex) {
BlockHeaderTip tip = new BlockHeaderTip();
tip.height = height;
@@ -65,6 +197,8 @@ private BlockHeaderTip tip(int height, String hex) {
@AfterEach
public void tearDown() throws Exception {
+ //The reorg tests above invalidate script hashes, which is the one piece of static state they leave behind
+ ElectrumServer.reorgInvalidatedScriptHashes.clear();
Network.set(null);
}
}
### src/test/java/com/sparrowwallet/sparrow/net/HeaderStoreTest.java
@@ -0,0 +1,375 @@
+package com.sparrowwallet.sparrow.net;
+
+import com.sparrowwallet.drongo.Network;
+import com.sparrowwallet.drongo.protocol.BlockHeader;
+import com.sparrowwallet.drongo.protocol.HeaderCheckpoints;
+import com.sparrowwallet.drongo.protocol.Sha256Hash;
+import com.sparrowwallet.drongo.protocol.VerificationException;
+import com.sparrowwallet.sparrow.SparrowWallet;
+import com.sparrowwallet.sparrow.io.Storage;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.math.BigInteger;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * The header store on regtest, whose trivial proof of work target is the only one a synthetic chain can be mined against, and whose empty checkpoints
+ * anchor the store at the genesis header. Every load re-verifies the file from that anchor, so these tests are as much about what a damaged file
+ * costs - the headers above the damage, never an unverified header - as about the round trip.
+ */
+public class HeaderStoreTest {
+ @TempDir
+ private static Path tempHome;
+
+ @BeforeAll
+ public static void setUpAll() {
+ //Config.get() caches its instance statically for the life of the JVM, so keep this test from loading the developer's real config
+ System.setProperty(SparrowWallet.APP_HOME_PROPERTY, tempHome.toString());
+ }
+
+ @AfterAll
+ public static void tearDownAll() {
+ System.clearProperty(SparrowWallet.APP_HOME_PROPERTY);
+ }
+
+ @BeforeEach
+ public void setUp() {
+ Network.set(Network.REGTEST);
+ File[] files = Storage.getHeadersDir().listFiles();
+ if(files != null) {
+ for(File file : files) {
+ assertTrue(file.delete());
+ }
+ }
+ }
+
+ @AfterEach
+ public void tearDown() {
+ Network.set(null);
+ }
+
+ @Test
+ public void storesAndReloadsAChain() throws IOException {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 20);
+ HeaderStore store = HeaderStore.load(checkpoints());
+
+ //An empty store is the anchor itself, and serves no heights at all
+ assertEquals(0, store.getTipHeight());
+ assertEquals(Network.REGTEST.getGenesisHash(), store.getTipHash());
+ assertEquals(1, store.getStartHeight());
+ assertNull(store.getHeader(0));
+ assertNull(store.getHeader(1));
+
+ for(BlockHeader header : chain) {
+ store.append(header);
+ }
+
+ assertEquals(20, store.getTipHeight());
+ assertEquals(chain.getLast().getHash(), store.getTipHash());
+ assertEquals(20 * HeaderStore.HEADER_LENGTH, storeFile("1").length());
+
+ HeaderStore reloaded = HeaderStore.load(checkpoints());
+ assertEquals(20, reloaded.getTipHeight());
+ for(int height = 1; height <= 20; height++) {
+ assertEquals(chain.get(height - 1).getHash(), reloaded.getHeader(height).getHash());
+ assertEquals(chain.get(height - 1).getHash(), reloaded.getHash(height));
+ }
+
+ //The anchor is the pinned hash rather than a stored record, and heights below it are not served
+ assertEquals(Network.REGTEST.getGenesisHash(), reloaded.getHash(0));
+ assertNull(reloaded.getHeader(0));
+ assertNull(reloaded.getHeader(21));
+ }
+
+ @Test
+ public void refusesAHeaderThatDoesNotExtendTheChain() throws IOException {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 5);
+ HeaderStore store = HeaderStore.load(checkpoints());
+ store.append(chain.getFirst());
+
+ //A header that does not link is refused before anything is written, so the store is left exactly as it was
+ assertThrows(VerificationException.class, () -> store.append(chain.getLast()));
+ assertEquals(1, store.getTipHeight());
+ assertEquals(HeaderStore.HEADER_LENGTH, storeFile("1").length());
+ }
+
+ /**
+ * A run carrying a header the chain state rejects appends the headers below it and stops there, which is what appending them one at a time did:
+ * the store keeps the progress it verified, and the run is refused from the header that failed.
+ */
+ @Test
+ public void appendsTheVerifiedPrefixOfARejectedRun() throws IOException {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 2, 1600000000L);
+ BlockHeader belowTarget = unmineHeader(chain.getLast(), 1600000002L);
+ List<BlockHeader> run = new ArrayList<>(chain);
+ run.add(belowTarget);
+ run.addAll(mineChain(belowTarget, 2, 1600000003L));
+
+ HeaderStore store = HeaderStore.load(checkpoints());
+ VerificationException e = assertThrows(VerificationException.class, () -> store.append(run));
+ assertTrue(e.getMessage().contains("proof of work"), e.getMessage());
+
+ assertEquals(2, store.getTipHeight());
+ assertEquals(chain.getLast().getHash(), store.getTipHash());
+ assertEquals(2 * HeaderStore.HEADER_LENGTH, storeFile("1").length());
+ assertEquals(2, HeaderStore.load(checkpoints()).getTipHeight());
+ }
+
+ @Test
+ public void repairsATornWrite() throws IOException {
+ appendChain(mineChain(Network.REGTEST.getGenesisHeader(), 10));
+
+ //The tip append is deliberately not synced, so a process killed mid write leaves a partial record
+ try(RandomAccessFile randomAccessFile = new RandomAccessFile(storeFile("1"), "rw")) {
+ randomAccessFile.seek(randomAccessFile.length());
+ randomAccessFile.write(new byte[HeaderStore.HEADER_LENGTH / 2]);
+ }
+
+ HeaderStore store = HeaderStore.load(checkpoints());
+ assertEquals(10, store.getTipHeight());
+ assertEquals(10 * HeaderStore.HEADER_LENGTH, storeFile("1").length());
+ }
+
+ @Test
+ public void dropsTheHeadersAboveACorruptedRecord() throws IOException {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 10);
+ appendChain(chain);
+
+ //Record 9 is the header at height 10, which no longer links to the header below it
+ try(RandomAccessFile randomAccessFile = new RandomAccessFile(storeFile("1"), "rw")) {
+ long position = 9 * HeaderStore.HEADER_LENGTH + 4;
+ randomAccessFile.seek(position);
+ int previousHashByte = randomAccessFile.read();
+ randomAccessFile.seek(position);
+ randomAccessFile.write(previousHashByte ^ 0xff);
+ }
+
+ HeaderStore store = HeaderStore.load(checkpoints());
+ assertEquals(9, store.getTipHeight());
+ assertEquals(chain.get(8).getHash(), store.getTipHash());
+ assertEquals(9 * HeaderStore.HEADER_LENGTH, storeFile("1").length());
+
+ //And the store carries on from there
+ store.append(chain.get(9));
+ assertEquals(10, store.getTipHeight());
+ }
+
+ /**
+ * A store written against an earlier checkpoint set cannot be read at this one's offsets, and holds nothing above the new anchor that is not a
+ * small re-download, so it is deleted rather than carried forward. Regtest anchors at genesis, so a file starting at height 0 is that shape.
+ */
+ @Test
+ public void supersedesAStoreFromAnEarlierCheckpointSet() throws IOException {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 10);
+ List<BlockHeader> records = new ArrayList<>();
+ records.add(Network.REGTEST.getGenesisHeader());
+ records.addAll(chain);
+ writeStoreFile("0", records);
+ writeStoreFile("1", chain);
+
+ HeaderStore store = HeaderStore.load(checkpoints());
+ assertFalse(storeFile("0").exists());
+
+ //The store for this anchor is the one that is read, and it starts at its own first record
+ assertEquals(10, store.getTipHeight());
+ assertEquals(1, store.getStartHeight());
+ assertNull(store.getHeader(0));
+ assertEquals(chain.getFirst().getHash(), store.getHeader(1).getHash());
+ assertEquals(chain.getLast().getHash(), store.getHeader(10).getHash());
+
+ BlockHeader next = mineChain(chain.getLast(), 1).getFirst();
+ store.append(next);
+ assertEquals(11, store.getTipHeight());
+ assertEquals(11 * HeaderStore.HEADER_LENGTH, storeFile("1").length());
+ assertEquals(next.getHash(), HeaderStore.load(checkpoints()).getHeader(11).getHash());
+ }
+
+ /**
+ * A superseded store is deleted even where this checkpoint set has no store of its own yet, so the space it holds is not carried indefinitely.
+ */
+ @Test
+ public void deletesASupersededStoreWithNothingToReplaceIt() throws IOException {
+ writeStoreFile("0", mineChain(Network.REGTEST.getGenesisHeader(), 4));
+
+ HeaderStore store = HeaderStore.load(checkpoints());
+ assertFalse(storeFile("0").exists());
+ assertTrue(storeFile("1").exists());
+ assertEquals(0, store.getTipHeight());
+ assertEquals(Network.REGTEST.getGenesisHash(), store.getTipHash());
+ assertEquals(0, storeFile("1").length());
+ }
+
+ @Test
+ public void ignoresAStoreStartingAboveTheAnchor() throws IOException {
+ //A file written by a later release, whose checkpoints reach higher than these: it cannot be read at these offsets, so a fresh one is started
+ writeStoreFile("500", mineChain(Network.REGTEST.getGenesisHeader(), 3));
+
+ HeaderStore store = HeaderStore.load(checkpoints());
+ assertEquals(0, store.getTipHeight());
+ assertTrue(storeFile("500").exists());
+ assertTrue(storeFile("1").exists());
+ }
+
+ @Test
+ public void discardsAStoreThatDoesNotDescendFromTheAnchor() throws IOException {
+ //Another network's headers, or a file copied from another machine: the record above the pin does not descend from it
+ BlockHeader foreign = new BlockHeader(1, Sha256Hash.wrap("00000000000000000000000000000000000000000000000000000000deadbeef"),
+ Sha256Hash.ZERO_HASH, null, 1600000000L, 0x207fffffL, 0);
+ writeStoreFile("1", List.of(foreign));
+
+ HeaderStore store = HeaderStore.load(checkpoints());
+ assertEquals(0, store.getTipHeight());
+ assertEquals(0, storeFile("1").length());
+ }
+
+ /**
+ * Removing a store that cannot be read is a cleanup, not a precondition: where the file cannot be deleted the load must still produce a working
+ * store rather than failing the session, since a failure here reaches the wallet as a server error on every height for as long as it lasts.
+ */
+ @Test
+ public void loadsAStoreThatCannotBeDeleted() throws IOException {
+ BlockHeader foreign = new BlockHeader(1, Sha256Hash.wrap("00000000000000000000000000000000000000000000000000000000deadbeef"),
+ Sha256Hash.ZERO_HASH, null, 1600000000L, 0x207fffffL, 0);
+ writeStoreFile("1", List.of(foreign));
+
+ //Where the directory denies removal the file is emptied in place instead, which is the same end state
+ assertTrue(Storage.getHeadersDir().setWritable(false));
+ try {
+ HeaderStore store = HeaderStore.load(checkpoints());
+ assertEquals(0, store.getTipHeight());
+ assertEquals(Network.REGTEST.getGenesisHash(), store.getTipHash());
+ assertEquals(0, storeFile("1").length());
+
+ BlockHeader header = mineChain(Network.REGTEST.getGenesisHeader(), 1).getFirst();
+ store.append(header);
+ assertEquals(1, store.getTipHeight());
+ assertEquals(header.getHash(), store.getTipHash());
+ } finally {
+ assertTrue(Storage.getHeadersDir().setWritable(true));
+ }
+ }
+
+ @Test
+ public void reorganisesToAChainOfAtLeastTheSameWork() throws IOException {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 10);
+ HeaderStore store = HeaderStore.load(checkpoints());
+ for(BlockHeader header : chain) {
+ store.append(header);
+ }
+
+ //Chain work on the test networks is the header count, so five headers above the fork point is what the branch has to beat
+ assertEquals(BigInteger.valueOf(10), store.getChainWork());
+ assertEquals(BigInteger.valueOf(5), store.chainStateAt(5).getChainWork());
+ assertEquals(5, store.chainStateAt(5).getHeight());
+ assertEquals(chain.get(4).getHash(), store.chainStateAt(5).getHash());
+
+ //A branch from height 5 that is one header longer, mined at a later time so that it is a different chain
+ List<BlockHeader> branch = mineChain(chain.get(4), 6, 1700000000L);
+ store.truncate(5);
+ assertEquals(5, store.getTipHeight());
+ assertEquals(chain.get(4).getHash(), store.getTipHash());
+ assertNull(store.getHeader(6));
+ for(BlockHeader header : branch) {
+ store.append(header);
+ }
+
+ assertEquals(11, store.getTipHeight());
+ assertEquals(BigInteger.valueOf(11), store.getChainWork());
+ assertEquals(branch.getLast().getHash(), store.getTipHash());
+
+ HeaderStore reloaded = HeaderStore.load(checkpoints());
+ assertEquals(11, reloaded.getTipHeight());
+ assertEquals(branch.getFirst().getHash(), reloaded.getHeader(6).getHash());
+ assertEquals(11 * HeaderStore.HEADER_LENGTH, storeFile("1").length());
+ }
+
+ @Test
+ public void truncatesToTheAnchorItself() throws IOException {
+ appendChain(mineChain(Network.REGTEST.getGenesisHeader(), 4));
+
+ HeaderStore store = HeaderStore.load(checkpoints());
+ store.truncate(0);
+ assertEquals(0, store.getTipHeight());
+ assertEquals(Network.REGTEST.getGenesisHash(), store.getTipHash());
+ assertEquals(0, storeFile("1").length());
+ assertThrows(IllegalArgumentException.class, () -> store.chainStateAt(1));
+ }
+
+ private static HeaderCheckpoints checkpoints() {
+ HeaderCheckpoints checkpoints = Network.REGTEST.getHeaderCheckpoints();
+ assertEquals(0, checkpoints.getMaxHeight());
+
+ return checkpoints;
+ }
+
+ private static File storeFile(String name) {
+ return new File(Storage.getHeadersDir(), name);
+ }
+
+ private static void appendChain(List<BlockHeader> chain) throws IOException {
+ HeaderStore store = HeaderStore.load(checkpoints());
+ for(BlockHeader header : chain) {
+ store.append(header);
+ }
+ }
+
+ private static void writeStoreFile(String name, List<BlockHeader> headers) throws IOException {
+ try(RandomAccessFile randomAccessFile = new RandomAccessFile(new File(Storage.getHeadersDir(), name), "rw")) {
+ for(BlockHeader header : headers) {
+ randomAccessFile.write(header.bitcoinSerialize());
+ }
+ }
+ }
+
+ private static List<BlockHeader> mineChain(BlockHeader previous, int count) {
+ return mineChain(previous, count, 1600000000L);
+ }
+
+ private static List<BlockHeader> mineChain(BlockHeader previous, int count, long startTime) {
+ List<BlockHeader> chain = new ArrayList<>();
+ for(int i = 0; i < count; i++) {
+ previous = mineHeader(previous, startTime + i);
+ chain.add(previous);
+ }
+
+ return chain;
+ }
+
+ private static BlockHeader unmineHeader(BlockHeader previous, long time) {
+ for(long nonce = 0; nonce < 1000; nonce++) {
+ BlockHeader header = new BlockHeader(1, previous.getHash(), Sha256Hash.ZERO_HASH, null, time, 0x207fffffL, nonce);
+ if(!header.verifyProofOfWork()) {
+ return header;
+ }
+ }
+
+ throw new IllegalStateException("Could not produce a regtest header below its target at time " + time);
+ }
+
+ private static BlockHeader mineHeader(BlockHeader previous, long time) {
+ for(long nonce = 0; nonce < 1000; nonce++) {
+ BlockHeader header = new BlockHeader(1, previous.getHash(), Sha256Hash.ZERO_HASH, null, time, 0x207fffffL, nonce);
+ if(header.verifyProofOfWork()) {
+ return header;
+ }
+ }
+
+ throw new IllegalStateException("Could not mine a regtest header at time " + time);
+ }
+}
### src/test/java/com/sparrowwallet/sparrow/net/HeaderSyncTest.java
@@ -0,0 +1,827 @@
+package com.sparrowwallet.sparrow.net;
+
+import com.github.arteam.simplejsonrpc.client.Transport;
+import com.google.common.net.HostAndPort;
+import com.sparrowwallet.drongo.Network;
+import com.sparrowwallet.drongo.Utils;
+import com.sparrowwallet.drongo.protocol.BlockHeader;
+import com.sparrowwallet.drongo.protocol.HeaderChainState;
+import com.sparrowwallet.drongo.protocol.Sha256Hash;
+import com.sparrowwallet.drongo.protocol.VerificationException;
+import com.sparrowwallet.sparrow.AppServices;
+import com.sparrowwallet.sparrow.ChainTip;
+import com.sparrowwallet.sparrow.EventManager;
+import com.sparrowwallet.sparrow.SparrowWallet;
+import com.sparrowwallet.sparrow.event.ChainReorgEvent;
+import com.sparrowwallet.sparrow.io.Storage;
+import com.google.common.eventbus.Subscribe;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CyclicBarrier;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * The header sync orchestration against a server that answers from a chain of the test's choosing: which divergences are detected and where, which
+ * candidates are adopted, and how often the server is asked for anything. The failure these cover is not a lost header but a false accusation - a
+ * store left on an orphaned header reconstructs no merkle branch, and every proof against it is reported as a dishonest server.
+ * <p>
+ * Regtest is the only network whose proof of work target a synthetic chain can be mined against, and its empty checkpoints anchor the store at the
+ * genesis header, so every height here is one the store itself serves.
+ */
+public class HeaderSyncTest {
+ //Mainnet heights 32248 to 32255, the close of difficulty period 15, whose last header is the compiled in pin at height 32255
+ private static final List<BlockHeader> PERIOD_15_CLOSE = Stream.of(
+ "0100000062f481d3ac76c0464800c12b32a724aa05b85aefb735f104407c7643000000004b7a6e15f0331c1619d739d5ddbfedc10208b09c7e44543a7f8450a6cb13049d57e03a4bffff001deada8502",
+ "01000000de07b0f62ba82662b23f920a8429f019731043affa17819e4b23dcd4000000009bed3868aee66f1babd0aced30ea3c4272265e9032758b65b6f9ff50960ff384b6e03a4bffff001da8beb001",
+ "01000000bfeeabd547eb6fc8fa4152725030d8ca0e03ce03f912700601fa3ad2000000000cfc1c060eea439947fb50b9f082ef626f66608cd32b2c93604e8c05caabda96f1e13a4bffff001d5a661f02",
+ "0100000019179cf0fe8ef7d751e007c11dd8097d77d52d126f1afea9c7f83129000000003c5e7fd03948b4e413c69bcbf6208d407910d9d3e0bdde65c572d46e14b526f146e23a4bffff001d9e369a01",
+ "01000000a90a01b124aa4ab2a31923511e138939a4053a5f3f2be7186e820d3600000000fb6e3ed118edc6171a2e50a8484ba7c33fe6bd6efb49e90b8b7d0b0ea5e800b862e23a4bffff001d6db69d00",
+ "010000002d5d7f4a4dad16f92e4b59d1903f34cc6acf40254f64718f32b25e3d00000000c2a630dbfefd5a55a939d39388d1daad1ff2ca22de59ddfd1464b494e61acb0410e93a4bffff001de3c38c1e",
+ "01000000672ae405fdb9e4f2a37ffa660a328e138fa08a9ac7ae99382aee270e00000000342fedae2d72975552ac0797556817d11006ec322e2bf97ce88f91caf39525794fe93a4bffff001d94282401",
+ "0100000049c1daab3b6536ff1b2633c3a316a6e06ec287676cdeec4ca7baae6b00000000ac10b36b8f354b3353207de15940a5edbc05bb8364af75b4b5409e7823f2b48923ec3a4bffff001dbd5fa412")
+ .map(hex -> new BlockHeader(Utils.hexToBytes(hex))).toList();
+
+ private static final long CHAIN_TIME = 1600000000L;
+ private static final long BRANCH_TIME = 1700000000L;
+
+ @TempDir
+ private static Path tempHome;
+
+ private ElectrumServerRpc previousElectrumServerRpc;
+ private CloseableTransport previousTransport;
+ private ChainReorgListener listener;
+
+ @BeforeAll
+ public static void setUpAll() {
+ //Config.get() caches its instance statically for the life of the JVM, so keep this test from loading the developer's real config
+ System.setProperty(SparrowWallet.APP_HOME_PROPERTY, tempHome.toString());
+ }
+
+ @AfterAll
+ public static void tearDownAll() {
+ System.clearProperty(SparrowWallet.APP_HOME_PROPERTY);
+ }
+
+ @BeforeEach
+ public void setUp() {
+ Network.set(Network.REGTEST);
+ File[] files = Storage.getHeadersDir().listFiles();
+ if(files != null) {
+ for(File file : files) {
+ assertTrue(file.delete());
+ }
+ }
+
+ ElectrumServer.headerStore = null;
+ ElectrumServer.lastReorgForkHeight = Integer.MAX_VALUE;
+ ElectrumServer.verifiedHistoricalHeaders.clear();
+ previousElectrumServerRpc = ElectrumServer.electrumServerRpc;
+ previousTransport = ElectrumServer.transport;
+ //The fake answers without the transport, but getTransport() would otherwise build one from the configured server
+ ElectrumServer.transport = new UnusedTransport();
+ listener = new ChainReorgListener();
+ EventManager.get().register(listener);
+ }
+
+ @AfterEach
+ public void tearDown() {
+ EventManager.get().unregister(listener);
+ ElectrumServer.electrumServerRpc = previousElectrumServerRpc;
+ ElectrumServer.transport = previousTransport;
+ ElectrumServer.headerStore = null;
+ ElectrumServer.lastReorgForkHeight = Integer.MAX_VALUE;
+ ElectrumServer.verifiedHistoricalHeaders.clear();
+ AppServices.setAnnouncedTip(null);
+ Network.set(null);
+ }
+
+ /**
+ * The tie: a stale block replaced at the store tip height. Equal work is accepted, because a client with one server can only verify against the
+ * chain that server serves - and because the loop that fetches headers would never look at a tip that is not above its own.
+ */
+ @Test
+ public void adoptsAReplacementTipOfEqualWork() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 10, CHAIN_TIME);
+ List<BlockHeader> branch = fork(chain, 9, 1);
+ HeaderStore store = seedStore(chain, 10);
+ FakeElectrumServerRpc fake = serve(branch);
+
+ new ElectrumServer().syncHeaders(new ChainTip(10, branch.getLast()));
+
+ assertEquals(10, store.getTipHeight());
+ assertEquals(branch.getLast().getHash(), store.getTipHash());
+ assertEquals(9, ElectrumServer.lastReorgForkHeight);
+ assertEquals(9, listener.getForkHeight());
+ assertEquals(1, fake.getChunkRequests());
+ }
+
+ @Test
+ public void refusesACandidateCarryingLessWork() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 10, CHAIN_TIME);
+ List<BlockHeader> branch = fork(chain, 5, 4);
+ HeaderStore store = seedStore(chain, 10);
+ serve(branch);
+
+ VerificationException e = assertThrows(VerificationException.class, () -> new ElectrumServer().syncHeaders(new ChainTip(9, branch.getLast())));
+ assertTrue(e.getMessage().contains("less work"), e.getMessage());
+
+ //The store keeps the heavier chain it already has, and nothing is recorded as reorganised
+ assertEquals(10, store.getTipHeight());
+ assertEquals(chain.getLast().getHash(), store.getTipHash());
+ assertEquals(chain.get(5).getHash(), store.getHeader(6).getHash());
+ assertEquals(Integer.MAX_VALUE, ElectrumServer.lastReorgForkHeight);
+ assertNull(listener.getForkHeight());
+ }
+
+ /**
+ * A divergence deeper than the reorg window cannot be linked to anything the store holds, and a chain that deep is a global event rather than a
+ * client concern: the store keeps what it has and the heights in dispute are refused until the server catches up.
+ */
+ @Test
+ public void refusesACandidateWithNoForkPointWithinTheReorgWindow() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 150, CHAIN_TIME);
+ List<BlockHeader> branch = fork(chain, 20, 130);
+ HeaderStore store = seedStore(chain, 150);
+ serve(branch);
+
+ VerificationException e = assertThrows(VerificationException.class, () -> new ElectrumServer().syncHeaders(new ChainTip(150, branch.getLast())));
+ assertTrue(e.getMessage().contains("shares no fork point"), e.getMessage());
+
+ assertEquals(150, store.getTipHeight());
+ assertEquals(chain.getLast().getHash(), store.getTipHash());
+ assertEquals(Integer.MAX_VALUE, ElectrumServer.lastReorgForkHeight);
+ assertNull(listener.getForkHeight());
+ }
+
+ /**
+ * A store tip that is off the server's chain while the server has moved far ahead of it. The fork point is at or below the store tip whatever the
+ * server has announced, so the search window has to sit at the store tip: anchoring it at the announced tip searches a range the store holds no
+ * height in, and the sync then refuses every height for good, with the file needing deletion by hand.
+ */
+ @Test
+ public void reconcilesAtTheStoreTipWhenTheServerIsFarAhead() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 10, CHAIN_TIME);
+ List<BlockHeader> branch = fork(chain, 9, 151);
+ HeaderStore store = seedStore(chain, 10);
+ FakeElectrumServerRpc fake = serve(branch);
+
+ new ElectrumServer().syncHeaders(new ChainTip(160, branch.getLast()));
+
+ assertEquals(160, store.getTipHeight());
+ assertEquals(branch.getLast().getHash(), store.getTipHash());
+ assertEquals(branch.get(9).getHash(), store.getHeader(10).getHash());
+ assertEquals(9, ElectrumServer.lastReorgForkHeight);
+ assertEquals(9, listener.getForkHeight());
+ //The chunk that found the divergence, the window that found the fork, and the forward chunk from the reconciled tip
+ assertEquals(3, fake.getChunkRequests());
+ }
+
+ /**
+ * The fork walk is what identifies the point to rewind to, so a window that does not chain to itself cannot be mined for one: the walk stops at the
+ * break rather than continuing past it to whatever lies below.
+ */
+ @Test
+ public void refusesAWindowThatDoesNotChainToItself() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 10, CHAIN_TIME);
+ List<BlockHeader> candidate = new ArrayList<>(chain.subList(0, 8));
+ BlockHeader unlinked = mineHeader(mineHeader(Network.REGTEST.getGenesisHeader(), BRANCH_TIME), BRANCH_TIME);
+ candidate.add(unlinked);
+ candidate.addAll(mineChain(unlinked, 1, BRANCH_TIME + 1));
+ HeaderStore store = seedStore(chain, 10);
+ serve(candidate);
+
+ VerificationException e = assertThrows(VerificationException.class, () -> new ElectrumServer().syncHeaders(new ChainTip(10, candidate.getLast())));
+ assertTrue(e.getMessage().contains("shares no fork point"), e.getMessage());
+
+ assertEquals(10, store.getTipHeight());
+ assertEquals(chain.getLast().getHash(), store.getTipHash());
+ assertNull(listener.getForkHeight());
+ }
+
+ /**
+ * Finding a fork point is not enough to be adopted: every header above it is put through the chain state, so a segment carrying one that does not
+ * meet its target is refused whole and the store keeps what it has.
+ */
+ @Test
+ public void refusesASegmentCarryingAnInvalidHeader() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 10, CHAIN_TIME);
+ List<BlockHeader> candidate = new ArrayList<>(chain.subList(0, 5));
+ candidate.addAll(mineChain(chain.get(4), 1, BRANCH_TIME));
+ BlockHeader belowTarget = unmineHeader(candidate.getLast(), BRANCH_TIME + 1);
+ candidate.add(belowTarget);
+ candidate.addAll(mineChain(belowTarget, 3, BRANCH_TIME + 2));
+ HeaderStore store = seedStore(chain, 10);
+ serve(candidate);
+
+ VerificationException e = assertThrows(VerificationException.class, () -> new ElectrumServer().syncHeaders(new ChainTip(10, candidate.getLast())));
+ assertTrue(e.getMessage().contains("proof of work"), e.getMessage());
+
+ assertEquals(10, store.getTipHeight());
+ assertEquals(chain.getLast().getHash(), store.getTipHash());
+ assertEquals(chain.get(5).getHash(), store.getHeader(6).getHash());
+ assertEquals(Integer.MAX_VALUE, ElectrumServer.lastReorgForkHeight);
+ assertNull(listener.getForkHeight());
+ }
+
+ /**
+ * The extension: a server on a fork announcing a tip above the store. The divergence is found by the prev hash check on the first header of the
+ * fetched chunk, on the thread that needed the height - a wallet history thread here, not the sync service - and the reorg runs there.
+ */
+ @Test
+ public void detectsAForkFromTheFirstHeaderOfAFetchedChunk() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 10, CHAIN_TIME);
+ List<BlockHeader> branch = fork(chain, 5, 7);
+ HeaderStore store = seedStore(chain, 10);
+ serve(branch);
+ AppServices.setAnnouncedTip(new ChainTip(12, branch.getLast()));
+
+ BlockHeader verified = new ElectrumServer().getVerifiedHeader(11);
+
+ assertEquals(branch.get(10).getHash(), verified.getHash());
+ assertEquals(11, store.getTipHeight());
+ assertEquals(branch.get(5).getHash(), store.getHeader(6).getHash());
+ assertEquals(5, ElectrumServer.lastReorgForkHeight);
+ assertEquals(5, listener.getForkHeight());
+ }
+
+ /**
+ * The same far ahead divergence reached the other way, by a wallet history thread asking for a height the sync service has not reached. The height
+ * it needs is no more related to where the fork lies than the announced tip is, so the window has to be anchored at the store tip here too.
+ */
+ @Test
+ public void reconcilesAtTheStoreTipWhenAHistoryThreadNeedsAFarHeight() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 10, CHAIN_TIME);
+ List<BlockHeader> branch = fork(chain, 9, 151);
+ HeaderStore store = seedStore(chain, 10);
+ serve(branch);
+ AppServices.setAnnouncedTip(new ChainTip(160, branch.getLast()));
+
+ BlockHeader verified = new ElectrumServer().getVerifiedHeader(150);
+
+ assertEquals(branch.get(149).getHash(), verified.getHash());
+ assertEquals(160, store.getTipHeight());
+ assertEquals(branch.get(9).getHash(), store.getHeader(10).getHash());
+ assertEquals(9, ElectrumServer.lastReorgForkHeight);
+ assertEquals(9, listener.getForkHeight());
+ }
+
+ /**
+ * The edge of the reorg window, where the work comparison is at its limit: a fork exactly MAX_REORG_DEPTH - 1 below the store tip is found and the
+ * candidate that replaces those headers is adopted, while one header deeper is out of reach and refused with the store untouched.
+ */
+ @Test
+ public void reorganisesAtTheDeepestReachableForkAndRefusesOneDeeper() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 200, CHAIN_TIME);
+
+ //The store holds 99 headers above the fork, and the candidate 100: the window starts at the header immediately above the fork
+ List<BlockHeader> reachable = fork(chain, 101, 100);
+ HeaderStore store = seedStore(chain, 200);
+ serve(reachable);
+ new ElectrumServer().syncHeaders(new ChainTip(201, reachable.getLast()));
+
+ assertEquals(201, store.getTipHeight());
+ assertEquals(reachable.getLast().getHash(), store.getTipHash());
+ assertEquals(reachable.get(101).getHash(), store.getHeader(102).getHash());
+ assertEquals(101, ElectrumServer.lastReorgForkHeight);
+ assertEquals(101, listener.getForkHeight());
+
+ //One header deeper the fork falls outside the window, and nothing of it can be linked to what the store holds
+ ElectrumServer.headerStore = null;
+ ElectrumServer.lastReorgForkHeight = Integer.MAX_VALUE;
+ for(File file : Storage.getHeadersDir().listFiles()) {
+ assertTrue(file.delete());
+ }
+
+ List<BlockHeader> unreachable = fork(chain, 100, 101);
+ HeaderStore reseeded = seedStore(chain, 200);
+ serve(unreachable);
+
+ VerificationException e = assertThrows(VerificationException.class, () -> new ElectrumServer().syncHeaders(new ChainTip(201, unreachable.getLast())));
+ assertTrue(e.getMessage().contains("shares no fork point"), e.getMessage());
+ assertEquals(200, reseeded.getTipHeight());
+ assertEquals(chain.getLast().getHash(), reseeded.getTipHash());
+ assertEquals(Integer.MAX_VALUE, ElectrumServer.lastReorgForkHeight);
+ }
+
+ /**
+ * The steady state, on both paths that sync: a header that extends the store tip is appended from the announcement itself, so a new block costs
+ * no request at all. The service passes the pair from its event; a history thread reads the same pair from the announced tip.
+ */
+ @Test
+ public void appendsAnAnnouncedHeaderWithoutFetching() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 12, CHAIN_TIME);
+ HeaderStore store = seedStore(chain, 10);
+ FakeElectrumServerRpc fake = serve(chain);
+
+ new ElectrumServer().syncHeaders(new ChainTip(11, chain.get(10)));
+ assertEquals(11, store.getTipHeight());
+ assertEquals(chain.get(10).getHash(), store.getTipHash());
+ assertEquals(0, fake.getChunkRequests());
+
+ AppServices.setAnnouncedTip(new ChainTip(12, chain.get(11)));
+ BlockHeader verified = new ElectrumServer().getVerifiedHeader(12);
+ assertEquals(chain.get(11).getHash(), verified.getHash());
+ assertEquals(12, store.getTipHeight());
+ assertEquals(0, fake.getChunkRequests());
+ }
+
+ /**
+ * The announced header is appended only where it descends from the store tip: the height it claims is a pre-check, and taking it on that alone
+ * would put a header of another chain through the chain state and turn an ordinary one block reorg into a refusal.
+ */
+ @Test
+ public void doesNotAppendAnAnnouncedHeaderThatDoesNotDescendFromTheStoreTip() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 10, CHAIN_TIME);
+ List<BlockHeader> branch = fork(chain, 9, 2);
+ HeaderStore store = seedStore(chain, 10);
+ FakeElectrumServerRpc fake = serve(branch);
+
+ //The announcement is at the height that extends the store, but on the chain that replaced its tip
+ new ElectrumServer().syncHeaders(new ChainTip(11, branch.getLast()));
+
+ assertEquals(11, store.getTipHeight());
+ assertEquals(branch.getLast().getHash(), store.getTipHash());
+ assertEquals(branch.get(9).getHash(), store.getHeader(10).getHash());
+ assertEquals(9, listener.getForkHeight());
+ //The chunk that exposed the divergence, then the window that resolved it
+ assertEquals(2, fake.getChunkRequests());
+ }
+
+ /**
+ * An announced header is no more trusted than a fetched one. The tip validation it has already passed only checks it against the target it claims
+ * for itself, so what the chain requires of it at that height is enforced here, where it is written.
+ */
+ @Test
+ public void refusesAnAnnouncedHeaderThatDoesNotMeetItsTarget() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 10, CHAIN_TIME);
+ HeaderStore store = seedStore(chain, 10);
+ serve(chain);
+ BlockHeader belowTarget = unmineHeader(chain.getLast(), CHAIN_TIME + 10);
+
+ VerificationException e = assertThrows(VerificationException.class, () -> new ElectrumServer().syncHeaders(new ChainTip(11, belowTarget)));
+ assertTrue(e.getMessage().contains("proof of work"), e.getMessage());
+
+ assertEquals(10, store.getTipHeight());
+ assertEquals(chain.getLast().getHash(), store.getTipHash());
+ }
+
+ /**
+ * Two threads syncing an empty store at once, which is the ordinary case on a cold install: the history threads of the open wallets and the sync
+ * service after its startup jitter. Without the lock both fetch the same chunk and the loser's append fails linkage, which reads as an invalid
+ * segment from an honest server.
+ */
+ @Test
+ public void fetchesEachChunkOnceWhenTwoThreadsRace() throws Exception {
+ int height = HeaderChainState.RETARGET_INTERVAL + 484;
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), height, CHAIN_TIME);
+ HeaderStore store = seedStore(chain, 0);
+ FakeElectrumServerRpc fake = serve(chain);
+ fake.setResponseDelayMillis(50);
+
+ CyclicBarrier barrier = new CyclicBarrier(2);
+ Callable<Void> sync = () -> {
+ barrier.await(10, TimeUnit.SECONDS);
+ new ElectrumServer().syncHeadersTo(height);
+ return null;
+ };
+
+ ExecutorService executorService = Executors.newFixedThreadPool(2);
+ try {
+ for(Future<Void> future : executorService.invokeAll(List.of(sync, sync))) {
+ future.get(60, TimeUnit.SECONDS); //rethrows a VerificationException from either thread
+ }
+ } finally {
+ executorService.shutdownNow();
+ }
+
+ assertEquals(height, store.getTipHeight());
+ assertEquals(chain.getLast().getHash(), store.getTipHash());
+ //One request per chunk: the thread that waited re-read the tip inside the loop and found nothing left to fetch
+ assertEquals(2, fake.getChunkRequests());
+ }
+
+ /**
+ * A stored header is never served while the announced tip disagrees with the store at that height. Serving the orphaned header instead is what
+ * turns a natural stale block into the dishonest server dialog: the proof for a transaction in the replacing block reconstructs no branch.
+ */
+ @Test
+ public void reconcilesBeforeServingAStoredHeaderTheAnnouncedTipContradicts() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 10, CHAIN_TIME);
+ List<BlockHeader> branch = fork(chain, 9, 1);
+ HeaderStore store = seedStore(chain, 10);
+ serve(branch);
+ AppServices.setAnnouncedTip(new ChainTip(10, branch.getLast()));
+
+ BlockHeader verified = new ElectrumServer().getVerifiedHeader(10);
+
+ assertEquals(branch.getLast().getHash(), verified.getHash());
+ assertEquals(branch.getLast().getHash(), store.getTipHash());
+ assertEquals(9, listener.getForkHeight());
+ }
+
+ @Test
+ public void servesAStoredHeaderWithoutFetchingWhileTheAnnouncedTipAgrees() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 10, CHAIN_TIME);
+ seedStore(chain, 10);
+ FakeElectrumServerRpc fake = serve(chain);
+ AppServices.setAnnouncedTip(new ChainTip(10, chain.getLast()));
+
+ //An in sync store never queues a proof behind a request of any kind
+ assertEquals(chain.get(7).getHash(), new ElectrumServer().getVerifiedHeader(8).getHash());
+ assertEquals(chain.get(9).getHash(), new ElectrumServer().getVerifiedHeader(10).getHash());
+ assertEquals(0, fake.getChunkRequests());
+ }
+
+ /**
+ * A chunk claiming more headers than it carries is a refusal like any other malformed response, not an exception escaping the sync. The response
+ * checks in the rpc layer already reject this shape, so what is pinned here is that the loop reading the chunk does not depend on them having run.
+ */
+ @Test
+ public void treatsAChunkShorterThanItsCountAsARefusal() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 12, CHAIN_TIME);
+ seedStore(chain, 10);
+ FakeElectrumServerRpc fake = serve(chain);
+
+ BlockHeaders malformed = new BlockHeaders();
+ malformed.count = 3;
+ malformed.max = HeaderChainState.RETARGET_INTERVAL;
+ malformed.hex = Utils.bytesToHex(chain.get(10).bitcoinSerialize()); //one header where three are claimed
+ fake.setMalformedResponse(malformed);
+
+ assertNull(new ElectrumServer().getVerifiedHeader(12));
+ }
+
+ /**
+ * The loaded store outlives the connection, so a file removed underneath it - by a user clearing the cache, or by another instance - has to be
+ * noticed. Every read of a file that is gone comes back empty, which the fork walk would otherwise report as a server sharing no fork point with
+ * a chain it served itself.
+ */
+ @Test
+ public void reloadsAStoreWhoseFileHasBeenRemoved() throws Exception {
+ //Further above the anchor than the reorg window reaches, as any real store is: the walk cannot fall back on the pinned hash to find a fork
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 150, CHAIN_TIME);
+ seedStore(chain, 150);
+ serve(chain);
+
+ for(File file : Storage.getHeadersDir().listFiles()) {
+ assertTrue(file.delete());
+ }
+
+ new ElectrumServer().syncHeaders(new ChainTip(150, chain.getLast()));
+
+ HeaderStore reloaded = ElectrumServer.getHeaderStore();
+ assertEquals(150, reloaded.getTipHeight());
+ assertEquals(chain.getLast().getHash(), reloaded.getTipHash());
+ }
+
+ /**
+ * A transport failure is not a refusal: the server has said nothing about these heights, so the wallet history fails as it does for any other
+ * failed call, rather than the height being reported as one the server could not substantiate.
+ */
+ @Test
+ public void reportsATransportFailureAsAServerException() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 12, CHAIN_TIME);
+ seedStore(chain, 10);
+ FakeElectrumServerRpc fake = serve(chain);
+ fake.setFailure(new ElectrumServerRpcException("Connection reset"));
+
+ assertThrows(ServerException.class, () -> new ElectrumServer().getVerifiedHeader(12));
+ }
+
+ /**
+ * A server without the call is a property of the server rather than of the height, and the caller acts on it by disabling verification for the
+ * session, so it must reach that caller as itself rather than as a refusal or a wrapped failure.
+ */
+ @Test
+ public void letsAnUnsupportedMethodReachTheCaller() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 12, CHAIN_TIME);
+ seedStore(chain, 10);
+ FakeElectrumServerRpc fake = serve(chain);
+ fake.setFailure(new UnsupportedMethodException("blockchain.block.headers", new IllegalStateException()));
+
+ assertThrows(UnsupportedMethodException.class, () -> new ElectrumServer().getVerifiedHeader(12));
+ }
+
+ /**
+ * A height at or below the last pin is verified by hash linkage to it, with no proof of work, difficulty or timestamp check: descent from a hash
+ * that is compiled in is what places a header at its height. These are the real mainnet headers closing difficulty period 15, whose last one is
+ * the pin at height 32255, so nothing here could be forged to pass.
+ */
+ @Test
+ public void verifiesAHistoricalHeaderByLinkageToItsPin() throws Exception {
+ Network.set(Network.MAINNET);
+ FakeElectrumServerRpc fake = serveFrom(PERIOD_15_CLOSE, 32248);
+
+ BlockHeader verified = new ElectrumServer().getVerifiedHeader(32250);
+
+ assertEquals(PERIOD_15_CLOSE.get(2).getHash(), verified.getHash());
+ //Fetched from the height asked for up to its pin, and every height between is now known
+ assertEquals(32250, fake.getLastStartHeight());
+ assertEquals(6, fake.getLastCount());
+ assertEquals(1, fake.getChunkRequests());
+ assertEquals(Network.MAINNET.getHeaderCheckpoints().getHash(32255), ElectrumServer.verifiedHistoricalHeaders.get(32255).getHash());
+
+ //A second height inside the range it already holds costs nothing
+ assertEquals(PERIOD_15_CLOSE.get(5).getHash(), new ElectrumServer().getVerifiedHeader(32253).getHash());
+ assertEquals(1, fake.getChunkRequests());
+ }
+
+ /**
+ * A request for a pinned height itself, where the range collapses to the single header the pin names and there is nothing to link.
+ */
+ @Test
+ public void verifiesAPinnedHeightFromItsOwnHeaderAlone() throws Exception {
+ Network.set(Network.MAINNET);
+ FakeElectrumServerRpc fake = serveFrom(PERIOD_15_CLOSE, 32248);
+
+ BlockHeader verified = new ElectrumServer().getVerifiedHeader(32255);
+
+ assertEquals(Network.MAINNET.getHeaderCheckpoints().getHash(32255), verified.getHash());
+ assertEquals(32255, fake.getLastStartHeight());
+ assertEquals(1, fake.getLastCount());
+ }
+
+ /**
+ * A later pass reaching below a range already verified links to the nearest header it holds rather than to the pin above it, so an already
+ * downloaded range is not downloaded again.
+ */
+ @Test
+ public void fetchesOnlyAsFarAsTheNearestVerifiedHeader() throws Exception {
+ Network.set(Network.MAINNET);
+ FakeElectrumServerRpc fake = serveFrom(PERIOD_15_CLOSE, 32248);
+ new ElectrumServer().getVerifiedHeader(32250);
+
+ BlockHeader verified = new ElectrumServer().getVerifiedHeader(32248);
+
+ assertEquals(PERIOD_15_CLOSE.getFirst().getHash(), verified.getHash());
+ //Anchored on the cached header at 32250 rather than on the pin at 32255
+ assertEquals(32248, fake.getLastStartHeight());
+ assertEquals(3, fake.getLastCount());
+ assertEquals(2, fake.getChunkRequests());
+ }
+
+ @Test
+ public void refusesAHistoricalRangeThatDoesNotLinkToItsPin() throws Exception {
+ Network.set(Network.MAINNET);
+ List<BlockHeader> tampered = new ArrayList<>(PERIOD_15_CLOSE);
+ BlockHeader original = tampered.get(4);
+ tampered.set(4, new BlockHeader(original.getVersion(), original.getPrevBlockHash(), original.getMerkleRoot(), null, original.getTime() + 1,
+ original.getDifficultyTarget(), original.getNonce()));
+ serveFrom(tampered, 32248);
+
+ assertNull(new ElectrumServer().getVerifiedHeader(32250));
+ assertTrue(ElectrumServer.verifiedHistoricalHeaders.isEmpty());
+ }
+
+ private static FakeElectrumServerRpc serveFrom(List<BlockHeader> chain, int baseHeight) {
+ FakeElectrumServerRpc fake = new FakeElectrumServerRpc(chain, baseHeight);
+ ElectrumServer.electrumServerRpc = fake;
+
+ return fake;
+ }
+
+ /**
+ * A run of the sync service happens on a background thread, so everything it touches has to be usable there. The connection check in particular
+ * cannot be the AppServices one, which reads a JavaFX Service and throws off the application thread.
+ */
+ @Test
+ public void runsTheServiceTaskFromABackgroundThread() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 12, CHAIN_TIME);
+ HeaderStore store = seedStore(chain, 10);
+ serve(chain);
+
+ runOffThread(() -> {
+ ElectrumServer.HeaderSyncService.syncAnnouncedHeaders(new ChainTip(12, chain.get(11)));
+ return null;
+ });
+
+ assertEquals(12, store.getTipHeight());
+ assertEquals(chain.getLast().getHash(), store.getTipHash());
+ }
+
+ /**
+ * getTransport() creates a transport where there is none, so a run firing after the connection closed has to do nothing at all rather than open
+ * one of its own outside the connection lifecycle.
+ */
+ @Test
+ public void doesNotSyncOnceTheConnectionHasGone() throws Exception {
+ List<BlockHeader> chain = mineChain(Network.REGTEST.getGenesisHeader(), 12, CHAIN_TIME);
+ HeaderStore store = seedStore(chain, 10);
+ FakeElectrumServerRpc fake = serve(chain);
+ ElectrumServer.transport = null;
+
+ runOffThread(() -> {
+ ElectrumServer.HeaderSyncService.syncAnnouncedHeaders(new ChainTip(12, chain.get(11)));
+ return null;
+ });
+
+ assertEquals(10, store.getTipHeight());
+ assertEquals(0, fake.getChunkRequests());
+ }
+
+ private static void runOffThread(Callable<Void> task) throws Exception {
+ ExecutorService executorService = Executors.newSingleThreadExecutor();
+ try {
+ executorService.submit(task).get(30, TimeUnit.SECONDS);
+ } finally {
+ executorService.shutdownNow();
+ }
+ }
+
+ private static HeaderStore seedStore(List<BlockHeader> chain, int toHeight) throws Exception {
+ HeaderStore store = ElectrumServer.getHeaderStore();
+ for(int height = 1; height <= toHeight; height++) {
+ store.append(chain.get(height - 1));
+ }
+
+ return store;
+ }
+
+ private static FakeElectrumServerRpc serve(List<BlockHeader> chain) {
+ FakeElectrumServerRpc fake = new FakeElectrumServerRpc(chain);
+ ElectrumServer.electrumServerRpc = fake;
+
+ return fake;
+ }
+
+ /**
+ * A chain sharing the given number of headers with the given one, and carrying the given number of its own above them.
+ */
+ private static List<BlockHeader> fork(List<BlockHeader> chain, int sharedHeaders, int branchHeaders) {
+ List<BlockHeader> branch = new ArrayList<>(chain.subList(0, sharedHeaders));
+ branch.addAll(mineChain(branch.getLast(), branchHeaders, BRANCH_TIME));
+
+ return branch;
+ }
+
+ private static List<BlockHeader> mineChain(BlockHeader previous, int count, long startTime) {
+ List<BlockHeader> chain = new ArrayList<>();
+ for(int i = 0; i < count; i++) {
+ previous = mineHeader(previous, startTime + i);
+ chain.add(previous);
+ }
+
+ return chain;
+ }
+
+ /**
+ * A header that links but does not meet the target, which regtest's trivial difficulty makes as easy to produce as a valid one.
+ */
+ private static BlockHeader unmineHeader(BlockHeader previous, long time) {
+ for(long nonce = 0; nonce < 1000; nonce++) {
+ BlockHeader header = new BlockHeader(1, previous.getHash(), Sha256Hash.ZERO_HASH, null, time, 0x207fffffL, nonce);
+ if(!header.verifyProofOfWork()) {
+ return header;
+ }
+ }
+
+ throw new IllegalStateException("Could not produce a regtest header below its target at time " + time);
+ }
+
+ private static BlockHeader mineHeader(BlockHeader previous, long time) {
+ for(long nonce = 0; nonce < 1000; nonce++) {
+ BlockHeader header = new BlockHeader(1, previous.getHash(), Sha256Hash.ZERO_HASH, null, time, 0x207fffffL, nonce);
+ if(header.verifyProofOfWork()) {
+ return header;
+ }
+ }
+
+ throw new IllegalStateException("Could not mine a regtest header at time " + time);
+ }
+
+ /**
+ * Answers header requests from a chain of the test's choosing, applying the same response checks a real server's answer is put through, and
+ * counting what it was asked for.
+ */
+ private static class FakeElectrumServerRpc extends SimpleElectrumServerRpc {
+ private final List<BlockHeader> chain;
+ private final int baseHeight; //the height of the header at index 0
+ private final AtomicInteger chunkRequests = new AtomicInteger();
+ private volatile int lastStartHeight;
+ private volatile int lastCount;
+ private volatile long responseDelayMillis;
+ private volatile RuntimeException failure;
+ private volatile BlockHeaders malformedResponse;
+
+ public FakeElectrumServerRpc(List<BlockHeader> chain) {
+ this(chain, 1);
+ }
+
+ public FakeElectrumServerRpc(List<BlockHeader> chain, int baseHeight) {
+ this.chain = List.copyOf(chain);
+ this.baseHeight = baseHeight;
+ }
+
+ @Override
+ public BlockHeaders getBlockHeadersChunk(Transport transport, int startHeight, int count) {
+ chunkRequests.incrementAndGet();
+ lastStartHeight = startHeight;
+ lastCount = count;
+ if(failure != null) {
+ throw failure;
+ }
+ if(malformedResponse != null) {
+ return malformedResponse; //returned unchecked, as an implementation that did not apply the response checks would
+ }
+
+ if(responseDelayMillis > 0) {
+ try {
+ Thread.sleep(responseDelayMillis);
+ } catch(InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ }
+
+ int index = startHeight - baseHeight;
+ int available = Math.max(0, Math.min(count, chain.size() - index));
+ List<BlockHeader> headers = available == 0 ? Collections.emptyList() : chain.subList(index, index + available);
+ BlockHeaders blockHeaders = new BlockHeaders();
+ blockHeaders.count = headers.size();
+ blockHeaders.max = HeaderChainState.RETARGET_INTERVAL;
+ blockHeaders.hex = headers.stream().map(header -> Utils.bytesToHex(header.bitcoinSerialize())).collect(Collectors.joining());
+
+ return ElectrumServerRpc.checkBlockHeaders(blockHeaders, startHeight, count, baseHeight + chain.size() - 1);
+ }
+
+ public int getChunkRequests() {
+ return chunkRequests.get();
+ }
+
+ public int getLastStartHeight() {
+ return lastStartHeight;
+ }
+
+ public int getLastCount() {
+ return lastCount;
+ }
+
+ public void setResponseDelayMillis(long responseDelayMillis) {
+ this.responseDelayMillis = responseDelayMillis;
+ }
+
+ public void setFailure(RuntimeException failure) {
+ this.failure = failure;
+ }
+
+ public void setMalformedResponse(BlockHeaders malformedResponse) {
+ this.malformedResponse = malformedResponse;
+ }
+ }
+
+ /**
+ * The event is dispatched on the thread that reconciled, so it is captured by the time the call returns.
+ */
+ public static class ChainReorgListener {
+ private volatile Integer forkHeight;
+
+ @Subscribe
+ public void chainReorg(ChainReorgEvent event) {
+ forkHeight = event.getForkHeight();
+ }
+
+ public Integer getForkHeight() {
+ return forkHeight;
+ }
+ }
+
+ /**
+ * A transport that reports itself connected without opening a socket, since the connection check reads the transport rather than the connection service.
+ */
+ private static class UnusedTransport extends TcpTransport {
+ public UnusedTransport() {
+ super(HostAndPort.fromParts("localhost", 1));
+ }
+
+ @Override
+ public String pass(String request) {
+ throw new UnsupportedOperationException("The fake server answers without the transport");
+ }
+
+ @Override
+ public boolean isConnected() {
+ return true;
+ }
+ }
+}Why this scored 61/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.