keep the results of every other page when one page of a batched history, mempool, transaction, block header or block stats request returns an error
What changed, and why it matters
This commit fixes a bug in how Sparrow Wallet talks to Electrum servers when asking for wallet history, mempool, transactions, block headers, or block stats in large batches. Previously, if one 'page' of the batched request failed, the wallet would throw away all results from later pages. After the fix, it keeps the successful results from every page and only marks or drops the specific items that failed. This makes the wallet more reliable and prevents it from missing transactions or showing an incomplete wallet history when a server has a transient error.
Review and merge. The change is defensive and improves robustness of server communication. No immediate incident response is indicated, but users relying on accurate wallet history should upgrade once released.
Security signals we found
Fixes partial data loss on batched RPC failures
Prevents wallet history/mempool from appearing incomplete due to server-side errors
Adds unit tests covering multi-page batch error handling
Evidence from the diff
The change refactors BatchedElectrumServerRpc to use a new PagedBatchRequestBuilder.executeTolerant() method. The old code called execute() and, on JsonRpcBatchException, either failed entirely or returned only the successes from the single page that threw. Because PagedBatchRequestBuilder sends requests in pages, a per-page JsonRpcBatchException would discard all subsequent pages. executeTolerant() iterates pages, accumulates successes from each page, and either substitutes an errorValue for failed keys or omits them when errorValue is null. Tests confirm that results from pages before and after an errored page are preserved.
Changed components
BatchedElectrumServerRpcPagedBatchRequestBuilderElectrum server RPC batching for history, mempool, transactions, block headers, block statsInspect captured patch +114 / −47
### src/main/java/com/sparrowwallet/sparrow/net/BatchedElectrumServerRpc.java
@@ -101,18 +101,13 @@ public Map<String, ScriptHashTx[]> getScriptHashHistory(Transport transport, Wal
}
try {
- return batchRequest.execute();
- } catch (JsonRpcBatchException e) {
if(failOnError) {
- throw new ElectrumServerRpcException("Failed to retrieve transaction history for paths: " + nodeRangesToString((Collection<String>)e.getErrors().keySet()), e);
- }
-
- Map<String, ScriptHashTx[]> result = (Map<String, ScriptHashTx[]>)e.getSuccesses();
- for(Object key : e.getErrors().keySet()) {
- result.put((String)key, new ScriptHashTx[] {ScriptHashTx.ERROR_TX});
+ return batchRequest.execute();
}
- return result;
+ return batchRequest.executeTolerant(DEFAULT_MAX_ATTEMPTS, new ScriptHashTx[] {ScriptHashTx.ERROR_TX}, e -> {});
+ } catch(JsonRpcBatchException e) {
+ throw new ElectrumServerRpcException("Failed to retrieve transaction history for paths: " + nodeRangesToString((Collection<String>)e.getErrors().keySet()), e);
} catch(Exception e) {
throw new ElectrumServerRpcException("Failed to retrieve transaction history for paths: " + nodeRangesToString(pathScriptHashes.keySet()), e);
}
@@ -128,18 +123,13 @@ public Map<String, ScriptHashTx[]> getScriptHashMempool(Transport transport, Wal
}
try {
- return batchRequest.execute();
- } catch(JsonRpcBatchException e) {
if(failOnError) {
- throw new ElectrumServerRpcException("Failed to retrieve mempool transactions for paths: " + nodeRangesToString((Collection<String>)e.getErrors().keySet()), e);
+ return batchRequest.execute();
}
- Map<String, ScriptHashTx[]> result = (Map<String, ScriptHashTx[]>)e.getSuccesses();
- for(Object key : e.getErrors().keySet()) {
- result.put((String)key, new ScriptHashTx[] {ScriptHashTx.ERROR_TX});
- }
-
- return result;
+ return batchRequest.executeTolerant(DEFAULT_MAX_ATTEMPTS, new ScriptHashTx[] {ScriptHashTx.ERROR_TX}, e -> {});
+ } catch(JsonRpcBatchException e) {
+ throw new ElectrumServerRpcException("Failed to retrieve mempool transactions for paths: " + nodeRangesToString((Collection<String>)e.getErrors().keySet()), e);
} catch(Exception e) {
throw new ElectrumServerRpcException("Failed to retrieve mempool transactions for paths: " + nodeRangesToString(pathScriptHashes.keySet()), e);
}
@@ -209,7 +199,6 @@ public String unsubscribeSilentPayments(Transport transport, String scanPrivKeyH
}
@Override
- @SuppressWarnings("unchecked")
public Map<Integer, String> getBlockHeaders(Transport transport, Wallet wallet, Set<Integer> blockHeights) {
PagedBatchRequestBuilder<Integer, String> batchRequest = PagedBatchRequestBuilder.create(transport, idCounter).keysType(Integer.class).returnType(String.class);
EventManager.get().post(new WalletHistoryStatusEvent(wallet, true, "Retrieving " + blockHeights.size() + " block headers"));
@@ -219,16 +208,13 @@ public Map<Integer, String> getBlockHeaders(Transport transport, Wallet wallet,
}
try {
- return batchRequest.execute();
- } catch(JsonRpcBatchException e) {
- return (Map<Integer, String>)e.getSuccesses();
+ return batchRequest.executeTolerant(DEFAULT_MAX_ATTEMPTS, null, e -> {});
} catch(Exception e) {
throw new ElectrumServerRpcException("Failed to retrieve block headers for block heights: " + blockHeights, e);
}
}
@Override
- @SuppressWarnings("unchecked")
public Map<Integer, BlockStats> getBlockStats(Transport transport, Set<Integer> blockHeights) {
PagedBatchRequestBuilder<Integer, BlockStats> batchRequest = PagedBatchRequestBuilder.create(transport, idCounter).keysType(Integer.class).returnType(BlockStats.class);
@@ -237,16 +223,13 @@ public Map<Integer, BlockStats> getBlockStats(Transport transport, Set<Integer>
}
try {
- return batchRequest.execute();
- } catch(JsonRpcBatchException e) {
- return (Map<Integer, BlockStats>)e.getSuccesses();
+ return batchRequest.executeTolerant(DEFAULT_MAX_ATTEMPTS, null, e -> {});
} catch(Exception e) {
throw new ElectrumServerRpcException("Failed to retrieve block stats for block heights: " + blockHeights, e);
}
}
@Override
- @SuppressWarnings("unchecked")
public Map<String, String> getTransactions(Transport transport, Wallet wallet, Set<String> txids) {
PagedBatchRequestBuilder<String, String> batchRequest = PagedBatchRequestBuilder.create(transport, idCounter).keysType(String.class).returnType(String.class);
EventManager.get().post(new WalletHistoryStatusEvent(wallet, true, "Retrieving " + txids.size() + " transactions"));
@@ -256,24 +239,13 @@ public Map<String, String> getTransactions(Transport transport, Wallet wallet, S
}
try {
- return batchRequest.execute();
- } catch(JsonRpcBatchException e) {
- Map<String, String> result = (Map<String, String>)e.getSuccesses();
-
- String strErrorTx = Sha256Hash.ZERO_HASH.toString();
- for(Object hash : e.getErrors().keySet()) {
- String txhash = (String)hash;
- result.put(txhash, strErrorTx);
- }
-
- return result;
+ return batchRequest.executeTolerant(DEFAULT_MAX_ATTEMPTS, Sha256Hash.ZERO_HASH.toString(), e -> {});
} catch(Exception e) {
throw new ElectrumServerRpcException("Failed to retrieve transactions for txids: " + txids.stream().map(txid -> "[" + txid.substring(0, 6) + "]").collect(Collectors.toList()), e);
}
}
@Override
- @SuppressWarnings("unchecked")
public Map<String, VerboseTransaction> getVerboseTransactions(Transport transport, Set<String> txids, String scriptHash) {
PagedBatchRequestBuilder<String, VerboseTransaction> batchRequest = PagedBatchRequestBuilder.create(transport, idCounter).keysType(String.class).returnType(VerboseTransaction.class);
for(String txid : txids) {
@@ -282,10 +254,7 @@ public Map<String, VerboseTransaction> getVerboseTransactions(Transport transpor
try {
//The server may return an error if the transaction has not yet been broadcasted - this is a valid state so only try once
- return batchRequest.execute(1);
- } catch(JsonRpcBatchException e) {
- log.debug("Some errors retrieving transactions: " + e.getErrors());
- return (Map<String, VerboseTransaction>)e.getSuccesses();
+ return batchRequest.executeTolerant(1, null, e -> log.debug("Some errors retrieving transactions: " + e.getErrors()));
} catch(Exception e) {
throw new ElectrumServerRpcException("Failed to retrieve verbose transactions for txids: " + txids, e);
}
### src/main/java/com/sparrowwallet/sparrow/net/PagedBatchRequestBuilder.java
@@ -150,9 +150,9 @@ public Map<K, V> execute(int maxAttempts) throws Exception {
* later pages from being sent, which is what makes a per-key error meaningful when the request spans more than one page.
*
* @param maxAttempts number of times to try each page
- * @param errorValue the value to record for keys the server returned an error for
+ * @param errorValue the value to record for keys the server returned an error for, or null to leave those keys out
* @param batchErrorCheck applied to each page's errors before they are substituted, to let the caller classify whole-batch failures
- * @return map of responses by request ids, covering every requested id
+ * @return map of responses by request ids, covering every requested id unless errorValue is null
*/
@NotNull
@SuppressWarnings("unchecked")
@@ -166,8 +166,10 @@ public Map<K, V> executeTolerant(int maxAttempts, V errorValue, Consumer<JsonRpc
} catch(JsonRpcBatchException e) {
batchErrorCheck.accept(e);
allResults.putAll((Map<K, V>)e.getSuccesses());
- for(Object key : e.getErrors().keySet()) {
- allResults.put((K)key, errorValue);
+ if(errorValue != null) {
+ for(Object key : e.getErrors().keySet()) {
+ allResults.put((K)key, errorValue);
+ }
}
}
}
### src/test/java/com/sparrowwallet/sparrow/net/ElectrumServerRpcTest.java
@@ -18,11 +18,16 @@
import java.nio.file.Path;
import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
+import java.util.Set;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertThrows;
/**
@@ -91,6 +96,86 @@ public String pass(String request) throws java.io.IOException {
}
}
+ /**
+ * Where errors are tolerated, a path the server errored on is marked as such and every other path keeps its history, including those on pages
+ * before and after the one with the error.
+ */
+ @Test
+ public void tolerantHistoryCoversEveryPathAcrossPages() {
+ List<String> paths = new ArrayList<>();
+ for(int i = 0; i < PagedBatchRequestBuilder.DEFAULT_PAGE_SIZE * 2 + 50; i++) {
+ paths.add("m/0/" + i);
+ }
+ Map<String, String> pathScriptHashes = new LinkedHashMap<>();
+ paths.forEach(path -> pathScriptHashes.put(path, "sh:" + path));
+ String failingPath = paths.get(PagedBatchRequestBuilder.DEFAULT_PAGE_SIZE + 25);
+ ErroringTransport transport = new ErroringTransport(Set.of("sh:" + failingPath));
+
+ Map<String, ScriptHashTx[]> result = new BatchedElectrumServerRpc(0, 25).getScriptHashHistory(transport, null, pathScriptHashes, false);
+
+ assertEquals(new HashSet<>(paths), result.keySet());
+ assertSame(ScriptHashTx.ERROR_TX, result.get(failingPath)[0]);
+ assertEquals(800000, result.get(paths.getFirst())[0].height);
+ assertEquals(800000, result.get(paths.getLast())[0].height);
+ }
+
+ /**
+ * A transaction the server errored on is marked with the error sentinel, and every other transaction is returned, whichever page it was on.
+ */
+ @Test
+ public void tolerantTransactionsCoverEveryTxidAcrossPages() {
+ List<String> txids = new ArrayList<>();
+ for(int i = 1; i <= PagedBatchRequestBuilder.DEFAULT_PAGE_SIZE * 2 + 50; i++) {
+ txids.add(String.format("%064x", i));
+ }
+ String failingTxid = txids.get(PagedBatchRequestBuilder.DEFAULT_PAGE_SIZE + 25);
+ ErroringTransport transport = new ErroringTransport(Set.of(failingTxid));
+
+ Map<String, String> result = new BatchedElectrumServerRpc(0, 25).getTransactions(transport, null, new LinkedHashSet<>(txids));
+
+ assertEquals(new HashSet<>(txids), result.keySet());
+ assertEquals(Sha256Hash.ZERO_HASH.toString(), result.get(failingTxid));
+ assertEquals("tx:" + txids.getFirst(), result.get(txids.getFirst()));
+ assertEquals("tx:" + txids.getLast(), result.get(txids.getLast()));
+ }
+
+ /**
+ * Answers history requests with a single confirmed entry and transaction requests with a placeholder, returning an error for the named parameters.
+ */
+ private static class ErroringTransport implements Transport {
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+
+ private final Set<String> failingParams;
+
+ ErroringTransport(Set<String> failingParams) {
+ this.failingParams = failingParams;
+ }
+
+ @Override
+ public String pass(String request) throws java.io.IOException {
+ ArrayNode responses = MAPPER.createArrayNode();
+ for(JsonNode node : MAPPER.readTree(request)) {
+ String param = node.get("params").get(0).asText();
+ ObjectNode response = responses.addObject();
+ response.put("jsonrpc", "2.0");
+ response.set("id", node.get("id"));
+ if(failingParams.contains(param)) {
+ ObjectNode error = response.putObject("error");
+ error.put("code", -32603);
+ error.put("message", "failed for " + param);
+ } else if(node.get("method").asText().equals("blockchain.scripthash.get_history")) {
+ ObjectNode entry = response.putArray("result").addObject();
+ entry.put("height", 800000);
+ entry.put("tx_hash", String.format("%064x", 1));
+ } else {
+ response.put("result", "tx:" + param);
+ }
+ }
+
+ return MAPPER.writeValueAsString(responses);
+ }
+ }
+
@Test
public void acceptsAFullResponse() {
assertDoesNotThrow(() -> ElectrumServerRpc.checkBlockHeaders(headers(2016, 2016, 2016), 800000, 2016, TIP));
### src/test/java/com/sparrowwallet/sparrow/net/PagedBatchRequestBuilderTest.java
@@ -73,6 +73,17 @@ public void tolerantExecutionKeepsEveryResultWhenNothingFails() throws Exception
assertFalse(result.containsValue("ERROR"));
}
+ @Test
+ public void tolerantExecutionWithoutAnErrorValueLeavesOutOnlyTheFailingKeys() throws Exception {
+ //Six requests over three pages, with the error in the middle page so that a page on each side of it must keep its results
+ FakeBatchTransport transport = new FakeBatchTransport(Set.of("c"), 0);
+ Map<String, String> result = builder(transport).executeTolerant(1, null, e -> {});
+
+ assertEquals(Set.of("a", "b", "d", "e", "f"), result.keySet());
+ assertFalse(result.containsValue(null));
+ assertEquals(3, transport.requests.size());
+ }
+
@Test
public void existingExecutionDiscardsResultsOutsideTheFailingPage() {
//The behaviour executeTolerant exists to avoid, asserted so that a change to either is a deliberate oneWhy this scored 36/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.