resend batch requests that have been skipped as making the batch response too large, so a large wallet history page syncs instead of failing every refresh
What changed, and why it matters
This commit fixes a wallet-sync reliability problem. When Sparrow asked an Electrum server for a large batch of transaction history data, some servers would skip part of the request because the answer would be too big. Previously those skipped requests were treated as failures, so a large wallet's history page could never finish refreshing. The change detects the 'response too large' signal from Frigate and ElectrumX servers and automatically re-sends the skipped items in smaller batches until they succeed or truly fail for another reason. It is a robustness improvement, not a typical security bug fix.
Treat as a normal reliability/bug-fix commit. Reviewers should verify that the resend recursion cannot be abused by a hostile or misconfigured Electrum server to cause excessive client-side network load, and confirm the fresh-id logic does not break idempotency assumptions elsewhere in the RPC layer. No urgent security patch is indicated.
Security signals we found
Error-code parsing now distinguishes server capacity signals from real request errors
Recursive resend with fresh request ids could, in theory, hide a malicious server's behavior if an attacker-controlled server repeatedly returns RESPONSE_TOO_LARGE for a subset of requests
No input validation changes; the patch only changes batch retry logic
No cryptographic, authentication, or privacy changes visible in the diff
Evidence from the diff
PagedBatchRequestBuilder.executePage now catches JsonRpcBatchException, filters responses whose ErrorMessage matches ElectrumServerRpc.isResponseTooLarge (Frigate code -32006 or ElectrumX -32600 with message starting ‘response too large’), and recursively re-executes only the skipped requests with fresh ids. The guard skipped.isEmpty() || skipped.size() == page.size() prevents infinite resend loops. executeBatch was extracted to preserve the original single-batch behavior. Tests cover Frigate, ElectrumX, mixed errors, tolerant execution, and no-resend edge cases.
Changed components
src/main/java/com/sparrowwallet/sparrow/net/ElectrumServerRpc.javasrc/main/java/com/sparrowwallet/sparrow/net/PagedBatchRequestBuilder.javasrc/test/java/com/sparrowwallet/sparrow/net/PagedBatchRequestBuilderTest.javaInspect captured patch +228 / −1
### src/main/java/com/sparrowwallet/sparrow/net/ElectrumServerRpc.java
@@ -3,6 +3,7 @@
import com.github.arteam.simplejsonrpc.client.Transport;
import com.github.arteam.simplejsonrpc.client.exception.JsonRpcBatchException;
import com.github.arteam.simplejsonrpc.client.exception.JsonRpcException;
+import com.github.arteam.simplejsonrpc.core.domain.ErrorMessage;
import com.sparrowwallet.drongo.protocol.HeaderChainState;
import com.sparrowwallet.drongo.protocol.VerificationException;
import com.sparrowwallet.drongo.wallet.BlockTransactionHash;
@@ -18,6 +19,15 @@ public interface ElectrumServerRpc {
/** The JSON-RPC standard code for a method the server does not implement, which bitcoind returns as well as Electrum servers. */
int METHOD_NOT_FOUND = -32601;
+ /** Frigate's code for a batch request it did not process because the batch response had already reached its size limit. */
+ int RESPONSE_TOO_LARGE = -32006;
+
+ /** The JSON-RPC standard code for an invalid request, which ElectrumX also returns for a request whose response would exceed its MAX_SEND. */
+ int INVALID_REQUEST = -32600;
+
+ /** How ElectrumX begins the message of an INVALID_REQUEST error for a response over its MAX_SEND, which it follows with the configured limit. */
+ String ELECTRUMX_RESPONSE_TOO_LARGE_PREFIX = "response too large";
+
/**
* Checks the connection is alive with server.ping, throwing ElectrumServerRpcException if the server does not answer.
*/
@@ -163,6 +173,12 @@ static boolean isMethodNotFound(JsonRpcException e) {
return e.getErrorMessage() != null && e.getErrorMessage().getCode() == METHOD_NOT_FOUND;
}
+ /** Whether the server left a batch request unprocessed because the batch response had reached its size limit, as Frigate and ElectrumX report it. */
+ static boolean isResponseTooLarge(ErrorMessage error) {
+ return error != null && (error.getCode() == RESPONSE_TOO_LARGE
+ || (error.getCode() == INVALID_REQUEST && error.getMessage() != null && error.getMessage().startsWith(ELECTRUMX_RESPONSE_TOO_LARGE_PREFIX)));
+ }
+
/**
* Applies Electrum's checks to a blockchain.block.headers response. A server returning at most a partial difficulty period cannot serve the
* header sync at all, and a response shorter than requested is only explicable by the server having reached its own announced tip.
### src/main/java/com/sparrowwallet/sparrow/net/PagedBatchRequestBuilder.java
@@ -176,10 +176,44 @@ public Map<K, V> executeTolerant(int maxAttempts, V errorValue, Consumer<JsonRpc
}
/**
- * Executes a single page, mapping any batch exception back to the request ids the caller supplied
+ * Executes a single page, sending again the requests a server left unprocessed once its response reached a size limit, which it answers normally
+ * in a smaller batch. A page is resent only when part of it was answered, so each resend is smaller than the page before it.
*/
@NotNull
+ @SuppressWarnings("unchecked")
private Map<K, V> executePage(JsonRpcClient client, List<Request<K>> page, int maxAttempts) throws Exception {
+ try {
+ return executeBatch(client, page, maxAttempts);
+ } catch(JsonRpcBatchException e) {
+ List<Request<K>> skipped = page.stream().filter(request -> ElectrumServerRpc.isResponseTooLarge(e.getErrors().get(request.id))).map(request -> new Request<K>(request.id, counter == null ? null : counter.incrementAndGet(), request.method, request.params)).toList();
+
+ if(skipped.isEmpty() || skipped.size() == page.size()) {
+ throw e;
+ }
+
+ Map<Object, Object> successes = new HashMap<>(e.getSuccesses());
+ Map<Object, ErrorMessage> errors = new HashMap<>(e.getErrors());
+ skipped.forEach(request -> errors.remove(request.id));
+ try {
+ successes.putAll(executePage(client, skipped, maxAttempts));
+ } catch(JsonRpcBatchException resent) {
+ successes.putAll(resent.getSuccesses());
+ errors.putAll(resent.getErrors());
+ }
+
+ if(errors.isEmpty()) {
+ return (Map<K, V>)successes;
+ }
+
+ throw new JsonRpcBatchException(e.getMessage(), successes, errors);
+ }
+ }
+
+ /**
+ * Executes a single batch, mapping any batch exception back to the request ids the caller supplied
+ */
+ @NotNull
+ private Map<K, V> executeBatch(JsonRpcClient client, List<Request<K>> page, int maxAttempts) throws Exception {
if(counter != null) {
Map<Long, K> counterIdMap = new HashMap<>();
BatchRequestBuilder<Long, V> batchRequest = client.createBatchRequest().keysType(Long.class).returnType(returnType);
### src/test/java/com/sparrowwallet/sparrow/net/PagedBatchRequestBuilderTest.java
@@ -145,6 +145,100 @@ public void pageSizeHalvedAfterATimeoutIsNeverBelowOne() throws Exception {
assertEquals(3, halved.requests.size());
}
+ @Test
+ public void requestsSkippedForResponseSizeAreSentAgain() throws Exception {
+ //A server answering two requests per batch, as a size limit reached on every second response would, leaves the page incomplete twice
+ SizeLimitedBatchTransport transport = new SizeLimitedBatchTransport(2, Set.of());
+ Map<String, String> result = builder(transport, 6).execute(1);
+
+ assertEquals(6, result.size());
+ for(String id : List.of("a", "b", "c", "d", "e", "f")) {
+ assertEquals("result:" + id, result.get(id));
+ }
+ assertEquals(3, transport.requests.size());
+ assertEquals(12, transport.ids.size());
+ assertEquals(12, new HashSet<>(transport.ids).size(), "Each request sent again must carry a new id");
+ }
+
+ @Test
+ public void requestsSkippedForResponseSizeAreSentAgainInTolerantExecution() throws Exception {
+ SizeLimitedBatchTransport transport = new SizeLimitedBatchTransport(1, Set.of("e"));
+ Map<String, String> result = builder(transport, 3).executeTolerant(1, "ERROR", e -> {});
+
+ assertEquals(Set.of("a", "b", "c", "d", "e", "f"), result.keySet());
+ assertEquals("ERROR", result.get("e"));
+ assertEquals("result:f", result.get("f"));
+ }
+
+ @Test
+ public void otherErrorsAreReportedAlongsideRequestsSentAgain() {
+ //An error the server returns for an answered request is not resent, and is reported with the results of the requests that were resent
+ SizeLimitedBatchTransport transport = new SizeLimitedBatchTransport(2, Set.of("b", "e"));
+ JsonRpcBatchException e = assertThrows(JsonRpcBatchException.class, () -> builder(transport, 6).execute(1));
+
+ assertEquals(Set.of("b", "e"), new HashSet<>(e.getErrors().keySet()));
+ assertEquals(Set.of("a", "c", "d", "f"), new HashSet<>(e.getSuccesses().keySet()));
+ assertEquals(3, transport.requests.size());
+ }
+
+ @Test
+ public void pageWithEveryRequestSkippedIsNotSentAgain() {
+ //A server answering nothing would never let a resend shrink, so the errors are returned rather than the page sent again
+ SizeLimitedBatchTransport transport = new SizeLimitedBatchTransport(0, Set.of());
+ JsonRpcBatchException e = assertThrows(JsonRpcBatchException.class, () -> builder(transport, 6).execute(1));
+
+ assertEquals(6, e.getErrors().size());
+ assertEquals(1, transport.requests.size());
+ }
+
+ @Test
+ public void requestsSkippedByElectrumXAreSentAgain() throws Exception {
+ //ElectrumX skips in the order responses complete, so the skipped requests can be anywhere in the page rather than at its end
+ FirstSendErrorTransport transport = new FirstSendErrorTransport(Map.of("b", ELECTRUMX_TOO_LARGE, "d", ELECTRUMX_TOO_LARGE));
+ Map<String, String> result = builder(transport, 6).execute(1);
+
+ assertEquals(6, result.size());
+ for(String id : List.of("a", "b", "c", "d", "e", "f")) {
+ assertEquals("result:" + id, result.get(id));
+ }
+ assertEquals(2, transport.requests.size());
+ }
+
+ @Test
+ public void otherInvalidRequestErrorsAreNotSentAgain() {
+ FirstSendErrorTransport transport = new FirstSendErrorTransport(Map.of("c", new ErrorReply(ElectrumServerRpc.INVALID_REQUEST, "Invalid Request")));
+ JsonRpcBatchException e = assertThrows(JsonRpcBatchException.class, () -> builder(transport, 6).execute(1));
+
+ assertEquals(Set.of("c"), new HashSet<>(e.getErrors().keySet()));
+ assertEquals(5, e.getSuccesses().size());
+ assertEquals(1, transport.requests.size());
+ }
+
+ @Test
+ public void pageWithEveryRequestSkippedByElectrumXIsNotSentAgain() {
+ //A response over the limit on its own is refused however small the batch, so a page answering nothing is returned with its errors
+ Map<String, ErrorReply> errors = new java.util.HashMap<>();
+ for(String id : List.of("a", "b", "c", "d", "e", "f")) {
+ errors.put(id, ELECTRUMX_TOO_LARGE);
+ }
+ FirstSendErrorTransport transport = new FirstSendErrorTransport(errors);
+ JsonRpcBatchException e = assertThrows(JsonRpcBatchException.class, () -> builder(transport, 6).execute(1));
+
+ assertEquals(6, e.getErrors().size());
+ assertEquals(1, transport.requests.size());
+ }
+
+ @Test
+ public void requestsSkippedByFrigateAndElectrumXAreBothSentAgain() throws Exception {
+ FirstSendErrorTransport transport = new FirstSendErrorTransport(Map.of("b", ELECTRUMX_TOO_LARGE, "e", new ErrorReply(ElectrumServerRpc.RESPONSE_TOO_LARGE, "response too large")));
+ Map<String, String> result = builder(transport, 6).execute(1);
+
+ assertEquals(6, result.size());
+ assertEquals("result:b", result.get("b"));
+ assertEquals("result:e", result.get("e"));
+ assertEquals(2, transport.requests.size());
+ }
+
private PagedBatchRequestBuilder<String, String> builder(Transport transport) {
return builder(transport, 2);
}
@@ -202,6 +296,89 @@ public String pass(String request) throws IOException {
}
}
+ /**
+ * Answers the given number of requests in each batch and returns the response too large error for the rest, as a server limiting the size
+ * of a batch response does, except those whose single parameter is named as failing.
+ */
+ private static class SizeLimitedBatchTransport implements Transport {
+ private final int answered;
+ private final Set<String> failingParams;
+ private final List<String> requests = new ArrayList<>();
+ private final List<Long> ids = new ArrayList<>();
+
+ SizeLimitedBatchTransport(int answered, Set<String> failingParams) {
+ this.answered = answered;
+ this.failingParams = failingParams;
+ }
+
+ @Override
+ public String pass(String request) throws IOException {
+ requests.add(request);
+ ArrayNode responses = MAPPER.createArrayNode();
+ int index = 0;
+ for(JsonNode node : MAPPER.readTree(request)) {
+ ids.add(node.get("id").asLong());
+ String param = node.get("params").get(0).asText();
+ ObjectNode response = responses.addObject();
+ response.put("jsonrpc", "2.0");
+ response.set("id", node.get("id"));
+ if(index++ >= answered) {
+ ObjectNode error = response.putObject("error");
+ error.put("code", ElectrumServerRpc.RESPONSE_TOO_LARGE);
+ error.put("message", "response too large");
+ } else if(failingParams.contains(param)) {
+ ObjectNode error = response.putObject("error");
+ error.put("code", -32603);
+ error.put("message", "failed for " + param);
+ } else {
+ response.put("result", "result:" + param);
+ }
+ }
+
+ return MAPPER.writeValueAsString(responses);
+ }
+ }
+
+ private static final ErrorReply ELECTRUMX_TOO_LARGE = new ErrorReply(ElectrumServerRpc.INVALID_REQUEST, "response too large (over 8,100,000 bytes");
+
+ private record ErrorReply(int code, String message) {}
+
+ /**
+ * Returns the given error for a request the first time its single parameter is sent, and a result every time after, so that a request sent
+ * again succeeds and one not sent again stays an error.
+ */
+ private static class FirstSendErrorTransport implements Transport {
+ private final Map<String, ErrorReply> firstSendErrors;
+ private final Set<String> sentParams = new HashSet<>();
+ private final List<String> requests = new ArrayList<>();
+
+ FirstSendErrorTransport(Map<String, ErrorReply> firstSendErrors) {
+ this.firstSendErrors = firstSendErrors;
+ }
+
+ @Override
+ public String pass(String request) throws IOException {
+ requests.add(request);
+ ArrayNode responses = MAPPER.createArrayNode();
+ for(JsonNode node : MAPPER.readTree(request)) {
+ String param = node.get("params").get(0).asText();
+ ObjectNode response = responses.addObject();
+ response.put("jsonrpc", "2.0");
+ response.set("id", node.get("id"));
+ ErrorReply reply = firstSendErrors.get(param);
+ if(sentParams.add(param) && reply != null) {
+ ObjectNode error = response.putObject("error");
+ error.put("code", reply.code());
+ error.put("message", reply.message());
+ } else {
+ response.put("result", "result:" + param);
+ }
+ }
+
+ return MAPPER.writeValueAsString(responses);
+ }
+ }
+
/**
* A transport that has seen a timeout, which is what the page size is halved on.
*/Why this scored 27/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.