common: fix BOLT 12 payer_proof marker-jump via next_marker helper
What changed, and why it matters
This commit fixes a mismatch in Core Lightning's BOLT 12 invoice proof code. The BOLT 12 specification had contradictory writer and reader rules: writers were required to jump from marker 239 to marker 1000000000, but readers were written to only accept consecutive numbers. As a result, a valid invoice proof created by the rules could be rejected by the same implementation's checker. The fix introduces a single shared helper function so both sides agree on what number comes next, and updates the test to verify the jump is accepted end-to-end.
Review and merge the fix, then ensure the updated test passes. Consider backporting to maintained release branches because the bug can cause valid BOLT 12 invoices to be rejected. No immediate emergency response is indicated because exploitation requires a compliant but boundary-triggering invoice, and the fix is straightforward.
Security signals we found
Protocol implementation bug: writer and reader disagree on valid marker sequence
Potential denial of service: valid BOLT 12 payer proofs could be rejected by CLN
Spec contradiction fix: aligns CLN with the writer-mandated 239 -> 1000000000 jump
Single shared helper introduced to prevent future writer/reader divergence
End-to-end test added to lock in the boundary behavior
Evidence from the diff
In common/bolt12_proof.c, the writer used ++last_type to emit proof_omitted_tlvs markers, while check_payer_proof validated omitted != prev_omitted + 1. At the boundary where the previous omitted marker is 239, the writer rule jumps to 1000000000 (skipping the reserved signature/payer-proof range 240..999999999), but the reader rejected that jump. The patch adds a static next_marker(prev) helper that returns the smallest valid marker greater than prev, handling the 239 -> 1000000000 transition. Both make_unsigned_proof_ and check_payer_proof now use this helper, eliminating the writer/reader asymmetry. The unit test is rewritten as a real end-to-end test that builds an invoice with 64 omitted dummy TLVs in the extension range and asserts check_payer_proof accepts the resulting […, 239, 1000000000] marker sequence.
Changed components
common/bolt12_proof.ccommon/test/run-bolt12_proof_marker_jump.cBOLT 12 payer_proof marker emission and validationInspect captured patch +256 / −83
### common/bolt12_proof.c
@@ -10,6 +10,52 @@
#include <common/utils.h>
#include <inttypes.h>
+/* BOLT-payer_proof #12:
+ * A *marker number* is a `bigsize` in the ranges `1..239` or
+ * `1000000000..3999999999`.
+ *
+ * The *successor marker* of `prev`, written `next_marker(prev)`, is the
+ * smallest marker number strictly greater than `prev`. So:
+ *
+ * next_marker(0) = 1
+ * next_marker(1..238) = prev + 1
+ * next_marker(239) = 1000000000
+ * next_marker(>=240, <1000000000) = 1000000000 (jumps the reserved gap)
+ * next_marker(1000000000..3999999998) = prev + 1
+ *
+ * Both the writer-side marker emission and the reader-side validation use
+ * this single helper, which resolves the writer/reader rule asymmetry
+ * raised on https://github.com/lightning/bolts/pull/1295#discussion_r3286972971
+ * by giving both sides one definition of "what comes after prev".
+ */
+static bigsize_t next_marker(bigsize_t prev)
+{
+ if (prev < 1)
+ return 1;
+ if (prev < 239)
+ return prev + 1;
+ if (prev < 1000000000)
+ return 1000000000;
+ /* prev is already in the high marker range. */
+ return prev + 1;
+}
+
+/* Is there an included non-signature TLV in `pptlv` whose successor marker
+ * is exactly `omitted`? Used by the reader to accept a marker that follows
+ * an included TLV rather than the previous marker. */
+static bool find_included_marker_predecessor(const struct tlv_payer_proof *pptlv,
+ bigsize_t omitted)
+{
+ for (size_t i = 0; i < tal_count(pptlv->fields); i++) {
+ const struct tlv_field *f = &pptlv->fields[i];
+ if (is_tlv_signature_field(f))
+ continue;
+ if (next_marker(f->numtype) == omitted)
+ return true;
+ }
+ return false;
+}
+
struct creator {
size_t n_inv, n_included;
const struct tlv_invoice *inv;
@@ -102,17 +148,20 @@ struct tlv_payer_proof *make_unsigned_proof_(const tal_t *ctx,
/* BOLT-payer_proof #12:
* - otherwise, if the TLV type is not zero:
* - MUST append a *marker number* to `proof_omitted_tlvs`
- * - If the previous TLV type was included:
- * - The *marker number* is that previous tlv type,
- * plus one.
- * - Otherwise, if `proof_omitted_tlvs` is empty:
- * - The *marker number* is 1.
- * - Otherwise:
- * - The *marker number* is one greater than the last
- * `proof_omitted_tlvs` entry.
+ * - The *marker number* is `next_marker(prev)`, where
+ * `prev` is the previous included TLV type (if the
+ * previous non-signature TLV was included), the last
+ * `proof_omitted_tlvs` entry (otherwise), or 0 (if
+ * `proof_omitted_tlvs` is still empty).
+ *
+ * Using `next_marker` here keeps the writer and the reader in
+ * lockstep across the `239 -> 1000000000` jump. See
+ * https://github.com/lightning/bolts/pull/1295#discussion_r3286972971
*/
- if (f->numtype != 0)
- tal_arr_expand(&omitted, ++last_type);
+ if (f->numtype != 0) {
+ last_type = next_marker(last_type);
+ tal_arr_expand(&omitted, last_type);
+ }
}
/* Arg for next_field_create and add_merkle */
@@ -366,23 +415,27 @@ const char *check_payer_proof(const tal_t *ctx,
}
/* BOLT-payer_proof #12:
*...
- * - `proof_omitted_tlvs` is not one greater than:
- * - an included TLV number, or
- * - the previous `proof_omitted_tlvs` or 0 if it is the first
- * number.
+ * - `proof_omitted_tlvs[i]` is not `next_marker(prev)`, where
+ * `prev` is either the previous `proof_omitted_tlvs` entry
+ * (or 0 if `i` is 0), or the TLV number of any included
+ * non-signature field.
+ *
+ * Mirroring the writer-side `next_marker` use removes the
+ * 239 -> 1000000000 carve-out that previously lived only on the
+ * writer. See
+ * https://github.com/lightning/bolts/pull/1295#discussion_r3286972971
*/
if (i > 0)
prev_omitted = pptlv->proof_omitted_tlvs[i-1];
else
prev_omitted = 0;
- if (omitted != prev_omitted + 1) {
- /* O(n^2) but doesn't matter */
- if (!find_tlv_num(pptlv, omitted - 1)) {
- return tal_fmt(ctx, "proof_omitted_tlvs[%zi] is"
- " not one greater than the previous %"PRIu64" nor an included tlv entry",
- i, prev_omitted);
- }
+ if (omitted != next_marker(prev_omitted)
+ && !find_included_marker_predecessor(pptlv, omitted)) {
+ return tal_fmt(ctx, "proof_omitted_tlvs[%zi] = %"PRIu64
+ " is not next_marker(previous=%"PRIu64
+ ") nor next_marker of any included TLV",
+ i, omitted, prev_omitted);
}
}
### common/test/run-bolt12_proof_marker_jump.c
@@ -1,31 +1,29 @@
-/* Reproduce BOLT 12 spec contradiction PR #1295 in CLN's reader.
+/* End-to-end test of the 239 -> 1000000000 marker-jump scenario from
+ * BOLT 12 PR #1295 discussion r3286972971.
*
* Spec discussion:
* https://github.com/lightning/bolts/pull/1295#discussion_r3286972971
*
- * In `lightning/bolts#1295` (head db3eff3a54, file 12-offer-encoding.md):
+ * The writer rule (lines 1041-1049, head db3eff3a54) and the reader rule
+ * (lines 1065-1067) disagree on what happens after a `proof_omitted_tlvs`
+ * entry of 239: the writer skips to 1000000000, the reader as originally
+ * written rejected anything that was not `prev + 1`.
*
- * - Writer rule (lines 1041-1049) mandates that the marker number after a
- * `proof_omitted_tlvs` entry of 239 is 1000000000 — i.e. the writer skips
- * past the signature/payer-proof range (TLV types 240..999999999).
- * - Reader rule (lines 1065-1067) only accepts the next entry if it is
- * `prev + 1` or `<included TLV> + 1`. No carve-out for the 239 → 1000000000
- * jump.
+ * This test exercises the fix on the CLN side, where both the writer-side
+ * marker emission and the reader-side validation share a single
+ * `next_marker(prev)` helper (see common/bolt12_proof.c). The test:
*
- * A reader applying the spec literally rejects a marker sequence the writer
- * rule mandates.
+ * 1. Builds an invoice with the usual required fields PLUS 64 unknown
+ * TLVs at types 1000000000..1000000063 (legal payload range).
+ * 2. Builds a `payer_proof` that includes only the required fields,
+ * omitting offer_issuer_id, the other optional invoice fields, and
+ * all 64 dummies. The marker sequence ends with [..., 239, 1000000000]
+ * because next_marker(239) == 1000000000.
+ * 3. Calls `check_payer_proof` and asserts it returns NULL, then sanity-
+ * checks that the produced marker sequence really did contain the
+ * 239 -> 1000000000 jump.
*
- * CLN's reader is a literal transcription of the bogus reader rule; see
- * `check_payer_proof` in common/bolt12_proof.c lines 339-388 (specifically the
- * `omitted != prev_omitted + 1` branch at line 379). Feeding it the
- * writer-rule-mandated sequence `[1, 2, …, 239, 1000000000]` produces an error
- * naming `1000000000` as "not one greater than the previous 239 nor an
- * included tlv entry" — that's the contradiction.
- *
- * This test deliberately FAILS until reader rule 1065 is amended to allow the
- * 239 → 1000000000 jump. The failure IS the reproduction.
- *
- * Parallel LDK reproduction lives in lightningdevkit/rust-lightning#4297, test
+ * Parallel LDK reproduction: lightningdevkit/rust-lightning#4297, test
* `spec_writer_reader_rules_contradict_on_gap_jump` in
* lightning/src/offers/merkle.rs (commit 0034c79ae).
*/
@@ -62,7 +60,8 @@ const struct siphash_seed *siphash_seed(void)
{ fprintf(stderr, "siphash_seed called!\n"); abort(); }
/* AUTOGENERATED MOCKS END */
-/* AAAA... BBBB... etc */
+#define N_DUMMIES 64
+
static struct pubkey *pubkey_for_letter(const tal_t *ctx, char letter)
{
struct secret secret;
@@ -74,58 +73,179 @@ static struct pubkey *pubkey_for_letter(const tal_t *ctx, char letter)
return pk;
}
+static secp256k1_keypair keypair_for_letter(char letter)
+{
+ struct secret secret;
+ secp256k1_keypair kp;
+
+ memset(&secret, letter, sizeof(secret));
+
+ if (secp256k1_keypair_create(secp256k1_ctx, &kp,
+ secret.data) != 1)
+ abort();
+ return kp;
+}
+
+static struct bip340sig *invoice_signature(const tal_t *ctx, struct tlv_invoice *inv, char letter)
+{
+ struct sha256 merkle, sha;
+ struct bip340sig *sig;
+ secp256k1_keypair kp = keypair_for_letter(letter);
+
+ tlv_update_fields(inv, tlv_invoice, &inv->fields);
+ merkle_tlv(inv->fields, &merkle);
+ inv->signature = tal(inv, struct bip340sig);
+ sighash_from_merkle("invoice", "signature", &merkle, &sha);
+
+ sig = tal(ctx, struct bip340sig);
+ assert(secp256k1_schnorrsig_sign32(secp256k1_ctx, sig->u8,
+ sha.u.u8,
+ &kp,
+ NULL) == 1);
+ return sig;
+}
+
+/* Include only the three required fields; omit everything else. Combined
+ * with the 64 dummy TLVs we inject in main(), the resulting marker sequence
+ * has the 239 -> 1000000000 jump near the end. */
+static bool include_required_only(const struct tlv_field *f, void *arg UNNEEDED)
+{
+ if (f->numtype == 0)
+ return false;
+ return f->numtype == TLV_INVOICE_INVREQ_PAYER_ID
+ || f->numtype == TLV_INVOICE_INVOICE_PAYMENT_HASH
+ || f->numtype == TLV_INVOICE_INVOICE_NODE_ID;
+}
+
+static bool sign(const char *messagename,
+ const char *fieldname,
+ const struct sha256 *msg,
+ struct bip340sig *sig,
+ secp256k1_keypair *kp)
+{
+ struct sha256 shash;
+
+ sighash_from_merkle(messagename, fieldname, msg, &shash);
+ return secp256k1_schnorrsig_sign32(secp256k1_ctx, sig->u8,
+ shash.u.u8,
+ kp,
+ NULL) == 1;
+}
+
int main(int argc, char *argv[])
{
- struct tlv_payer_proof *pptlv;
+ struct tlv_invoice *inv;
struct preimage preimage;
- bigsize_t *omitted;
+ struct tlv_payer_proof *proof;
const char *err;
+ secp256k1_keypair kp;
+ bool saw_239 = false, saw_1e9_after_239 = false;
+ size_t n_omitted;
common_setup(argv[0]);
- /* Construct just enough of a payer_proof for `check_payer_proof` to
- * reach the proof_omitted_tlvs validation loop (lines 339-388 of
- * common/bolt12_proof.c). All eight required-field pointers must be
- * non-NULL, and sha256(proof_preimage) must equal invoice_payment_hash.
- * We deliberately leave pptlv->fields as the empty array from
- * tlv_payer_proof_new, so `find_tlv_num` always returns false and we
- * isolate the gap-jump bug from the "is included field" check.
- */
- pptlv = tlv_payer_proof_new(tmpctx);
+ memset(&preimage, 0x1, sizeof(preimage));
- pptlv->invreq_payer_id = pubkey_for_letter(pptlv, 'A');
- pptlv->invoice_node_id = pubkey_for_letter(pptlv, 'B');
+ /* Standard minimal invoice (same skeleton as run-bolt12_proof.c). */
+ inv = tlv_invoice_new(tmpctx);
+ inv->invreq_metadata = tal_arrz(inv, u8, 16);
+ inv->offer_issuer_id = pubkey_for_letter(inv, 'A');
- memset(&preimage, 0x1, sizeof(preimage));
- pptlv->proof_preimage = tal_dup(pptlv, struct preimage, &preimage);
- pptlv->invoice_payment_hash = tal(pptlv, struct sha256);
- sha256(pptlv->invoice_payment_hash, &preimage, sizeof(preimage));
-
- /* Stub signatures: bytes are irrelevant since the marker-loop check
- * runs before any signature verification. */
- pptlv->signature = talz(pptlv, struct bip340sig);
- pptlv->proof_signature = talz(pptlv, struct bip340sig);
- pptlv->proof_missing_hashes = tal_arr(pptlv, struct sha256, 0);
- pptlv->proof_leaf_hashes = tal_arr(pptlv, struct sha256, 0);
-
- /* The marker sequence the spec writer rule mandates for 240 consecutive
- * omitted non-signature TLVs:
- * [1, 2, 3, …, 239, 1000000000]
- * The 240th marker jumps to 1000000000 because TLV types 240..999999999
- * are the signature/payer-proof range. */
- omitted = tal_arr(pptlv, bigsize_t, 240);
- for (size_t i = 0; i < 239; i++)
- omitted[i] = i + 1;
- omitted[239] = 1000000000;
- pptlv->proof_omitted_tlvs = omitted;
-
- err = check_payer_proof(tmpctx, pptlv);
+ inv->invreq_amount = tal(inv, u64);
+ *inv->invreq_amount = 1;
+ inv->invreq_payer_id = pubkey_for_letter(inv, 'B');
+
+ inv->invoice_paths = tal_arr(inv, struct blinded_path *, 1);
+ inv->invoice_paths[0] = tal(inv->invoice_paths, struct blinded_path);
+ sciddir_or_pubkey_from_pubkey(&inv->invoice_paths[0]->first_node_id,
+ pubkey_for_letter(tmpctx, 'C'));
+ inv->invoice_paths[0]->first_path_key = *pubkey_for_letter(tmpctx, 'D');
+ inv->invoice_paths[0]->path = tal_arr(inv->invoice_paths[0], struct blinded_path_hop *, 1);
+ inv->invoice_paths[0]->path[0] = tal(inv->invoice_paths[0]->path,
+ struct blinded_path_hop);
+ inv->invoice_paths[0]->path[0]->blinded_node_id = *pubkey_for_letter(tmpctx, 'E');
+ inv->invoice_paths[0]->path[0]->encrypted_recipient_data = tal_arrz(inv->invoice_paths[0]->path[0], u8, 16);
+ inv->invoice_blindedpay = tal_arr(inv, struct blinded_payinfo *, 1);
+ inv->invoice_blindedpay[0] = tal(inv->invoice_blindedpay, struct blinded_payinfo);
+ inv->invoice_blindedpay[0]->fee_base_msat = 1;
+ inv->invoice_blindedpay[0]->fee_proportional_millionths = 2;
+ inv->invoice_blindedpay[0]->cltv_expiry_delta = 3;
+ inv->invoice_blindedpay[0]->htlc_minimum_msat = AMOUNT_MSAT(4);
+ inv->invoice_blindedpay[0]->htlc_maximum_msat = AMOUNT_MSAT(5);
+ inv->invoice_blindedpay[0]->features = NULL;
+
+ inv->invoice_created_at = tal(inv, u64);
+ *inv->invoice_created_at = 1733458312;
+ inv->invoice_payment_hash = tal(inv, struct sha256);
+ sha256(inv->invoice_payment_hash, &preimage, sizeof(preimage));
+ inv->invoice_amount = tal(inv, u64);
+ *inv->invoice_amount = 1;
+ inv->invoice_node_id = pubkey_for_letter(inv, 'F');
+
+ /* Inject 64 dummy unknown TLVs at *odd* types in the extension range
+ * (1000000001, 1000000003, ...). Unknown even TLVs are rejected by
+ * BOLT-12's general TLV rule, so we keep them odd to be ignored on
+ * decode while still appearing in inv->fields and counting toward the
+ * marker emission. These get omitted in the proof; combined with the
+ * included required fields, the writer-side marker emission walks
+ * through the 1..239 range and then jumps to 1000000000 on the 64th
+ * dummy. */
+ for (size_t i = 0; i < N_DUMMIES; i++) {
+ struct tlv_field f;
+ f.meta = NULL;
+ f.numtype = 1000000001 + 2 * i;
+ f.length = 1;
+ f.value = tal_arrz(inv, u8, 1);
+ f.value[0] = (u8)(i + 1);
+ tal_arr_expand(&inv->fields, f);
+ }
+
+ inv->signature = invoice_signature(inv, inv, 'F');
+
+ /* We skip the usual encode/decode round-trip because invoice_decode
+ * enforces invoice-level invariants (signature check, required-field
+ * check) that are not what this test is exercising. inv->fields is
+ * already canonicalised by invoice_signature's call to
+ * tlv_update_fields. */
+
+ /* Build proof: include the three required fields, omit everything
+ * else. The 64 trailing omitted dummies push the marker counter past
+ * 239, triggering the gap-jump emission. */
+ proof = make_unsigned_proof(tmpctx, inv, &preimage, "marker-jump",
+ include_required_only, NULL);
+ kp = keypair_for_letter('B');
+ proof->proof_signature = payer_proof_signature(proof, proof, sign, &kp);
+
+ /* The fix under test: with the next_marker helper used on both writer
+ * and reader sides, the produced sequence ends with [..., 239,
+ * 1000000000] and check_payer_proof accepts it. */
+ err = check_payer_proof(tmpctx, proof);
if (err)
- fprintf(stderr,
- "check_payer_proof rejected the writer-rule-mandated"
- " 239 → 1000000000 jump: %s\n", err);
+ fprintf(stderr, "check_payer_proof returned: %s\n", err);
assert(err == NULL);
+ /* Sanity-check that the marker sequence actually did contain the
+ * 239 -> 1000000000 jump we were trying to exercise. Without this
+ * check the test could go green for the wrong reason. */
+ n_omitted = tal_count(proof->proof_omitted_tlvs);
+ for (size_t i = 0; i < n_omitted; i++) {
+ bigsize_t m = proof->proof_omitted_tlvs[i];
+ if (m == 239)
+ saw_239 = true;
+ if (m == 1000000000 && i > 0
+ && proof->proof_omitted_tlvs[i-1] == 239)
+ saw_1e9_after_239 = true;
+ }
+ fprintf(stderr, "marker sequence (%zu entries):", n_omitted);
+ for (size_t i = 0; i < n_omitted; i++)
+ fprintf(stderr, " %"PRIu64, proof->proof_omitted_tlvs[i]);
+ fprintf(stderr, "\n");
+ fprintf(stderr, "saw 239: %s, saw 1000000000 immediately after: %s\n",
+ saw_239 ? "yes" : "no",
+ saw_1e9_after_239 ? "yes" : "no");
+ assert(saw_239);
+ assert(saw_1e9_after_239);
+
common_shutdown();
return 0;
}Why this scored 60/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.