test(clear_signing): name the skipped cases
What changed, and why it matters
This commit only improves a test report for Trezor's clear-signing feature. It changes how skipped test cases are listed in a nightly summary, showing each skipped case's source file and calldata size instead of just counting them by reason. There is no change to firmware, wallet logic, cryptography, or any user-facing security behavior.
No security action needed. This is a test-reporting-only change and can be treated as routine QA tooling.
Security signals we found
No strong security signals were identified.
Evidence from the diff
The diff modifies two test-only files: tests/clear_signing_tests/nightly.py and tests/clear_signing_tests/summary.md.mako. It adds a helper to format per-case skip details and updates the Mako template to render a list of skipped cases grouped by reason rather than a count table. No production code, no device firmware, no crypto, no parsing of untrusted input, and no privilege changes are involved.
Changed components
tests/clear_signing_tests/nightly.pytests/clear_signing_tests/summary.md.makoInspect captured patch +25 / −9
### tests/clear_signing_tests/nightly.py
@@ -38,7 +38,12 @@
import click
import requests
from mako.template import Template
-from registry import definitions_source, load_cases, read_results
+from registry import (
+ RegistryCase,
+ definitions_source,
+ load_cases,
+ read_results,
+)
from trezorlib import definitions
@@ -334,6 +339,14 @@ def fetch_baseline(
MAX_LINES = 300
+def _skipped_line(case: RegistryCase) -> str:
+ """A skipped case, with the registry test file (descriptor) it comes from."""
+ line = f"- `{case.test_id}` in `{case.test_file}`"
+ if case.tx is not None:
+ line += f": {len(case.tx.data)} bytes of calldata"
+ return line
+
+
@dataclass
class _ModelRow:
model: str
@@ -430,13 +443,17 @@ def summarize(
_ModelRow(model, counts, report_url, differing_url, baseline.get(model))
)
- skipped = Counter(case.skip_reason for case in cases if case.skip_reason)
+ # skip reason -> cases
+ skipped: dict[str, list[str]] = defaultdict(list)
+ for case in cases:
+ if case.skip_reason:
+ skipped[case.skip_reason].append(_skipped_line(case))
summary = Template(filename=str(SUMMARY_TEMPLATE), strict_undefined=True).render(
inputs=inputs,
total=len(cases),
run=len(to_run),
rows=rows,
- skipped=skipped.most_common(),
+ skipped=sorted(skipped.items(), key=lambda item: -len(item[1])),
problems=sorted(problems.items()),
blind=sorted(blind.items()),
no_format=sorted(no_format.items()),
### tests/clear_signing_tests/summary.md.mako
@@ -43,14 +43,13 @@ ${total} mainnet cases of all registry providers: ${run} run on each model, ${to
The UI report compares the screens of each model with those of the last nightly run (`fixtures.json` when there is none). There is nothing to accept: the screens of this run are what the next one compares with.
-<details><summary>Skipped cases (${sum(count for _, count in skipped)})</summary>
+<details><summary>Skipped cases (${sum(len(lines) for _, lines in skipped)})</summary>
-| Reason | Cases |
-|---|---:|
-% for reason, count in skipped:
-| ${reason} | ${count} |
-% endfor
+% for reason, lines in skipped:
+**${reason}** (${len(lines)})
+${capped(lines)}
+% endfor
</details>
<details><summary>Failed or not run (${len(problems)})</summary>Why this scored 15/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.