tests: allow running a sample of test cases, enable for qemu
What changed, and why it matters
This commit only changes how automated tests are run. It adds an option to run a random 33% sample of test cases when testing in a slow emulator (QEMU), in order to save CI time. It does not change any wallet, firmware, or cryptographic code that end users interact with.
No security action required. This is a test-optimization change. If desired, the project can document the sampling strategy and ensure the 33% sample still provides adequate regression coverage over time.
Security signals we found
No strong security signals were identified.
Evidence from the diff
The patch modifies CI/test scripts to introduce a --sample-percent argument to test_jade.py. When enabled, only a random subset of JSON test-case files are loaded, unless sampling is disabled for a specific test group (e.g., multisig registration, because later tests depend on it). The default remains 100%, and QEMU CI is switched to 33%. No production, cryptographic, or protocol logic is altered.
Changed components
CI/test configuration: gitlab/flash.ymlCI/test helper script: main/qemu/qemu_ci_flash.shTest runner: test_jade.pyInspect captured patch +23 / −5
diff --git a/gitlab/flash.yml b/gitlab/flash.yml
index 1d0f83c..1d7a674 100644
--- a/gitlab/flash.yml
+++ b/gitlab/flash.yml
@@ -14,7 +14,7 @@
script:
- mv $(echo $CI_JOB_NAME | sed "s/^${CI_JOB_STAGE}/build_test/") build
- ./main/qemu/make-flash-img.sh
- - ./main/qemu/qemu_ci_flash.sh
+ - ./main/qemu/qemu_ci_flash.sh --sample-percent=33
flash_qemu:
extends: .flash_qemu_template
diff --git a/main/qemu/qemu_ci_flash.sh b/main/qemu/qemu_ci_flash.sh
index 836a62b..8f4f9a4 100755
--- a/main/qemu/qemu_ci_flash.sh
+++ b/main/qemu/qemu_ci_flash.sh
@@ -50,4 +50,4 @@ cp "${FW_FULL}.hash" "${FW_PATCH}.hash"
python jade_ota.py --log=INFO --skipble --serialport=tcp:localhost:30121 --fwfile=${FW_PATCH}
# Run the tests - long timeout for bcur-fragment iteration test in 'run_remote_selfcheck()/selfcheck.c'
-python test_jade.py --log=INFO --skipble --qemu --serialport=tcp:localhost:30121 --serialtimeout=900
+python test_jade.py --log=INFO --skipble --qemu --serialport=tcp:localhost:30121 --serialtimeout=900 $*
diff --git a/test_jade.py b/test_jade.py
index b91546d..17970c2 100644
--- a/test_jade.py
+++ b/test_jade.py
@@ -8,6 +8,7 @@ import json
import base64
import random
import logging
+import math
import argparse
import subprocess
import threading
@@ -160,8 +161,16 @@ def _read_json_file(filename):
# Helper to read json test files into a list
-def _get_test_cases(pattern):
- return (_h2b_test_case(_read_json_file(f)) for f in glob.glob('./test_data/' + pattern))
+def _get_test_cases(pattern, allow_sampling=True):
+ filenames = [f for f in glob.glob('./test_data/' + pattern)]
+ if allow_sampling and filenames and args.sample_percent != 100:
+ # Test only args.sample_percent percentage of the files, but
+ # test all files if there are only a small number of them
+ num_files = len(filenames)
+ if num_files > 8:
+ num_files = int(math.ceil((args.sample_percent / 100.0) * num_files))
+ filenames = random.sample(filenames, num_files)
+ return (_h2b_test_case(_read_json_file(f)) for f in filenames)
BLE_TEST_PASSKEYFILE = 'ble_test_passkey.txt'
@@ -3205,7 +3214,8 @@ def _check_multisig_registration(jadeapi, multisig_data):
def test_generic_multisig_registration(jadeapi):
# Generic multisig - check register multisig wallets and get receive addresses
- for multisig_data in _get_test_cases(MULTI_REG_TESTS):
+ # Run all of these tests since later test cases rely on them :(
+ for multisig_data in _get_test_cases(MULTI_REG_TESTS, allow_sampling=False):
_check_multisig_registration(jadeapi, multisig_data)
# Ensure the 1of1 is registered at the end - same name will be used to overwrite
@@ -4300,6 +4310,12 @@ if __name__ == '__main__':
dest='no_legacy_flow',
help='Do not use the legacy sign_tx flow (use the AE flow instead)',
default=False)
+ parser.add_argument("--sample-percent",
+ action="store",
+ dest="sample_percent",
+ type=int,
+ help="Run only a random sample of test cases",
+ default=100)
parser.add_argument('--log',
action='store',
dest='loglevel',
@@ -4311,6 +4327,8 @@ if __name__ == '__main__':
jadehandler.setLevel(getattr(logging, args.loglevel))
logger.debug(f'args: {args}')
+ if args.sample_percent != 100:
+ logger.warning(f'WARNING: Testing reduced test cases ({args.sample_percent}% sample)')
args.spts = args.serialport and not args.serialport.startswith('/dev/tty/') and not args.qemu
manage_agents = args.agentkeyfile and not args.skipble and \
Why this scored 15/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.