CI: reduce parallelism for pytest.
What the developer wrote
Authored by Rusty Russell
CI: reduce parallelism for pytest.
In November 2022 we seemed to increase parallelism from 2 and 3 to 10!
That is a huge load for these CI boxes, and does explain some of our
flakes.
We only run in parallel because some tests sleep, but it's diminishing
returns (GH runners have 4 VCPUs, 16GB RAM).
This reduces it so:
- Normal runs are -n 4
- Valgrind runs are -n 2
- Sanitizer runs are -n 3
If I use my beefy build box (64BG RAM) but reduce it to 4 CPUs:
Time for pytest -n 5:
Time for pytest -n 4:
Time for pytest -n 3:
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
What changed, and why it matters
This commit only changes how many tests run at the same time in the project's automated CI (continuous integration) system. It lowers the number of parallel test jobs from 10 to 4 in most cases, and sets specific values for Valgrind and sanitizer test runs. There is no change to the actual Core Lightning software, its network protocol, wallet handling, or any user-facing behavior. It is purely an infrastructure/reliability tweak to reduce flaky test failures caused by overloaded CI machines.
No security action needed. This is a CI configuration change. Reviewers may optionally verify that the reduced parallelism actually improves CI reliability and does not significantly slow feedback times.
Security signals we found
No strong security signals were identified.
Evidence from the diff
The patch modifies .github/workflows/ci.yaml, adjusting PYTEST_PAR environment variables and pytest -n arguments. It reduces parallelism for normal CI runs from 10 to 4, Valgrind runs from a hardcoded 3 to an env-controlled 2, and sanitizer runs from a hardcoded 2 to an env-controlled 2 (despite the commit message saying sanitizer runs are -n 3, the diff sets PYTEST_PAR: 2 for that job). No application code, cryptography, networking, database, or RPC logic is touched.
Changed components
.github/workflows/ci.yamlInspect captured patch +8 / −5
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
index 60884992..7dda6172 100644
--- a/.github/workflows/ci.yaml
+++ b/.github/workflows/ci.yaml
@@ -312,7 +312,7 @@ jobs:
- name: Test
env:
SLOW_MACHINE: 1
- PYTEST_PAR: 10
+ PYTEST_PAR: 4
TEST_DEBUG: 1
TEST_DB_PROVIDER: ${{ matrix.TEST_DB_PROVIDER }}
TEST_NETWORK: ${{ matrix.TEST_NETWORK }}
@@ -430,7 +430,7 @@ jobs:
COMPAT: 1
CFG: ${{ matrix.CFG }}
SLOW_MACHINE: 1
- PYTEST_PAR: 10
+ PYTEST_PAR: 4
TEST_DEBUG: 1
TEST_DB_PROVIDER: ${{ matrix.TEST_DB_PROVIDER }}
TEST_NETWORK: ${{ matrix.TEST_NETWORK }}
@@ -524,8 +524,9 @@ jobs:
env:
SLOW_MACHINE: 1
TEST_DEBUG: 1
+ PYTEST_PAR: 2
run: |
- VALGRIND=1 uv run eatmydata pytest tests/ -n 3 ${PYTEST_OPTS} ${{ matrix.PYTEST_OPTS }}
+ VALGRIND=1 uv run eatmydata pytest tests/ -n ${PYTEST_PAR} ${PYTEST_OPTS} ${{ matrix.PYTEST_OPTS }}
- name: Upload test results
if: always()
uses: actions/upload-artifact@v4
@@ -613,8 +614,10 @@ jobs:
run: tar -xvjf cln-compile-clang-sanitizers.tar.bz2
- name: Test
+ env:
+ PYTEST_PAR: 2
run: |
- uv run eatmydata pytest tests/ -n 2 ${PYTEST_OPTS} ${{ matrix.PYTEST_OPTS }}
+ uv run eatmydata pytest tests/ -n ${PYTEST_PAR} ${PYTEST_OPTS} ${{ matrix.PYTEST_OPTS }}
- name: Upload test results
if: always()
uses: actions/upload-artifact@v4
@@ -737,7 +740,7 @@ jobs:
COMPAT: 1
CFG: ${{ matrix.CFG }}
SLOW_MACHINE: 1
- PYTEST_PAR: 10
+ PYTEST_PAR: 4
TEST_DEBUG: 1
TEST_DB_PROVIDER: ${{ matrix.TEST_DB_PROVIDER }}
TEST_NETWORK: ${{ matrix.TEST_NETWORK }}
Why this scored 15/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.