The CI Matrix

The CI Matrix#

Serena’s CI has a problem most projects never meet: sixty-odd languages, each with its own toolchain, and no runner that could hold them all at once. The answer is a matrix with opinions — and the opinions are written down, in the header of .github/workflows/pytest.yml. This page walks through them; when CI surprises you, that header is the primary source.

Five batches, three operating systems#

Tests run as parallel matrix jobs over (OS × batch). The five batches are marker-based — jvm, native, other-langs, niche, and a catch-all for everything unmarked — and the batching is deliberately by pytest marker, not pytest-xdist: tests share fixtures, language-server processes and on-disk resources, so in-process parallelism would race them.

Two properties keep the partition honest: each batch installs only the toolchains its languages need, and catch-all is defined as the negation of all the named groups — every test lands in exactly one batch, and adding a language cannot desync the split.

Where the matrix thins out#

The default is the full cross-product, with two kinds of exception:

  • niche runs on ubuntu only. Its toolchains (lean4, ocaml, nix, julia, R, perl) are heavy and slow — not worth the Windows and macOS cost.

  • An OS-specific language inside an all-OS batch (swift on macOS, haskell on Linux) is skipped one level down, by the central guard in test/conftest.py — never by the matrix.

That split matters when you add a language: OS gating belongs in the conftest, not in the workflow file.

The ceilings#

GitHub’s free tier caps runs at 20 concurrent jobs and 5 concurrent macOS jobs. The current matrix is 13 jobs, 4 of them macOS — inside both limits, and anything that adds batches or operating systems has to stay inside them too, or jobs queue. Superseded runs on the same ref are cancelled outright to free capacity, and every job carries a 60-minute timeout: a healthy warm-cache run peaks around 16 minutes, and the ceiling exists because a hung job once burned the full six hours.

Caching#

Language servers are cached per (OS × batch); the uv venv is keyed on uv.lock; several individual toolchains and the web fixtures’ npm state have caches of their own. Most of the niche toolchains restore from cache, which is why that batch stays fast despite its size.

What this means for your pull request#

  • poe lint and poe type-check run once per OS, in the catch-all batch — the format gate is the first thing to fail, in the first minute.

  • Your language’s tests run in whichever batch its marker belongs to; nothing to configure.

  • A red run is usually one red job. Read that job’s log before concluding anything about the other twelve.

  • The docs build (Sphinx, warnings as errors) and a spell check run alongside the tests.