Faster tests

Three things make a test job finish sooner on SuperCI. Try them in this order.

1. A bigger machine, with every CPU in use

runs-on: superci-16cpu

A machine with four times the CPUs costs four times as much per minute and finishes in about a quarter of the time, so the job costs about the same and ends sooner. One suite we measured on AWS (1,596 Playwright tests in a real browser, one worker per CPU):

This only works when the test tool uses every CPU. Not all do by default:

On the suite above, Playwright’s default left a third of the speed unused.

Some suites cannot use more workers: their tests share one database or one running app, and fail when run side by side. For those, go to the next step.

2. A matrix, when tests cannot share a machine

GitHub’s matrix runs the same job several times at once. Each copy gets its own machine, so each has its own database and its own app. The test tool’s shard flag gives each copy its part of the suite:

jobs:
  test:
    runs-on: superci
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v5
      - run: npm ci
      - run: npx playwright test --shard=${{ matrix.shard }}/4

Playwright, Jest and Vitest all take --shard=1/4.

Every copy repeats the job’s setup, so a matrix pays off when the tests take longer than the setup. For tests that can run side by side it is no faster than one bigger machine: on the suite above, eight 4-CPU machines and one 32-CPU machine both finished in 66 seconds.

Many machines at once can also go over your AWS account’s spot limit. See Spot machines.

3. Less setup

On a short job, setup is most of the time. Two things we measured on AWS:

• Playwright without --with-deps. SuperCI’s AWS machines have the tools of GitHub’s own Ubuntu runners, and with them the system packages browsers need. npx playwright install chromium took 4 seconds; with --with-deps it took 17.
• A cache only for what is slow to make. actions/cache works on SuperCI’s machines as it does on GitHub’s. It pays off for built dependencies and compiled code. It does not for things that download in seconds: restoring the browser from the cache took 3 seconds, downloading it 4.

Docker images are slow either way (pulling Playwright’s took 26 seconds). Pull fewer, or smaller ones.

What does not help

A job’s machine takes about 15 seconds to start on AWS and about 7 on Cloudflare, whatever its size. A job whose tests take 10 seconds will not get much faster with more CPUs; send it to the provider that starts sooner with runs-on: superci-cloudflare.