# Spot machines

On AWS, jobs run on spot machines unless you say otherwise. Spot machines are AWS's spare capacity, at a much lower price. The price of that is that AWS can take one back with two minutes' notice.

## When AWS takes a machine back

1. The machine hears AWS's notice, two minutes before it goes, and lets the job go on. A job that ends by itself in that time passes, and nothing more happens.
2. Twenty seconds before the machine goes, a job still running has its runner stopped, so it fails at once instead of hanging.
3. The dashboard shows the job as interrupted.
4. When the job's run has finished, SuperCI runs the job again. GitHub will not restart one job while its run is still going, so there is no earlier moment. A run that passed without it (the job had `continue-on-error`) is left as it is.
5. The new attempt does not go to a spot machine: it goes to the next provider in your [order](/docs/order-and-limits) below AWS's spot machines, which is AWS on-demand unless you changed that.

On GitLab the job is retried the same way.

If AWS takes back two machines within a quarter of an hour, new jobs skip spot machines for half an hour.

The pool that took a machine back (a machine type in a zone) is asked last for a quarter of an hour, so the next jobs do not follow it there.

### Matrix jobs

A matrix has `fail-fast` on unless you turn it off: one job failing cancels the others, and the re-run then repeats all of them. With machines that can be taken back, set it off, so an interruption costs one job:

```yaml
strategy:
  fail-fast: false
  matrix:
    shard: [1, 2, 3, 4]
```

### Many jobs at once

AWS lets an account ask for only so many machines in a moment, and a new account for few. When a spot request is refused for that, SuperCI asks again after a short wait. One still refused after its waits starts on an on-demand machine rather than keep the job waiting, and the job says so: "On-demand: AWS refused the spot request for its rate".

When spot machines are the only place a job can run (on-demand is off and no other provider fits), it runs on spot again rather than not at all.

The same when the new attempt does not fit your account's on-demand limit: a 32-CPU job in an account that may run 16 on-demand CPUs goes back on a spot machine, in another zone or of another type first, and its row says so. And a job with no spot machine to be had now, whose on-demand machine that limit refuses, waits for a spot machine rather than fail.

## When AWS has no spot machine

Several machine types fit a job, each in several zones: each pair is a pool, with its own price and its own spare machines. AMD's, Intel's and, for arm64, Graviton's families of the size asked for are all considered; one with more memory than asked (r8a, say) only where no other has the memory with as many CPUs. SuperCI asks the pool where a job costs least first: its spot price for the speed it gives (an Intel CPU, half a core, counts a quarter slower). If it has none, the others are asked in one request, which takes about three seconds however many they are: AWS gives a machine from the pool with the most to spare, so the one least likely to be taken back. Only the pools within a quarter of the best price for a job are asked that way first, so a machine at three times the price is never taken while a cheaper one has room; the others come after. No spot pool priced over the on-demand price of the cheapest machine that fits is asked at all.

With several regions, SuperCI reads each one's spot prices and asks first, region by region, only the pools within a quarter of the cheapest in any of them: the first region in your list counts 15% cheaper, since its cache and Docker layers are there, and a pool that just had no machine waits its turn. So when N. Virginia has no c8a left, Ohio's c8a is asked before N. Virginia's r8a at three times the price. Then each region is asked whole, where what is left costs least. Every pool is asked before the job goes anywhere else. A pool that had no machine within the last minute is not asked again, by that job or by one queued beside it. If none has a spot machine for the job, the job goes to the next provider in your order. With nothing else that can run it, it waits and keeps trying.

`superci jobs retrieve ID` shows what was asked and what AWS answered, and a job's row says when its machine was not the first choice.

Your AWS account also has a limit on how many spot CPUs it runs at once in a region. Many jobs at once can go over it, and the ones over it are treated the same way: with on-demand next in your order, they run on-demand, at about twice the price. You can [ask AWS for a higher limit](https://console.aws.amazon.com/servicequotas/home/services/ec2/quotas/L-34B43A08).

On-demand machines have a limit of their own. When both limits are in use by your own jobs, the next job waits for one of them to end, as it would on GitHub's runners. When a job asks for more CPUs than the limits allow at all, it fails saying which limits to raise.

## Jobs that must not be interrupted

A job that should never run twice, like a deploy, can ask for an on-demand machine in its label:

```yaml
runs-on: superci/spot=false
```

To keep every job off spot machines, tick **No spot machines** on the default machine under **Workflows**, or run `superci machine update --spot=false`.

## What it costs

A spot machine's price is its zone's spot price while it ran; an on-demand machine's is AWS's list price. The dashboard estimates a job's cost while it runs and settles it at the prices AWS billed once it has ended.
