# Order and limits

With more than one provider, **Runners** shows them as a list. Drag to reorder. The list is the rule for where jobs go.

## How a job is placed

A job goes to the first provider in the list that can run its machine and has room. It moves on to the next when:

- the provider cannot run that machine (too large, no GPUs, no Windows);
- the provider is at its limit of jobs at once, or over its monthly budget;
- the provider cannot start the machine now: AWS has no spot machine for it, or starting it failed.

When every provider that could run it is full, the job waits and is tried again as room appears. When none can run it at all, it fails at once on GitHub with the reason.

## AWS on-demand is its own row

AWS appears twice in the list: **AWS** is its spot machines, and **AWS on-demand** its on-demand ones. On-demand starts right under AWS, so a job that finds no spot machine gets an on-demand one.

You decide what happens instead:

- **Move it.** Put Cloudflare between the two, and a job with no spot machine goes to Cloudflare first.
- **Cap it.** Give on-demand its own monthly budget.
- **Turn it off.** Then a job that only AWS can run waits for a spot machine.

## Limits

Each provider's settings have two limits:

| Limit | What it does | Default |
|---|---|---|
| Jobs at once | When this many run there, the next job goes to the next provider or waits. | 20 |
| Monthly budget | When this month's estimated spend reaches it, jobs go to the next provider. | none |

Spend is estimated the way the dashboard shows it, and includes the time a runner waits for its job. A budget on **AWS** counts its spot machines; **AWS on-demand** has its own.

## Limits for a month and for a day

Below the providers, **Limits** has two more rows, each with what was spent this month and today:

| Row | What it counts | At a limit |
|---|---|---|
| Everything | All providers together. | No job starts until the month (or the day) is over, or until you raise it. |
| Each account | Its runners together: AWS's spot and on-demand machines, or [each further account's](/docs/several-accounts). | Jobs go to the providers of other accounts in your order. With none, they fail saying so. |

Each row takes a monthly limit and a daily one. A daily limit catches a day that runs away: without one, a month's limit can be spent in an afternoon. Days are UTC. Nothing is set until you set it. In a terminal:

```sh
superci limits update --monthly-usd=500 --daily-usd=50
superci accounts update aws --monthly-usd=300 --daily-usd=30
```

`none` lifts one. A job refused at a limit fails on GitHub with the reason, so nobody waits for a runner that will not come.

Three things to know:

- A limit counts the machines jobs ran on, as SuperCI estimates them. Your cloud's bill also has storage and small fees.
- Jobs already running when a limit is reached finish.
- The runner that tells GitHub why a job was refused is a small machine for under a minute.

## A budget in AWS itself

SuperCI's limits stop a job before its machine starts, but they count SuperCI's own estimate. AWS knows the real bill. In the AWS account's limit dialog, **In AWS itself** adds a budget in your AWS account for the same monthly limit:

| Choice | What AWS does |
|---|---|
| Nothing | Nothing. This is how it starts. |
| AWS emails me at 80% and 100% | Writes to the address you give when the account's real bill passes 80% and 100% of the limit. |
| AWS emails me, and stops new machines at 100% | That, and at 100% it refuses new machines to SuperCI's control plane until the next month begins. |

```sh
superci accounts update aws --monthly-usd=300 --aws-budget=emails --email=you@example.com
superci accounts update aws --aws-budget=stops
superci accounts update aws --aws-budget=lift     # takes the refusal off now
superci accounts update aws --aws-budget=none     # removes all of it
```

What to know:

- **It is made with your sign-in.** SuperCI's control plane gets no permission for it. A budget you made yourself is never touched.
- **AWS counts the whole account, and hours late.** Its figures arrive a few times a day, so it corrects SuperCI's limit rather than replacing it. In an account that other things run in, their spending counts too.
- **Credits do not hide spending.** The budget counts the bill before credits.
- **Changing the limit changes the budget with it.** Raising it after AWS stopped machines lifts the stop.
- **When AWS has stopped machines**, a job fails on GitHub saying so, and the AWS account's row shows "Stopped by AWS's budget" with **Lift**. The stop lets one small kind of machine through: the one that tells GitHub why a job was refused.
- **It costs nothing**: AWS charges nothing for budgets, nor for the first two with an action.

## Keeping a job on one provider

A job that moves to another provider runs in that provider's environment: a different image, and no access to a network of your own on AWS. If a workflow depends on one provider, name it in the label:

```yaml
runs-on: superci/provider=aws
```

That job runs on AWS only (spot, then on-demand), and waits or fails rather than going elsewhere.
