Runs and Scheduling

This page introduces what a run is, how one is created, how recurring runs are scheduled, and how runs relate to the agents that execute them.

What a run is

A run is one attempt at executing a selection of cases against a target, at a moment in time. It has three inputs:

A run is not the same as a case, and not the same as a suite:

• A case says what should happen. It is a definition.
• A suite is a selection of cases. It is a definition.
• A run is an execution of a selection against a target. It produces results, has a start time, an end time and a status.

The same case run twice against two targets produces two runs and two sets of results. That separation is what makes trend reporting possible.

A run records the target it addressed, so a result can always be read as "this capability, on this protocol, on this device family, with this data". The run itself is stored in test_runs, with id, name, suite_id, status, started_at, completed_at and created_at.

name: Nightly — CCC / Apple
suite: [regression]
project: proj.a
target:
  protocol: CCC
  device_type: apple
  oem: "*"
  agent: bench-01

Creating a run

Creating a run is a resolution step followed by an execution step. The resolution happens once, when the run is created, and its output is snapshotted into the run.

1. Select the capabilities — from a suite, or from an explicit list.
2. Resolve each capability to the variant matching the target profile. A capability with no matching variant is recorded as skipped, with the reason "no matching variant". It does not fail the run.
3. Bind the parameters: the project's values, overridden by the variant's own values.
4. Render each procedure, substituting the parameters into every step. A placeholder that cannot be resolved is an error for that execution — the rendering fails fast rather than sending $SIM_IMSI to a bench.
5. Snapshot the rendered steps and a hash of them into the run's executions, so that editing a case afterwards cannot rewrite history.
6. Dispatch the steps to the agent chosen for the target. Each step is sent as a command, and the studio writes the command to agent_commands with status pending before it goes out.
7. Collect the per-step results and write the execution outcomes back.

Step 5 is the one that matters for trust. A hash of the rendered procedure is recorded with every execution, so a report can show that a failure appeared at the same time as a script change — or that the script did not change and the failure is real.

Each execution is stored in executions, with run_id, case_id, status, result, duration_ms, error_message and executed_at.

execution:
  case: tc.provision
  variant: v.provision.ccc.apple
  target: bench-01
  script_hash: "9f2c…"
  status: pending

Scheduling

A recurring run is described by a schedule: a cron expression, the selection to run, the target, and whether it is enabled. The cron expression is stored on the schedule as cron_expression.

• A schedule fires at its cron times and creates a run from its stored selection and target.
• The schedule does not overlap with itself: if the previous run from the same schedule is still running, the new one is either skipped or queued, according to the schedule's policy.
• A schedule can be paused and resumed without deleting it.
• A run created by a schedule records the schedule it came from, so a history view can group runs by schedule.

Cron expressions are the natural format because they are timezone-aware when a timezone is given with them, and because they can express both "every night at 02:00" and "every hour on the hour" without a new vocabulary.

Concurrency and agents

A run needs an agent. The agent is either chosen explicitly as part of the target, or selected from the pool.

The platform offers three things for agent selection:

• Live presence is derived from the socket map — an agent is online only while its socket is connected. There is no stored "online" flag to go stale.
• The agent list is scoped to agents whose database status is active and that have a live socket.
• Each agent declares its capabilities at registration. A capability string that a target requires — for example that it can open serial ports — can be matched against what the agent declares.

Concurrency is a policy per agent:

The realistic constraint is physical, not logical. A bench has one serial port; two runs writing to it corrupt each other's results. Serial per agent is therefore the sensible default for hardware-facing work, with bounded concurrency for cases that do not touch a device.

Status and lifecycle

The studio tracks a command from the moment it is written to agent_commands until the result comes back:

pending  ──▶  in_progress  ──▶  completed

A run moves through the same three states, and its status in test_runs records where it is:

pending  ──▶  in_progress  ──▶  completed

A run that could not be carried out — no agent available, a resolution error, an infrastructure failure — is recorded as failed, and one stopped by a person is recorded as aborted. An aborted run keeps the results it already has: partial results are useful, and discarding them makes an aborted run indistinguishable from one that never started.

Individual executions have their own status in executions, and a run's outcome is derived from them rather than set independently. An execution that is still waiting carries pending, one the agent is working on carries in_progress, and one that has finished carries its outcome. The distinction between an execution that failed an assertion and one that hit an infrastructure problem is covered in Results and reports.

See also

• The Test Case Model — what a run resolves.
• Data-driven matrices — how many executions a run produces.
• Results and reports — the outcomes and their evidence.
• The Agent Pool — which agents are online and how they are assigned.