Results and Reports

This page introduces the result model, the difference between a failure and an error, what evidence is kept, and how results are reported and exported.

The result model

Every execution of a case produces one result. A result carries:

Field	Meaning
`run_id` / `case_id`	What ran, in which run.
`variant`	Which implementation of the capability ran.
`target`	The protocol family, device type and OEM combination it ran against.
`status`	The outcome — one of the values below.
`duration_ms`	How long the execution took end to end.
`error_message`	Why it did not pass, when it did not pass.
`result`	The per-step outcome: status, duration, message.
`script_hash`	A hash of the rendered procedure that ran.

The executions table carries these as run_id, case_id, status, result, duration_ms, error_message and executed_at.

Two things sit on top of that:

Per-step results. A failed execution is much easier to act on when you can see that step 4 of 6 failed and the rest passed, rather than only seeing "failed".
A script hash. This makes history trustworthy. When the same case fails in one run and passes in the next, the hash tells you whether the procedure changed in between. Two different hashes for the same variant mean the case was edited; the same hash means the product changed.

execution:
  case: tc.provision
  variant: v.provision.ccc.apple
  target_id: tgt.ccc.apple
  script_hash: "9f2c…"
  status: failed
  duration_ms: 41230
  error_message: "step 4: reply status expected 0x00, got 0x21"
  result:
    - { ordinal: 1, status: passed, duration_ms: 120 }
    - { ordinal: 2, status: passed, duration_ms: 310 }
    - { ordinal: 3, status: passed, duration_ms: 980 }
    - { ordinal: 4, status: failed, duration_ms: 1500, message: "expected 0x00, got 0x21" }
    - { ordinal: 5, status: skipped }

Pass, fail, error

The distinction between these outcomes is the most important thing in the model, because it decides whether a red result sends someone to a bug or to a broken bench.

Outcome	Meaning	Whose problem
Passed	Every assertion in the procedure was satisfied.	—
Failed	An assertion was evaluated and did not hold. The system under test did not behave as required.	The product.
Error	The execution could not be completed — the agent was unreachable, a parameter was missing, a port would not open, the renderer left an unresolved placeholder.	The setup.
Skipped	The execution was deliberately not run — the target had no matching variant, or the run was aborted before reaching it.	Nobody's problem, but it is not a pass.

Two rules follow:

An error is never a failure. If the bench was down, the product was not tested, and calling it a failure corrupts the pass rate and hides a real regression behind infrastructure noise.
A skip is never a pass. A case that was skipped because no variant matched is a coverage gap, and it belongs in the report as a gap.

A run's own status is derived from its executions: any error makes the run failed as an execution problem, any failure makes it failed as a product problem, and the two are reported separately so that the cause is visible.

Evidence

Evidence is attached to the execution that produced it rather than to the run as a whole: the stdout of step 4 belongs with step 4.

Evidence	Where it comes from
Standard output	The command result's `stdout`.
Standard error	The command result's `stderr`.
Exit code	The command result's `exitCode`.
Duration	The command result's `durationMs`.
Serial port metadata	Path, manufacturer, serial number, vendor and product ids, delivered as a device event.
Device events	Hot-plug and connection events, delivered as device events over the socket.

The studio records serial metadata, not the byte stream

The bytes coming back from a serial port are delivered to the agent, and the interactive serial console lives on the agent's own page at http://localhost:3001 on the machine under test. The studio keeps the port's identity — path, manufacturer, serial number, vendor id, product id — so a result says which hardware produced it. See Remote Shell and Serial.

The platform's evidence for a command is therefore the text it produced. Anything richer has to be produced by the script itself and returned as part of stdout, or written on the bench and retrieved by a later command.

Reporting

Reporting is where the model pays off, and it is built around two questions that a flat list of cases cannot answer.

Coverage by capability

Grouping results by capability answers "is our feature set complete?" — for each capability, how many of its variants were exercised and how many passed:

Capability	proj.a	proj.b	Total
Provision and enrol	4/5 passed	5/5 passed	9/10
Revoke	2/2 passed	—	2/2

The totals are meaningful because the capability is the shared unit: the same case is counted in every project that ran it.

Compatibility by target

Grouping results by target answers "does our bench fleet work?" — for each capability and each protocol family, device type and OEM combination, whether it passed:

Capability	CCC / Apple	CCC / Google	ICCE	ICCOA	COT
Provision and enrol	pass	pass	no variant	fail	no variant

The two views are complements: the first is a product question, the second is an interoperability question. Reporting only one of them gives half the answer.

Trend over time

For a given capability and variant, the pass rate over successive runs, with the script-hash changes marked. A failure that appears at the same point as a procedure edit is a different problem from one that appears without an edit, and the trend view is where that is visible.

Exporting

A report is exported as a machine-readable table with one row per execution, carrying the case, the variant, the target, the outcome, the duration and the error message. The same rows are available in the executions table, joined to test_runs for the run's name, target and timings, and to test_cases for the case's title and priority.

Exporting is what lets a run be correlated with an external system — a nightly report posted to a chat channel, a build pipeline that fails on a regression, a dashboard that reads the table directly.