Writing a Test Case

This page introduces the anatomy of a test case — its parts, how steps and assertions are anchored, where parameters come from, and what a reviewer should check before a case is saved.

Anatomy of a case

A case is made of four parts:

The intent is the part that survives longest. When a protocol changes, the variant changes and the intent is untouched — which is exactly why a report can still group results by capability.

A case belongs to a suite, and it can belong to more than one: the same capability can be part of a smoke suite and a regression suite without being duplicated.

The case file

id: tc.provision
title: Provision and enrol a digital key
description: The device completes enrolment and the platform can query the key state.
category: functional
priority: high
status: active
preconditions:
  - Device is powered on and attached to the bench
  - SIM is active and entitled
acceptance: The device acknowledges the enrolment and the platform reports ACTIVE
suite: [smoke, regression]

priority defaults to medium and status to active. The intent is written in domain terms: anyone can judge the acceptance criteria without reading a single script.

Steps and assertions

Steps live on a variant, not on the capability. Each step has an ordinal — the execution order, starting at 1 — and a kind:

Each step also carries the command to run, the script or payload to run it with, and an optional timeout in milliseconds. The commands are the ones the agent implements: shell, get_system_info, get_serial_ports, open_serial, close_serial, serial_write.

Assertions are anchored to a step

An assertion never floats on its own. It is attached to a step that produced a value, and it carries the extractor that reads that value:

The separation matters. The extractor knows that the status byte is the first byte of a hex reply for one OEM and the fifth for another; the matcher knows only that the value must equal 0x00. A reused matcher works everywhere, and the same step can carry several assertions:

- ordinal: 4
  kind: action
  command: serial_write
  script: "$ENROL_FRAME"
  assertions:
    - index: 0
      extractor: "reply.hex[0:2]"
      matcher: "equals"
      expected: "0x00"
    - index: 1
      extractor: "reply.elapsed_ms"
      matcher: "less_than"
      expected: 15000

The index is what lets one step carry several checks: index 0 and index 1 are separate assertions on the same step, each with its own extractor, matcher and expected value. They are reported separately, so a step that produces a wrong status and a slow reply shows two failures rather than one.

A step with no assertion is not a failure — it is a step whose outcome nobody checks. check, setup and teardown steps are often of this kind; verify steps should never be.

Parameters

Parameters are the data a procedure reads. They are declared by name and supplied by the project running the case. Typical ones: BENCH_HOST, PLATFORM_URL, SIM_IMSI, SIM_ICCID, MSISDN, DEV_PORT, IMEI, and raw frame templates such as ENROL_FRAME.

Two rules apply:

• Declare the placeholder, never the literal. If a frame contains the IMSI, the frame is "$ENROL_FRAME" and the IMSI is substituted at render time. A literal in a script is a case that only works on one project.
• Unresolved placeholders are an error. If a parameter is missing at render time, the execution is marked as an error and nothing is sent to the device. Sending the literal string $SIM_IMSI to a bench is never correct.

Sensitive values — the ones that identify a subscriber or a credential — are marked as secret so they are masked in the interface and stored encrypted.

Preconditions and teardown

Preconditions belong to the capability: the state the world must be in for the case to mean anything (the device is attached, the SIM is entitled, the platform account exists). They are checked by check steps at the start of a variant, and their failure is a configuration error rather than a product failure.

Teardown belongs to the variant, because only the variant knows what it opened. A variant that opens a serial port must close it — otherwise the next run on the same bench starts with a port already held. Teardown steps run even when the procedure fails part way through.

Checklist

Before a case is considered ready:

• The title and description use domain terms, with no protocol, model or port in them.
• The acceptance criteria can be judged without reading any script.
• Every difference between two projects is expressed as a parameter, not as a copied step.
• Every protocol or device difference is expressed as a variant, not as a parameter.
• Each step has an ordinal, a kind, a command and a timeout.
• Every verify step carries at least one assertion, with an extractor and a matcher.
• A step with several checks gives each of them its own index.
• No script contains a literal that should be a parameter.
• The variant closes anything it opens.
• The case is marked draft until someone has run it on a real bench.

See also

• The Test Case Model — the four layers and why they are separated.
• Data-driven matrices — generating cases from a table instead of writing them.
• Remote Shell and Serial — the commands a step can invoke.