Data-driven Matrices

This page introduces how a table of data becomes a set of test cases, how the expansion works, and how two matrices are combined into a cross-product of cases.

What a matrix is

A matrix is a table where each row is a data set and each column is a parameter. Running the matrix means running the same procedure once per row, with that row's values bound to the parameters.

The alternative is writing one case per row by hand. With twenty IMSIs across four protocol families that is eighty cases, and eighty cases drift apart: one gets an extra step, another gets its timeout changed, and nobody can tell which of them still reflects the original intent.

A matrix keeps one procedure and one intent. The data varies; the flow does not.

one capability  ──▶  one variant  ──▶  one procedure  ──▶  N rows  ──▶  N executions

Every row produces its own execution with its own result, so the coverage report counts rows, not cases.

The matrix file

The matrix lives next to the case it feeds, in the repository, and is reviewed like code:

# matrices/provision.yaml
name: Provision — SIM matrix
case: tc.provision
variant: v.provision.ccc.apple
parameters:
  - name: SIM_IMSI
    source: column
  - name: SIM_ICCID
    source: column
  - name: MSISDN
    source: column
  - name: BENCH_HOST
    source: constant
    value: 10.20.1.5

case names the capability the matrix feeds and variant names the implementation it runs against. The source of a parameter decides where its value comes from:

Source	Meaning
`column`	Take the value from the column with the same name in the row.
`constant`	The same value for every row.
`sequence`	A generated value — an index, a timestamp, a random string.
`derived`	Computed from other values in the row, such as a hex frame built from the IMSI.

A row that is missing a column value is an error, not a blank. The execution is marked as an error and the matrix run reports which rows are incomplete.

The data file

The rows themselves are usually a CSV, so that a lab engineer can maintain them in a spreadsheet:

SIM_IMSI,SIM_ICCID,MSISDN,EXPECT_STATUS
460001234567890,89860419101880277149,+8613800000001,0x00
460011234567890,89860419101880277150,+8613800000002,0x00
460021234567890,89860419101880277151,+8613800000003,0x21

Parameters and expansion

At run time the matrix expands row by row:

Read the row into a parameter set, named by the column.
Overlay the project's parameter values for keys the matrix does not define.
Overlay the variant's overrides, which win over both.
Render the procedure — the same rendering any other run uses.
Execute and record one execution per row.

Each execution records the matrix it came from and which row within it, so a report can point at the exact line of the data file that produced a failure.

Two things follow from this:

A row can be expected to fail. The EXPECT_STATUS column above is a per-row expectation. A row whose expected value differs from the others is a legitimate case, not a mistake.
The matrix does not change the flow. If a row needs a different flow, that is a variant, not a column. A matrix that has to express "and then do something else" is a sign that the split between procedure and parameter is wrong.

Combining matrices

Two matrices can be crossed: the rows of one are paired with the rows of the other, producing a cross-product of executions.

provision × SIM matrix   (4 rows)
        × protocol matrix (3 rows)
        ─────────────────────────────
                    12 executions

The rule for crossing is that the matrices must not write the same parameter. If two matrices both define SIM_IMSI, the result is ambiguous and the combination is rejected at validation time rather than resolved by an ordering rule nobody remembers.

Ordering is stable: the first matrix varies slowest, the last varies fastest. This makes a combined run reproducible — the same matrices in the same order always produce the same sequence of executions, which matters when you are comparing two runs.

Crossing is also where the cost shows up. A third matrix of eight rows turns twelve executions into ninety-six. Matrices are cheap to write and expensive to run, so the number of rows should be chosen deliberately rather than by pasting a whole spreadsheet in.

Checklist

Before a matrix is committed:

The matrix names exactly one case and one variant.
Every parameter used by the procedure is either a column, a constant, a sequence or a derived value — no parameter is left to chance.
Two matrices that will be crossed never define the same parameter.
The expected outcome is per row, not per matrix, when rows differ.
The row count is deliberate, and the cost of the resulting run is understood.
Sensitive columns are marked as secret.
A run with a deliberately failing row is expected and its result is distinguishable from a genuine regression.