DK Studio/Documentation
ProductChangelog

Root-cause analysis

When a run of a few hundred cases comes back with a dozen failures, the work that follows is manual: read each error, decide whether it is a product bug, a test problem or a flaky bench, and group the ones that look alike. Root-cause analysis does the grouping and proposes candidate causes, with the evidence attached to each one.

How a run is read

  1. A run finishes and its executions are recorded.
  2. The failures are grouped by similarity — the same error text, the same failing step, the same symptom.
  3. Each group is a candidate issue, with a summary and the evidence behind it.
  4. An engineer confirms, dismisses or reclassifies each group.

Step 4 is where the value is. A machine that reads error strings correctly saves an engineer an hour of copy-paste; a machine that diagnoses correctly saves the investigation itself, and that is a much harder claim to make.

Grouping failures

A cluster is built from the execution records a run produces. The signal is shared:

  • the same error message, or the same shape of error message;
  • the same step of the same case failing;
  • the same symptom appearing on a set of cases that share a capability or a device.

Cases that fail for different reasons are kept apart even when their error text matches, because a cluster that merges two problems hides both.

Summarising a cluster

Each cluster gets a short summary that a person can read in a sentence: what failed, how many times, on which cases, and the signature they share. The summary is what you see in the list of clusters, and it is written from the executions themselves — the error text, the result, the exit code, the duration, the step that failed.

Suggesting a cause

A cause suggestion is a ranked list, and each item carries its reasoning and the evidence it rests on: which executions it explains, which error messages or result lines it points at, and what would confirm or rule it out.

The evidence available is what the run recorded:

  • Error messages and results. The error message of each execution, and the result text, exit code and duration stored with it.
  • Shell output. The stdout and stderr a command produced, kept in the command result.
  • Bench traces. The serial byte streams stay on the agent, so a CAN log or a protocol trace reaches the analysis only when a bench script captures it and it comes back as part of a result.

A suggestion that cannot point at evidence is not a suggestion; it is a guess, and it does not appear in the list.

Limits

  • A suggestion is a hypothesis, never a verdict. It is ranked by resemblance to patterns in the data, and it can be wrong in ways that are hard to see: the same error text can follow from a product defect, a broken fixture, a misconfigured bench or a bad test. Treat a suggestion as a place to start looking, never as a conclusion, and never file a defect from one without checking.
  • Similar-looking failures are not the same failure. A cluster that looks like ten cases is sometimes one bug reported ten ways, and sometimes ten bugs that happen to share an error message.
  • The bench is not the product. A large share of failures in a hardware lab come from the rig, not from the device. An analysis that reads errors without the surrounding execution context attributes bench noise to the product.
  • Coverage of the data limits the confidence. The analysis sees the executions of a single run. A failure that appears only every twentieth run is invisible to it, and a cluster is only as complete as the records behind it.

See also