DK Studio/Documentation
ProductChangelog

Screen vision

Screen vision captures what is on the screen of a device on your bench and lets you ask questions about it. The useful form of the feature is not "what does this look like" but "does this screen match the expectation" — the same way a test can fail.

What it does

A digital key bench often ends in a human interface: a companion app on a phone, an HMI on the head unit, a pairing screen on a reader. Screen vision brings that interface into the run. It has two halves.

  1. Capture. An image of the screen is taken on the machine under test and brought back to the platform over the agent command channel, alongside the command record that produced it.
  2. Understanding. You ask a question about an image the platform holds, and the platform answers it.

The image is kept with the execution it belongs to. An answer that cannot be traced back to the frame it was computed from is not auditable, so the two are stored together.

Asking about a screen

The questions that make sense for this domain are about state and content:

  • Does the screen show the expected pairing or authentication state?
  • Is the error message the one the specification requires, or a different one?
  • Is a particular element present, visible and enabled?

For these to be usable as test assertions, the answer has to be something a test can pass or fail on. "Yes, the screen looks correct" is not an assertion. "The screen shows the string Pairing failed: session timeout" is one.

Questions need ground truth

The most useful form of this feature is checking a screen against something the platform already knows — a specification string, an expected state, a value from a serial log. Asking a model what it sees produces an interpretation; asking it whether the screen matches a known expectation produces something a test can fail on.

How it fits a run

A screen check is a step like any other: it happens at a defined point in the procedure, it has an expectation attached to it, and it produces a pass or a fail that is recorded with the rest of the execution. The capture is part of the step, so the image that produced the verdict is the image stored against the result.

Limits

  • Accuracy on a moving interface. A screen that animates, scrolls, flashes a toast or updates a counter can be captured mid-transition. An answer computed from that frame describes that frame, and an assertion built on it fails for reasons that have nothing to do with the product. Settle the interface before you assert on it.
  • Reading non-Latin scripts. Bench devices are used across markets, and the interfaces under test include Chinese and other non-Latin text. Recognition of those scripts is materially less reliable than Latin text, and a near-miss string is not a usable verdict — either it matches or it does not. A low-confidence answer is surfaced as such rather than rounded into a pass.
  • A screenshot is an observation, not a verdict. The capture says what was on the screen at one instant on one device. It says nothing about the behaviour that produced it, and it does not distinguish a product defect from a fixture that was never put in the right state. Only a stated expectation attached to the capture turns it into a result.
  • The answer depends on the expectation, not on the model. If the expectation is vague, the answer is vague. Writing the check down in a form that can be matched exactly is what makes the feature reliable.

See also