Testing mobile wallets

The system-level car key lives in a wallet on a phone or a watch, and the vehicle maker's own app is often part of the same journey. This page covers how a handset is attached to a bench, how the studio drives it, how screen evidence is captured, how a compatibility sweep across handset models is expressed, and how screen-level assertions are made.

Who holds what

The studio drives the handset indirectly. The agent holds the physical connection — the USB cable and the ADB transport — and the studio holds the case, the step sequence and the result. A wallet case is an ordinary case: a procedure dispatched to an agent, with the outcomes recorded against the run.

Attaching a handset

The handset is connected to the bench over USB, and the platform tooling on the bench talks to it over ADB. The agent reports what is attached over the socket as a device_event, and the studio records the event alongside the rest of the run.

Before a case runs, the studio checks that the attached device is the one the case expects — the same model, the same serial. A case that runs against the wrong handset produces results that look valid and are not, so the check belongs at the start of the procedure, as a check step.

The handset also has to be in a state where the wallet can be exercised: awake, unlocked, and signed in to the wallet account the case uses. Those are preconditions of the case, and they are checked rather than assumed.

Driving the wallet

There are two routes into a wallet, and a case may use either or both.

The cloud route is an ordinary cloud case, and it is described in Testing the cloud services. The vehicle maker's own app is driven the same way as the platform wallets: the app is launched on the handset, the flow is navigated, and the state is read from the screen.

Each wallet ecosystem has its own navigation. The procedure that reaches a pairing confirmation in one ecosystem is a different variant of the same capability in another, which is exactly what the variant layer of the case model is for.

Capturing screen evidence

A capture is taken on the bench, brought back over the agent channel, and stored with the execution it belongs to. The capture is part of the step, so the image that produced a verdict is the image stored against the result — an auditor can look at what the screen actually showed.

Two rules make that evidence worth keeping:

• Capture after the interface settles. A screen that animates, scrolls or shows a toast can be captured mid-transition, and the frame you keep is the frame you asserted on.
• Capture the state you assert on, not a later one. The capture belongs to the step that reads it. A screen captured after the test has moved on is a picture of the wrong moment.

A compatibility sweep

A sweep across handset models is a data-driven matrix. The rows are the devices, the columns are the parameters, and each row produces its own execution with its own result.

# matrices/wallet-compatibility.yaml
name: Wallet compatibility
case: tc.wallet.pair
variant: v.wallet.pair.pairing
parameters:
  - name: HANDSET_MODEL
    source: column
  - name: OS_VERSION
    source: column
  - name: WALLET_APP_VERSION
    source: column

HANDSET_MODEL,OS_VERSION,WALLET_APP_VERSION
iPhone 15,iOS 17.4,9.2
Pixel 8,Android 14,24.1

The coverage report then counts rows rather than cases, and a model that fails pairing on one OS version is visible as exactly that.

Screen-level assertions

A screen assertion asks a question about a captured image and turns the answer into a pass or a fail. The question has to be one a test can fail on: "does the screen show the string Pairing failed: session timeout" is an assertion; "does the screen look correct" is not.

The screen-vision capability produces those answers, and it is described in Screen vision. Two habits keep the assertions honest:

• Assert against ground truth. Check the screen against a string the platform already knows — a specified error code, an expected state — rather than against what the image seems to contain.
• Keep the answer exact. A near-miss string is not a pass. An answer that cannot be matched exactly is surfaced as such, and the execution is failed rather than rounded into a green result.

Screen evidence does not replace protocol-level verification. A wallet that renders the right state while talking to the wrong endpoint is still a defect, and only the protocol assertions find it.

Failure modes and boundaries

• Recognition of non-Latin scripts is less reliable than Latin text. Bench devices are used across markets, and the wallet interfaces under test include Chinese and other non-Latin text. Assert on exact strings, and treat a low-confidence answer as a failure rather than a pass.
• A capture is an observation, not a verdict. It says what was on the screen at one instant on one device. It does not distinguish a product defect from a fixture that was never put into the right state.
• One handset per bench. A bench with two devices attached is ambiguous about which one the agent is driving. Attach one, and let the matrix move between benches.
• The screen is one of several surfaces. A wallet flow also touches the handset's secure element and the backend. A green screen does not mean the key was issued — see Testing the cloud services.

Related pages

• Testing the cloud services — the backend half of the wallet journey.
• Screen vision — asking questions about a captured screen.
• Data-driven matrices — how the sweep is written.
• The Test Case Model — where a screen assertion belongs.
• Device topology and physical connection — what the bench owns.
• Results and reports — how the evidence is kept.