Testing the cloud services

The digital key cloud is where a key is issued, distributed, used and revoked, and where the certificates that bind a vehicle, the cloud and a handset together are held. This page covers how the studio exercises the backend interfaces, how a cloud call becomes a step of a case, how interface performance is measured and recorded, how key issuance and distribution are verified, and how the certificate chain is checked.

Where the cloud sits

The cloud is reached from the studio, not from the agent. A bench command needs the physical device; a cloud call does not. The studio issues the request itself and asserts on the answer, so a cloud case runs whether or not an agent is online, and nothing is dispatched to a bench.

This is the opposite of the bench-driven work in Remote shell and serial, where the agent does the work.

Exercising a backend interface

A cloud step is one request plus a set of expectations on the response. The studio sends the request, waits for the answer, and captures what came back: the status code, the response headers, the body, and the time the call took. The elapsed time is the same measurement the platform records as duration_ms on the execution.

Cloud step expectations use the same two-part shape as every other expectation: an extractor that reads a value out of the response, and a matcher that decides whether it passes. The extractor is specific to the interface under test; the matcher is portable, so equals, less_than or contains is reused across cloud interfaces and bench cases.

- ordinal: 2
  kind: action
  script: "$KEY_ISSUE_REQUEST"
  timeout_ms: 10000
  assertions:
    - index: 0
      extractor: "response.status"
      matcher: "equals"
      expected: 200
    - index: 1
      extractor: "response.body.keyId"
      matcher: "equals"
      expected: "$EXPECTED_KEY_ID"

Measuring interface performance

Two numbers matter for a backend interface: how long a call takes, and how often it fails.

• Latency is the elapsed time of the call, recorded on the execution as duration_ms. An assertion such as response.elapsed_ms less_than 15000 turns a budget into a result that can fail.
• Error rate is derived from the executions of a run: the share that ended in a failed execution against the share that ran. A matrix across regions or accounts gives the rate per row, so a slow region is visible next to a slow interface.

The distinction between the two kinds of failure matters, and the platform keeps them apart:

A performance budget belongs to the first row only when the interface answered. A timeout is not a latency regression, and counting it as one hides the real defect.

Verifying key issuance and distribution

Issuance is a three-party claim, and a test is only as good as the weakest link in it: the backend issued a key, the vehicle carries that key, and the handset holds it.

The key matches. The key identifier and its fingerprint returned by the issuance interface are compared with the key provisioned to the vehicle, read from the vehicle side of the bench, and with the key held in the wallet. The three must be the same key, not three keys of the same shape. The comparison is an equality assertion on a fingerprint value, which makes it portable across projects.

Use is one-time. A digital key that has been used must not be usable again. The case presents the key a second time and asserts that the interface refuses it. The refusal is the expected result, so it is written as its own expectation rather than as an error.

Revocation takes effect. After the key is revoked, the interface refuses it, the vehicle stops honouring it, and the wallet no longer offers it. Each of those is a separate expectation, because they are separate systems, and a revocation that reaches the backend but not the handset is a real defect that a single assertion would have missed.

Checking the certificate chain

The certificates that bind a vehicle, the cloud and a handset together form a chain of trust, and the chain is checked as a whole rather than one certificate at a time.

A failure in any of them is recorded against the step that made the check, with the certificate and the failing property named. A chain that fails on the issuer is a different defect from one that fails on the binding, and the report has to tell them apart.

Failure modes and boundaries

• A cloud interface that cannot be reached at all produces an errored execution for every step that depends on it. The run continues, and the report shows the dependency, so one unreachable interface does not look like a hundred product failures.
• Timeouts on cloud steps are set per step, in the same timeout_ms field as any other step, and they are enforced by the studio rather than by the bench.
• The studio reaches the cloud over the network the studio sits on. In an in-house installation that is your network, and the routes to the backend are part of what you operate — see Network requirements. Key material and certificates that a case prints into a response body are stored with the run, so keep secrets out of the assertions and out of the captured text.

Related pages

• Testing mobile wallets — the handset half of the same journey.
• The Test Case Model — the layers a cloud step belongs to.
• Data-driven matrices — sweeping an interface across regions or accounts.
• Results and reports — how a failure and its evidence are kept.
• Remote shell and serial — the bench-driven half of the work.