One AEB run produces an unexpected result. The team repeats the same scenario twice and both reruns behave normally. That shows the behavior does not readily reproduce, but the first result is still unexplained. The team now has to decide whether this anomaly is understood well enough to move on or whether it needs further investigation.
Examining the video may not provide the answer. Two runs that look almost identical can differ in lateral position, trajectory, timing, object detection, environmental conditions or system state. The relevant difference may even have occurred earlier in the run. A previous detection or change in system state can affect behavior that only becomes visible later.
Understanding the original result requires looking beyond whether the behavior happened again.
Narrowing the investigation
An inconsistent result gives engineers somewhere to start, but the point where the unexpected behavior becomes visible may not be where the problem began. A change in sensor output, system conditions or an earlier event may only become apparent later in the run. Root-cause investigation often means working backwards through that history to understand what preceded the behavior and which differences are relevant.
Repeating the scenario helps establish whether the same behavior occurs again. When the next run looks normal, however, the original anomaly is still there to explain. More than one underlying mechanism may be consistent with the same visible outcome. The available evidence has to help narrow those possibilities, not simply add more normal runs.
A validation program may contain a very large population of events, but only a small subset may be genuinely useful for investigating a particular result. Engineers first need to find events that are sufficiently comparable, then understand which differences between them could matter. A shared condition may support one explanation, while a difference between two otherwise comparable events may weaken another or rule it out.
As those comparisons are made, fewer events may remain relevant and some explanations become less plausible. Engineering judgment is still needed to understand what those differences mean and where closer investigation is warranted. At scale, that narrowing matters because engineering judgment cannot be applied with the same depth to every event.
Ten more runs showing behavior the team already understands may increase confidence that the expected response is common without explaining the original exception. A carefully chosen comparison that helps eliminate a plausible explanation may tell the team more than another repetition of an already familiar result.
The significance of what remains unexplained
The objective is not to explain every anomalous result. A small variation with no credible effect on the engineering decision may require little additional investigation. A single unexplained behavior with potentially significant consequences may deserve much more scrutiny, even when a far larger population of runs has produced the expected outcome.
How often the exception occurred is only part of what the team has to consider. If the conditions around it are understood and the remaining uncertainty is unlikely to affect the decision, further investigation may add little. If an important explanation still fits the evidence, another test or comparison may still be worth pursuing.
Enough evidence for which decision?
The same evidence can be sufficient for one engineering decision and insufficient for another.
A team may need relatively little evidence to decide that an event requires investigation. Closing the anomaly requires greater confidence that the relevant explanations have been examined. A software release, or another decision with greater consequences, may require more confidence again.
Mileage, driving hours, test runs and scenario counts provide useful context, but none establishes sufficiency on its own. Their value depends on whether the collected evidence represents the conditions and uncertainty relevant to the decision. That includes whether the evidence covers the conditions in which the decision has to hold.
Thousands of kilometres of highway driving in good weather may provide substantial evidence for one engineering question while contributing much less to an investigation involving pedestrian behavior in poor visibility.
When a measurement becomes evidence
A KPI value of 0.8 can be calculated correctly and still be difficult to use as evidence. An engineer making a decision from that result needs to know what was measured, which software configuration produced it, which data population was included and which acceptance criterion was applied.
The underlying evidence also needs to be reconstructable. Signals, events, video and system state lose investigative value when the team cannot reliably establish how they relate in time and to one another.
The result may later be reviewed by people who were not involved in producing it. They need enough context to understand the conditions behind the measurement and judge what the result actually supports.
Deciding when additional evidence is useful
There is almost always another test the team could run, another drive it could collect, another simulation it could execute or another group of events it could examine. Each one consumes engineering time, test capacity, compute resources, fleet availability or program schedule. Additional evidence earns that cost when it has a realistic chance of resolving something that still matters.
No fixed number of runs, kilometres or hours determines that point. If an important explanation remains unresolved and another test or comparison could distinguish it from the alternatives, continuing the investigation has a clear purpose.
The team has to ask whether the next piece of evidence could change what it decides to do. If it is unlikely to change either the engineering conclusion or the action that follows, its value begins to fall.
An investigation does not need to explain every possible source of uncertainty before an engineering decision can be made. It does need to leave the team with a clear enough understanding of what remains unexplained, whether it could affect the decision and whether further evidence is likely to change what the team does next.
Where does your team draw that line when an important result remains unexplained?
