Marketplace seller software review methodology: failed test reruns

When a marketplace seller software test fails, a later success should not make the first result disappear. Decide in advance when to rerun, preserve the initial observation, record what changed and report both outcomes. This method is for controlled review tests, not live marketplace operations. If no action was run, report it as not tested rather than as a failure.

Treat each attempt as a separate observation

An initial attempt and a rerun are separate observations, potentially made under different conditions. Keep each result intact with its own configuration and supporting evidence. A successful rerun does not cancel the initial failure; it records what happened later. This method applies to controlled review tests, not live marketplace operations.

The record should distinguish what you observed from what you think caused it. "The action failed under this recorded configuration" reports an observation. "The service cannot perform this action" makes a broader claim that one test may not support. If the cause is unclear, say that it remains unclear rather than filling the gap with an assumption.

The review record should make the sequence easy to follow: initial attempt, any permitted change, rerun and the evidence for each. It is narrower than a feature score or supplier verdict, and sits beside the benchmark research protocol, which addresses the broader research task. For wider evaluation context, consult The Magenta Book; neither record replaces the other.

Set the rerun rule before changing anything

Write the rerun rule before altering the configuration or trying again. It should say what action may be repeated, who can authorise a change, what evidence must be retained and when a rerun is not appropriate. Setting the rule first helps keep the decision to rerun separate from the result.

This helps prevent a string of informal attempts from becoming an undocumented search for a successful outcome. The rule should also distinguish a repeat under the original conditions from an attempt after a change, and state what may change and why if a configuration change is permitted. For configuration management guidance, see the NIST publication on configuration management.

If the original action cannot be reconstructed from the available record, state that limitation: a later attempt cannot be presented as a reproduction of conditions you cannot establish. The Orange Book's risk management guidance can inform how you describe this uncertainty. If a proposed change was not authorised by the rule, preserve the first result and identify the departure. Do not present the altered attempt as an equivalent retry.

The review record should identify who observed the test and who authorised any change, so that the sequence can be understood later. A planning template can keep those gates and responsibilities visible, including what was authorised before the test proceeded.

Preserve the first failed action

Make the initial entry before changing the configuration or starting a rerun. Include the action attempted, the expected result, what actually happened, the configuration used and the evidence that supports the observation. Keep this entry unchanged even if a later result is more favourable.

The quality checklist also uses evidence IDs and an exit test. For tests where timing or event order matters, retain machine-readable event evidence where available. The Aqua Book's guidance on quality analysis provides a reference for the evidence discipline used in this kind of record.

A screenshot may help show an authorised view, but it should not replace event evidence when the conclusion depends on timing or state order. Capture screenshots only when authorised; a competitor evidence ledger can keep relevant captures apart from the rerun log. Retain the evidence needed to interpret each observed result.

Use bounded descriptions of what the evidence shows. If the result cannot be interpreted, mark it unresolved and explain the limitation. Do not turn a missing record into a presumed result, or infer an outcome that the available evidence does not support.

Record the rerun without overwriting the original

Create a separate entry for every attempt and connect it to the same test action. A simple log can use these fields:

Keep the changed configuration and its outcome together. A reader should be able to tell whether the rerun used the original conditions or followed a permitted change. If the repeat departed from the written rule, record the departure beside its result rather than treating it as confirmation. This keeps the attempted repeat distinct from a result under the original conditions.

Report both outcomes in sequence

A concise report should give the initial result first, then state whether a rerun took place and what changed beforehand. This preserves the order of events and helps readers distinguish the first observation from a later result. Report the conditions for each attempt rather than combining them into one outcome.

Say whether the rerun followed the prewritten rule. If it did not, describe the deviation and avoid presenting the second outcome as a clean reproduction. The review methods provide a wider comparison context for keeping that result within its test boundary.

A useful reporting structure is:

Use placeholders like these as a reporting structure, not as a substitute for evidence. If there was no rerun, say so. If a result was not interpretable, describe that limitation rather than labelling it a successful or failed outcome without support.

Keep the conclusion within the test boundary

A controlled rerun record does not establish how a system behaves in production. Do not imply that live listings, payments, customer messages or long-term operation were tested unless they were. The record shows what happened during the recorded attempts, not whether a wider requirement has been met.

Keep the review record separate from a live deployment or its reversal. If the work moves to changing or rolling back a live implementation, that is a different task from repeating a controlled test. The implementation guidance addresses that separate stage.

Before publishing, check that the first result remains visible, each rerun has its own evidence and explanation, and the conclusion reflects the full sequence. Where no action took place, say plainly that there is no failed action to report. This is also a boundary for a selection checklist: the record supports only the conditions that were tested.