Strategy
Part of A store conversion plan built on a defined journey and predeclared stop rules
Auditing a live conversion change for strategy mistakes
Audit a live ecommerce conversion change for strategy mistakes: seven checks, the evidence to pull for each and the UK rule behind it, before the next release.
A change that is already live can still be audited. Seven checks, each with the evidence to pull and the UK rule behind it, show whether the change corrupts a merchant decision or compromises a customer journey. The scope is an online seller operating in England.
What to take away
- A placed order is not payment, fulfilment or a retained purchase, so pull the cancellation, return, refund and dispute states.
- Revealing a mandatory price late can change completion patterns while making the offer less transparent.
- A desired experiment does not authorise a tag, cookie, script or device match before privacy review.
- Before-and-after order movement is not causal without a suitable counterfactual and a predeclared plan.
- Record the date and outcome of every rule check, and stop the change if one fails.
Open the audit with the change record
Start from what shipped, not from the dashboard. Write the date of the change, the journey state it touched and the owner who approved it. Every later check refers back to this record.
Then list the rate the change was meant to move and the states beside it. If the post-order states were never defined, the first finding is that gap, and the audit continues with the checks below.
Leave merchant thresholds blank. The audit finds evidence; a named reviewer decides what it means for the live change.
Pull the order and price evidence
Three checks sit on the order and the payable total. Each has a quick test and a record to read, summarised in the table.
| Check | Question to ask | Evidence to pull | Rule behind it |
|---|---|---|---|
| Post-order states | Were cancellation, return, refund and dispute states kept beside the rate? | Order-state records, named owner for each state | Consumer Rights Act 2015; Consumer Contracts Regulations 2013 |
| Late price | Does a mandatory amount appear later than before the change? | Screens of every checkout step, before and after | Digital Markets, Competition and Consumers Act 2024 |
| Default extras | Does an optional payment start selected? | Choice event, kept apart from completion | Consumer Contracts Regulations 2013 |
The Consumer Rights Act 2015 gives a right to a refund, repair or replacement for faulty goods. Under the Consumer Contracts (Information, Cancellation and Additional Charges) Regulations 2013 a consumer can cancel most online orders, and GOV.UK says customers can cancel their order up to 14 days after delivery.
For price, the Digital Markets, Competition and Consumers Act 2024 treats drip pricing as a banned practice. The same 2013 Regulations require the total price before the consumer is bound, and the CMA's price-transparency guidance covers mandatory fees, taxes and charges.
On extras, CMA additional-charge guidance says a consumer must actively agree to an optional payment. A pre-ticked box is not express consent under the 2013 Regulations. If the interface requires an opt-out, the consumer-law gate stops the change.
Pull the permission and access evidence
Two checks look at who is measured and who can use the route. Both are easy to skip because the headline rate looks fine without them.
For measurement, ask what tags, cookies, scripts or device matches the experiment added and whether privacy review came first. The ICO's storage and access technology guidance explains the PECR and UK GDPR treatment, and the checkout data map shows where to look. Missing identifiers stay visible as missingness.
For access, ask who the measured denominator leaves out. A rate can hide customers excluded by inaccessible navigation, authentication, error handling or support. The Equality Act 2010 requires service providers not to discriminate and to make reasonable adjustments, so give accessibility defects an independent stop rule.
Pull the comparison and safety evidence
The last two checks test the claim and the product. Both can overturn a result that looked clean.
For the claim, ask what else changed. Orders can move with stock, promotion, season, channel mix, outages and fulfilment, so a before-and-after gap is not an effect. HM Treasury's impact-evaluation quality guidance distinguishes monitoring from attribution and explains the need for a suitable counterfactual.
For the product, ask whether the page can complete orders while presenting an item that should be corrected or withdrawn. The General Product Safety Regulations 2005 set the safety duty, and OPSS guidance says UK sellers must not sell products they know, or should know, are unsafe. Check the OPSS recalls and safety notices before an item stays live.
Record the audit and its stop rule
For each check, name the journey state, the evidence owner, the specialist gate, the correction and the rollback condition. A rate increase cannot compensate for unsafe products, broken payments, weak security or any failure above.
Check the CMA's published enforcement cases, the ICO's enforcement action and the OPSS recalls and safety notices for every rule the decision relies on. Record the date and outcome. A failed check stops the change before exposure, not after it.
The wider planning guide shows how the same gates are set before a change ships, so the next release starts with them in place.
Before you act
- Write the change date, journey state and approving owner.
- Pull post-order states and name an owner for each.
- Screen every checkout step for late mandatory price information.
- Confirm privacy review came before any collection.
- Include the full intended population in the measured denominator.
- Check for a product recall or safety notice.
Common questions
Can a rate increase offset a failed audit check?
No. A rate increase cannot compensate for unsafe products, broken payments, weak security or any of the failures above. A named reviewer approves only their own boundary, and a failed check stops the change.
What should be pulled first when auditing a live change?
The change record: date, journey state touched and approving owner. Every later check refers back to it, and a missing record is itself a finding.
Does before-and-after movement prove the change worked?
No. Orders can change with stock, promotion, season, channel mix, outages and fulfilment. A causal claim needs a suitable counterfactual and a plan declared before exposure.
