Measurement
Part of Conversion reporting you can reproduce, from order-state ledger to causal boundary
Conversion measurement mistakes that hide customer harm in price, access and security
Three conversion measurement mistakes that hide customer harm: a price route, an accessibility count and a transaction trail that no dashboard rate can clear.
A conversion dashboard can look healthy while customers are being harmed. Three gaps are easy to miss: a checkout price that rises late, an accessibility count taken for proof, and a transaction trail that nobody watches. Each needs evidence that a conversion rate cannot supply.
What to take away
- A higher completion rate never excuses a hidden mandatory charge or a confusing optional choice.
- An automated accessibility issue count does not show whether disabled customers can finish a task.
- Security evidence and incidents belong in an independent queue, not in a commercial score.
- Each of the three checks is a stop gate, so it is never weighted against a rate.
- When a check changes a decision, correct the decision record as well as the chart.
Mistake one: rewarding a defective price route
Do not read a rising completion rate as proof that the price route is sound. A hidden mandatory charge can lift completions while still misleading customers. The rate records that people finished, not whether they were told the total they would pay.
The CMA's price-transparency guidance covers mandatory fees, taxes and charges that belong in the pricing information. Walk the journey from the first price shown to the payment page, and note each point where a charge appears for the first time.
Treat the result as a consumer-review item, not a metric weight. A confusing optional choice, such as a pre-selected extra, goes in the same queue. If the review finds a defect, the journey is stopped or corrected whatever the completion rate says.
Mistake two: treating an accessibility count as impact
An automated scan counts issues, and the count is not an outcome. It cannot tell you whether a disabled customer can find a product, pay, cancel or reach support. A low count and a failed task can both be true at once.
Under the Equality Act 2010, a service provider has a duty to make reasonable adjustments, which is why the task matters more than the tally. Test product choice, payment, cancellation and support as tasks, with a qualified reviewer, rather than reading scan output as a success rate.
Record four things beside any accessibility result: the task, the method, the affected population and who reviewed it. Keep the entry out of the conversion measures themselves, because a scan score divided by anything is not a conversion rate.
Mistake three: ignoring the transaction trail
Analytics alone may not expose abuse, tampering or a broken transaction trail. Security teams watch different signals from the commercial team, and a good conversion week says nothing about either set.
The NCSC describes transaction monitoring as a capability to detect and respond to abnormal and suspicious user activity on online services. Logging and security monitoring are operating concerns, so a merchant should know who owns them before a release changes the journey.
Keep incidents in a queue that the conversion owner cannot close. A commercial result cannot approve a security finding, and a security finding can stop a release regardless of the rate. Tracking and data-handling questions sit beside it in the checkout data map.
The three checks side by side
| Check | What a rate shows | What it can hide | Evidence to record |
|---|---|---|---|
| Price route | Customers who completed | A charge shown late or a pre-ticked extra | Price at each step, review note |
| Accessibility | Tasks started and finished | Tasks a disabled customer cannot finish | Task, method, population, reviewer |
| Security | Orders placed | Abuse, tampering, a broken trail | Alert log, incident owner, status |
A table like this belongs on the reporting dashboard as three independent rows. None of them feeds the headline figure, and none of them is traded off against it. Blank is a valid entry while a check has not been done.
Correct the decision, not only the chart
When a check overturns a result, fixing the chart is half the job. The decision that rested on that result also needs correcting, or the same mistake returns in the next review.
Preserve the original definition, the affected periods, the reason, the replacement version and the owner. The same habit appears in reproducible conversion reporting, where a restatement keeps the first figure visible. Buyer thresholds stay blank until the merchant has comparable evidence.
Check any strong early result against these three gates. If a release has already gone out, read the stop rules in the plan before deciding whether to pause it.
Before you act
- Walk the journey from first price to payment and note where each charge first appears.
- Test product choice, payment, cancellation and support as tasks, not as scan counts.
- Name the owner of security logging and transaction monitoring.
- Keep price, accessibility and security results in separate queues.
- Record the original figure, reason and replacement version for any corrected decision.
Common questions
Can a high completion rate offset a pricing problem?
No. A completion rate shows that customers finished, not that the price was clear. A hidden mandatory charge or a confusing optional choice is a separate stop gate and is reviewed on its own, whatever the rate says.
Does a clean accessibility scan show that disabled customers can buy?
No. A scan counts issues, but it does not describe whether disabled customers can complete product, payment, cancellation or support tasks. Record the task, method, affected population and a qualified review instead of turning scan output into a success rate.
Who approves a result when security evidence is open?
No commercial result can approve it. Security evidence and incidents stay in an independent queue, and a conversion gain does not close them. The release waits until the security owner records the outcome.
