Comparing five conversion delivery models against an in-house baseline

What to take away

The five routes are in-house, fixed-scope diagnostic, managed programme, experiment on demand and performance-linked, and they differ in who owns the build, the analysis and the evidence risk.
Illustrative planning bands onlydiagnostic £2,000 to £6,000 one-off, managed retainer £3,000 to £8,000 a month, per-test £1,500 to £4,000, performance-linked 5% to 12% of incremental contribution.
An England store under 20,000 monthly sessions should keep the in-house baseline and buy at most one fixed-scope diagnostic a year.
My pick for 20,000 to 100,000 monthly sessions is experiment on demand, because every test is pre-specified and the counterfactual stays reproducible.
Consumer information, accessibility, privacy and payment duties stay with the merchant under all five routes.

The five delivery models and who delivers each

The five delivery models

Representative provider

In-house baseline: Your own analyst and front-end developer, on Shopify or Adobe Commerce
Fixed-scope diagnostic: A specialist CRO consultancy selling a paid audit
Managed programme: A full-service digital agency on retainer
Experiment on demand: A licence for Optimizely, VWO or AB Tasty, run by a contractor
Performance-linked: An agency paid on incremental contribution

Merchant retains

In-house baseline: Definition, build, analysis
Fixed-scope diagnostic: Implementation and the decision
Managed programme: Product ownership and final approvals
Experiment on demand: The test queue and deployment authority
Performance-linked: Legal responsibility and outcome acceptance

Main evidence risk

In-house baseline: A capacity or skill gap stays hidden
Fixed-scope diagnostic: Advice is mistaken for a tested effect
Managed programme: Supplier process blurs accountability
Experiment on demand: One isolated test ignores journey dependencies
Performance-linked: Attribution becomes a payment dispute

Five conversion delivery models

In-house

Merchant retains: All
External delivery: None
Evidence risk: Hidden gaps
Exit test: Reproducible records

Diagnostic

Merchant retains: Implementation
External delivery: Audit
Evidence risk: Advice as effect
Exit test: Clean export

Managed

Merchant retains: Ownership
External delivery: Operations
Evidence risk: Obscured accountability
Exit test: Transfer assets

On-demand

Merchant retains: Queue
External delivery: One test
Evidence risk: Isolated test
Exit test: Repeatable config

These are procurement models, not measured market shares.

What each route costs against the in-house baseline

Figures are illustrative planning bands, not quotes. No supplier price was examined, and the internal figure assumes you already employ the people.

In-house baselineno external fee, but an illustrative internal cost of £4,000 to £9,000 per decision cycle in analyst and developer time.
Fixed-scope diagnostic£2,000 to £6,000 one-off, delivered in four to six weeks.
Managed programme: £3,000 to £8,000 a month, usually on a six or twelve month term.
Experiment on demand£1,500 to £4,000 per pre-specified test, plus the platform licence.
Performance-linked5% to 12% of incremental contribution, or a fixed success fee of £5,000 to £20,000.

The seven conversion pricing structures table measures retainer, per-test and success-fee bases on identical meters, which is the fair way to compare them.

Example: one decision cycle for a Leeds store

This is a worked illustration, not a real merchant or a measured result. Take a Leeds garden furniture retailer with 45,000 monthly sessions and an evidence file untouched for a year.

One decision cycle

Baseline. Record basket-to-order completion for 30 days and freeze the event dictionary.
Approved change. Move the delivery charge above the payment button in the basket summary.
Controlled exposure. Split sessions 50/50, with a stopping rule of 2,000 sessions per arm.
Analysis. Read the predeclared primary metric plus a guardrail on refunds and contacts.
Rollback. Revert the change if the guardrail breaches, before the analysis is published.
Close-out. Export the configuration, the raw data and the decision note into the evidence file.

Under experiment on demand the contractor runs steps 2 and 3, and the merchant keeps the rest.

Which model suits which merchant

One rule per size band, and no exceptions for confidence.

Under 20,000 monthly sessionsthe in-house baseline, plus one fixed-scope diagnostic a year if the team is thin.
20,000 to 100,000 monthly sessions with a front-end developer: experiment on demand.
Above 100,000 monthly sessions or multi-market trading: a managed programme with a named internal owner.
Performance-linkedonly where the payment event can be reproduced without argument.

Before shortlisting anyone, read why this site's conversion provider shortlist is empty and what supplier evidence would fill it.

Contracts, tracking duties and the evidence file

The Consumer Contracts (Information, Cancellation and Additional Charges) Regulations 2013 set the checkout disclosures, and those duties do not move with the contract. Consent for cart recovery emails sits under PECR, so read the ICO guide to PECR before an abandoned-basket sequence goes live.

Put these lines in the file before you sign:

Predeclared population, numerator, denominator, allocation unit, period, stopping rule and uncertainty method.
A named owner for each tracking purpose, activation approval, rights requests and deletion.
A counterfactual, exclusions and final calculation that survive an independent review.
Non-compensating approvalsproduct safety, consumer information, accessibility, privacy, security, payment and tax.
An exit test that transfers open work, assets and histories.

A performance-linked fee is the fragile case. HM Treasury's impact evaluation guidance separates observed change from credible attribution, which is the split a success fee depends on. Payment on placed orders is the weakest version, because cancellations and returns arrive after the invoice. If neither side can reproduce the counterfactual, use a fixed fee.

Work through the conversion supplier contract questions before signing, because exit terms are negotiated last. The A/B testing reference covers allocation, significance and stopping rules.

Common questions

Which model should a small England store choose?

The in-house baseline, run to the same decision-cycle standard, plus one fixed-scope diagnostic a year. That keeps the evidence file under your control until session volume justifies a retainer.

How much more does an external model cost than in-house?

On the illustrative bands, a managed retainer of £3,000 to £8,000 a month overtakes an internal cycle of £4,000 to £9,000 after roughly two cycles. Add your own review time to every external figure.

What proof shows an external model beat my baseline?

A predeclared counterfactual with the same population, metric, period and stopping rule, reproduced by someone outside the delivery team. Without it, the result describes performance rather than comparing models.

When is a performance-linked fee worth the argument?

Only where the payment event is a retained order or a contribution figure that both sides can recalculate from source data. If the attribution window is still contested, a fixed fee is safer.