Split testing looks simple — show two versions, measure which performs better. The ways it goes wrong are consistent and mostly statistical.
The basic method
Visitors are randomly assigned to variants, outcomes are measured, and the difference is assessed for statistical significance.
Randomisation is what makes this a genuine experiment rather than an observation, and it is why the results support causal claims that analytics cannot.
Stopping early
The most common error and the most damaging.
Checking results repeatedly and stopping when significance appears dramatically inflates the false positive rate.
Because with enough looks, random variation will cross the threshold at some point regardless of whether a real effect exists.
Which means a substantial proportion of declared wins in casually run testing programmes are noise.
Fixing the sample size in advance, or using sequential methods designed for repeated checking, addresses this.
Sample size
The reason most small sites cannot run meaningful tests.
Detecting a small effect requires a large sample, and the required sample grows rapidly as the effect size falls.
Which means a site with modest traffic can only detect large effects, and a test showing no difference may simply lack the power to find one.
Calculating required sample size before starting tells you whether the test is worth running, and it frequently is not.
Multiple comparisons
Testing many variants or many metrics simultaneously increases the chance of a spurious result.
Which is why testing one hypothesis with one primary metric is more reliable than testing ten things and reporting whatever moved.
Declaring the primary metric before the test removes the temptation to select afterwards.
Novelty and primacy effects
Returning visitors respond to change itself, which can produce a temporary effect that fades.
Which means short tests on sites with high return visitor rates can measure reaction to novelty rather than to the change.
Running for full business cycles, generally at least a couple of weeks, and segmenting new against returning visitors both help.
Seasonality and external events
Running a test across a period with unusual traffic distorts results if the variants are affected differently.
Which is generally handled by randomising continuously rather than by time, so both variants experience the same conditions.
What actually moves conversion
Evidence across many programmes points at the same areas repeatedly.
Page speed, where the relationship with conversion is well established and the effect is larger than most design changes.
Removing steps and fields from checkout, since each is an opportunity to abandon.
Clarity about cost, particularly delivery charges, since unexpected costs at checkout are the most cited reason for abandonment in surveys.
Trust signals where the purchase carries risk.
And mobile experience, which is where the majority of traffic is and where problems are most common.
The realistic expectation
Most tests show no effect. A small proportion show meaningful improvement.
Which means the value of a programme comes from running many tests rather than from any single one, and that requires enough traffic to make it viable.
For lower-traffic sites, qualitative research — session recordings, user testing, customer interviews — generally produces more useful direction than underpowered tests.
Personalisation
Showing different content to different visitors based on their characteristics or behaviour.
Which sounds obviously beneficial and frequently is not, since it fragments traffic into segments too small to measure and adds complexity that introduces errors.
Evidence for personalisation producing meaningful uplift outside of large-scale operations is thinner than the tooling market implies.
The version that reliably works is the simplest — showing returning customers something different from first-time visitors.
Qualitative methods
Where the hypotheses should come from.
Session recordings show where people hesitate, scroll past and abandon.
Moderated user testing, with a handful of participants attempting a real task, reliably surfaces problems that quantitative data indicates without explaining.
Customer support enquiries are a free source of exactly this, since repeated questions identify what the site fails to communicate.
Documenting results
Recording every test including the failures builds institutional knowledge and prevents the same idea being retested annually.
Which is the difference between a programme and a series of experiments, and most organisations never do it.
Technical foundations
Before testing anything, the basics.
Load speed, particularly on mobile connections, has a well-established relationship with conversion and is measurable with free tools.
Broken checkout paths on specific browsers or devices are more common than teams assume and are found only by testing on real devices.
Which means a technical audit frequently produces larger gains than a testing programme, at lower cost, and it is where to start.
Trust and friction
Guest checkout, clear delivery costs shown early, and visible contact details all address specific documented reasons for abandonment.
Requiring account creation before purchase remains one of the most costly common decisions in online retail.
Which is a documented and easily reversed decision, and it remains in place at a surprising number of retailers.