Conversion is an anomaly

Depending on who you believe and the context, average site eCommerce conversion rates vary between 0% and 12%. That's not very helpful. In my own experience, defining conversion as number of completed checkouts divided by total number of site visitors, that rate varies between 0.20% to 2.00%.

That fact has important implications for analysis, bias, and making causal statements about what causes conversion.

Specifically:

• When doing an experiment, the lower the conversion rate, the greater the number of visitors that are required to make a truthful causal statement that something causes conversion.

• As a consequence, poorly converting sites that could benefit from experimentation the most are the most disadvantaged.

• Methods that are more common in the machine learning community may actually be more appropriate than what we'd call 'traditional statistical analysis'.

As A Result:

• If the traffic to a given site is low, it is even more important to test big things that matter, than it is to fiddle with something likely to be trivial. Take big risks.

• It is preferable to increase the efficiency of the site by converting visitors into customers than it is to incur high incremental costs from driving more unqualified traffic.

• We may have more success if we treat conversion as an anomaly detection problem as opposed to a regression problem.