A/B Testing is the process of running an experiment to see if one way of doing something provides an advantage over another.

For example, the most common A/B Test is to test whether a change in a creative increases the CTR.

There are three crucial elements to doing A/B Tests properly:

  1. You need to split your audience into two (or more groups), and expose them to the different test conditions.
  2. You need to make sure that everything else about the groups remains constant.
  3. You need to do some simple statistics to test whether any difference is significant.


The test you need to do is called a Chi Squared test of statistical significance.

Use Even Miller's Awesome Online Tool to do the test.

This test will produce a statistic called a p-value.

Simply stated the p-value indicates the probability that the difference you see could have occured by chance.

Interpreting Your P-Value

If your p-value is very low, i.e 0.001 or lower you can be fairly confident that you are looking at a genuine difference created by the change in the creative. Awesome !!!

If your p-value is larger that 0.1 then you are getting into the territory of uncertainty.
There are two possibilities:

  1. You do not have enough data to measure the effect.
  2. There is no difference between the two groups.


To know the answer to this you need to calculate the statistical power of the test you did Evan Miller has another awesome tool for estimating this