An A/B testing campaign has four statistics: one for version A, one for version B, a comparison of both versions with information about the winner and the overall results of the campaign. You find them in the campaign's left menu on the Statistics page. The comparison of the versions tells you the most, it shows right at the top which version won.

How to read the comparison of versions A and B

  • The first section shows the winning version, its subject line and the time interval you set for picking the winner.
  • The Overall overview part shows the main statistics of both versions in one chart.
  • Next comes a comparison of the results of version A and version B in percentages.
  • Charts show how the individual statistics developed over time, separately for version A and version B.
  • The last part compares the clicks on individual links in both versions. Look here for the answer when you tested buttons or links.

Be strict when you read the result. A difference of a few tenths of a percent in a small test group is not a win, it is chance. Treat only a difference that repeats as a finding. Write down your test results, for example in a shared spreadsheet: what you tested, the size of the groups and the result.

What the statistics of version A and version B show

The statistics of each version start with the Overall overview with these figures:

  • Sent: the number of unique recipients the version was sent to.
  • Delivered: the number of unique recipients it was delivered to.
  • Not delivered: the number of unique recipients it was not delivered to. The list with the reasons is at the bottom of the page, the article Reasons for undelivered emails explains them.
  • Opened in total: the number of all opens. If one recipient opens the email 10 times, the number is 10.
  • Opened uniquely: the number of recipients who opened the email. Each one is counted only once.
  • Not opened: the number of unique recipients who did not open the email.
  • Clicks in total: the number of all clicks on links. If one recipient clicks each of 5 links once, the number is 5.
  • Clicks uniquely: the number of recipients who clicked at least one link.
  • Social media icons: for each icon in the email, the number of unique recipients who clicked it.
  • Marked as SPAM: the number of unique recipients who marked the email as spam.
  • Unsubscribed: the number of unique recipients who unsubscribed using the link in the email footer.

Below the overview are the percentages for deliverability, opens and clicks, click statistics by link and lists of recipients who opened the email, did not open it, clicked a link or unsubscribed. The article What the terms in statistics mean explains the terms in detail.

What the overall statistics show

The fourth statistics, Overall statistics, add up the results of version A, version B and the sending of the winning version to the remaining recipients. They show the result of the campaign as a whole and are useful for comparing it with other campaigns.

How to export the statistics

You export the statistics to Excel as an xlsx file. Each tab has its own export:

  • Version A and version B: the same export as for a simple email campaign, for the given version. It contains an overview with the name, subject line and time of sending, contact lists with the company, tags, time of the action and device, the daily trend, links and reasons for unsubscribing.
  • A/B campaign comparison: a comparison overview, percentage statistics, the daily trend by version and the URL comparison. Each URL is on its own row and the social sharing columns no longer overwrite the unsubscribe column. This export has no contact lists.
  • Final statistics: an overview with the campaign name, the daily trend, contact lists with the company, tags, time of the action and device, reasons for unsubscribing and a comparison of links by version.

Percentages are numbers with a percentage format in all exports, so Excel calculates with them right away.

How to set up the next test is described in Setting up mailing in an A/B testing campaign.

Do you have questions? We will be happy to answer them. Just contact our customer support at +420 605 163 892, e-mail: support@onquanda.com.

Frequently asked questions

Four. Statistics for version A, statistics for version B, a comparison of both versions and overall statistics for all sent emails, including the sending of the winning version.

In the statistics comparing versions A and B. The very first section shows the winning version, its subject line and the time interval you set for picking the winner.

A difference of a few tenths of a percent in a small test group is more likely chance. Treat only a difference that repeats in further tests as a finding. Larger test groups help too.

The overall statistics add up the results of version A, version B and the sending of the winning version to the remaining recipients. They show the result of the campaign as a whole.

Less than elsewhere. Quanda filters out more than 80% of the automatic opens created by Mail Privacy Protection in Apple Mail. When you test content or buttons, pick the winner by clicks, which automatic opens do not affect.

Yes. You export the statistics to an xlsx file that contains all data from the page arranged into sheets.