A/B Testing Tool

Finally, an A/B testing tool for nonprofits and change-makers that’s simple to use! For many, the biggest barrier to testing is ensuring that results are statistically significant. We’ve broken down the testing process into easy steps, and developed sample size and significance checking tools that are easy to use. Part of developing an easy-to-use tool was making assumptions. You can read all about the assumptions in our FAQ.

Now let’s get testing!

Step 1: Decide what to test

First you’ll need to figure out exactly what you want to test. Is it a subject line? A call to action?

Step 2: Determine the sample size

If you want your test results to be reliable and statistically significant, sample size is important. Use this calculator to figure out what your recommended sample size is.

For the first question, select “Very sure” if it would be a very big deal if your results were wrong (like if you’re hoping to make a universal change to your email template, for example). Only use “Somewhat sure” if it's a one-time test (like a subject line test) and you're not super concerned about your reliability of your results.

For “Expected response rate” go with your best guess for the open rate, click-through rate, or whatever you're planning to measure.

Use the next question to evaluate how big of a difference you expect this test to make for your results. It's typical to see increases around 15% for subject line testing and increases around 25% for email draft testing. If you're not sure, start with Medium.

Sample Finder

If the suggested sample size is impractical, think about what else you could test. For example, if the size of the test group is impossibly big for action rate, consider testing the click-through rate instead. Check our FAQ for more info.

Step 3: Run the test in your email platform

Next you'll need to log into your email platform, create your test groups (making sure they're each as big as the suggested sample size at a minimum), and run your test. Come back to this page when you have your results. Check our FAQ for more info about running the test.

Step 4: Check your results

Are your results statistically significant? Use this calculator to find out. Be sure to select “Very sure” if you need to be extremely sure your results are accurate (like if you’re hoping to make a universal change to your email template).

Significance Inspector

If your results are statistically significant, congratulations! That means you can confidently roll out the winning version to the rest of your list. If you're testing a universal change (like a change to a template), you should run the test a few more times first.

If your results aren't statistically significant, learn about why this could be in our FAQ.

Have questions about the assumptions we made when creating these calculators? Check out our FAQ.

Click here to share your experience with the A/B testing tool or to report any issues.