Controlled comparative studies

Controlled comparative studies (often called A/B testing or split testing) can be done in several different ways, but they all have similar characteristics. These tests will give you quantitative data on how well the public uses your content.

Before you do a controlled comparative study, you should do paraphrase testing or usability testing and refine your writing based on what you learn in these smaller scale studies. Controlled comparative studies (especially for paper documents) are best near the end of the process. This is because controlled testing will tell you if your content is working, but it won’t tell you why.

Define your goals

Before you do a controlled study, you should know what results you will consider a success. For instance:

• Do you want more calls or emails regarding a certain program?
• Do you want fewer calls or emails asking for clarification?
• Do you want more people to return an application or a payment?
• Do you want fewer errors on forms people fill out?

Having answers to these questions will help you determine which version of your document or web page is successful.

Testing websites

Set your web server to send each variation on a specific schedule (every other call to the page, or one today and the other tomorrow, or one for a certain longer period and the other for the next equal period). Just be sure you can track whatever measure you want by which version the web visitor saw.

Testing print documents

Send a small test group of people the new version of your document. Let’s say you’re sending the new version to 700 people. You should also send 700 people (your control group) the old document. Track the responses to all 1400 documents and compare the results.

Note that it is much easier to test results when people return a written response than when you try to track the number of phone calls you receive. (If you have a statistician or actuarial staff, they can tell you how many people you should use to make your study scientifically valid. If your agency doesn’t have an expert on staff to help you, statistics books will give you a formula to determine a good sample size for your study.)

There are numerous other ways of collecting quantitative data. For instance, you can record what percentage of your “before” letters generates correct responses compared to your “after letters,” or what percentage of each letter results in your customer calling or emailing you for an explanation.