Simplify Your Testing Process

“Do not let trifles disturb your tranquility of mind…. Life is too precious to be sacrificed for the nonessential and transient…. Ignore the inconsequential!” – Grenville Kleiser

You’ve got a top-notch copywriter – one who’s made tens of millions of dollars for his clients. He’s at your disposal, ready to answer your biggest questions about what makes a direct-mail sales package attractive to a prospective customer. Do you ask him about the best color for a headline? Or the best type style to use? Or the best paper stock?

During a question-and-answer session at ETR’s recent Info Marketing Bootcamp, copywriting expert James Sheridan warned against getting bogged down in such things.

“Sure, you can test them,” said James,” but those are details that don’t matter.”

Testing is widely accepted as one of the most powerful tools a direct marketer can use. But testing insignificant choices is one of the most common mistakes direct marketers make.

In an effort to “perfect” or “tweak” a promotional package, they test all kinds of things that have no real chance of producing a meaningful change in response rate. What they get is usually in the plus or minus 10 percent to 25 percent range.

That range is enough for many. For me, it’s inadequate.

I am not a statistician, so I can’t make the statistical argument – but I can tell you this: Getting a 10 percent or even a 25 percent lift on a test cell of 5,000 or even 25,000 names is not always reliable. It may sometimes be enough to validate going forward with something you want to do anyway (say, changing your price or your guarantee or the bonus you want to offer), but it may not give you the level of confidence you are probably looking for.

I’ve had 20 years of experience in planning and reacting to more than a billion dollars’ worth of direct-mail testing. I’ve seen – and have been responsible for – just about every kind of test you can imagine. During that time, I came up with all kinds of interesting theories that I proved through testing, satisfying myself with 25 percent lifts, only to see those results negated on back-tests.

This happened not just once but hundreds of times. In fact, I think it’s fair to say that back-testing nullified my prior tests more times than not.

It’s Amazing How Many Foolish Tests People Will Spend Money on

A young copywriter has the idea that several paragraphs in a sales letter are so bad that they must be rewritten and retested. The results come in – and, sure enough, the new package outpulls the old by 28 percent. A month later, however, the product manager screws up and mails the “inferior” copy, and guess what? It beats the new version by 22 percent.

An art director wants to demonstrate that his trendy new typeface will make a difference. He gets a 15 percent lift and is satisfied.

A product manager wants to prove to his client that he is right about something – almost anything – and creates a test to vindicate his point of view.

There are many reasons for testing insignificant options. Yet, every test costs money. Every test reduces profits.

I remember when I was younger and more foolish. My mentor, JSN, and I noticed that all the really successful packages for financial products at that time had one thing in common: the color scheme. They all used black and burgundy ink on cream-colored stock. We rushed to test it, and got a 30 percent lift. It was a “Eureka!” moment.

We immediately converted all our packages for financial products to this “ultimate” color scheme. Luckily, we also did a little back-testing.

It’s a little like looking at an oil stain on an old window. If you really want to see the visage of the Virgin Mary, you will.

A Simple Rule of Thumb

After thinking about this problem for a long time and reading some serious statistical stuff recently (that I only half understood but that validated my experience), I came up with a rule of thumb. I offer it to you for consideration: Test nothing except that which can potentially give you a 100 percent lift in response.

It’s radical thinking, but it does two things that are very good for your mailing results:

1. It dramatically reduces useless testing and, thus, increases your profits.
2. It forces you to do the hard thinking necessary to come up with truly breakthrough testing ideas.

[Note: There is one exception to this rule. If you are considering making a change for the purpose of making the package less expensive to mail – e.g., changing the envelope size to reduce postage costs – you must test it first to make sure it doesn’t decrease response. In this case, you would be testing not in an attempt to lift your response rate but to make sure you’re not going to decrease it.]

The Question Is This: What Can Possibly Double Response Rates?

Well, first I’ll tell you what won’t work: Changing your color scheme, paper stock, or type size. Making your body copy smoother or cleaner. (Forget Strunk & White.) Correcting your data. Substituting a Johnson Box for a regular headline. (You’ve seen a Johnson Box: an outline of a box on top of the salutation inside of which the headline and/or lead goes.) Changing the person who signs the sales letter. Adding marginalia. Using a live stamp vs. an indicia vs. a meter. Changing the response method. Using testimonials. Using a product photo vs. an illustration. Your selection of premiums. Most forms of personalization. Individual involvement devices. Most lift notes. A BRE vs. a self-mailer. And (usually) providing an 800 number.

I’m not saying these things don’t matter. They do. When you design your direct-mail sales package, you need to keep all of them in mind and use your judgment to make the most appropriate choices. But though they matter as a group, they are insignificant individually. So don’t test them unless you have a very good reason to… unless you think they can make a very big difference.

What CAN double your response rate?

• your headline
• using teaser copy on your envelope
• the subject line in your e-mail solicitation
• your lead paragraph

These are the most important copy tests, because they incorporate – in the most visible way – the fundamental promise of your promotion. The most important job of the copywriter is to choose the “Big Promise” of the promotion. If you get that right, you are 80 percent home.

What else can make a difference?

• your offer (an often-overlooked element)
• the length of your sales letter (longer is usually better)
• the size of your package
• the number of premiums offered

There may be a few others – but the point I’m trying to make here is that the number of things you should test is (1) probably smaller than you thought and (2) certainly smaller than you’d surmise if you listened to most experts. Remember, most of those direct-marketing experts are copywriters who see, relatively speaking, very few test results and aren’t responsible for the bottom line.

[Ed. Note: Michael Masterson is one of the marketing experts behind ETR’s brand-new Internet business-building program. This breakthrough program gives you an all-inclusive, A-to-Z blueprint for starting your own powerhouse Internet business – from learning how to pick a product and set up a website to discovering copywriting secrets from the masters, techniques to help you create an e-mail list, the best ways to market your product, and more. Getting your name on our hotlist is likely to be your only chance to get into this very limited program, which we expect to sell out quickly.] [Ed. Note: Mark Morgan Ford was the creator of Early To Rise. In 2011, Mark retired from ETR and now writes the Palm Beach Letter. His advice, in our opinion, continues to get better and better with every essay, particularly in the controversial ones we have shared today. We encourage you to read everything you can that has been written by Mark.]