Email marketing testing playbook Conversion rates for emails are higher than social media, direct traffic, and search. Pretty impressive. Even so, every marketer wants to do better. And in a fluctuating economy and a competitive online environment, who can blame them? But before marketers go wildly changing their regular emails in the hope of better results. We want to encourage them to take a step back before hitting send. And test. As marketers, we know that to meet our objectives we must continuously test and optimise strategies and tactics. Despite this, according to the DMA Marketers Email Tracker 2020, testing is still one of the least sophisticated areas of email marketing, with over 50% of marketers only dabbling in this key practice due to a lack of time, internal buy-in and skill set. Testing and analysis isn’t the most exciting or glamorous side of email marketing. But it is one of the most important. It stops businesses and marketers from making decisions in a vacuum or with little thought. And instead, offers them solid, actionable data and insights. To help marketers out, we have created the email marketing testing playbook. Where we will discuss: Why testing is important Different types of testing Variables to test And offer a 5 step testing plan to follow Because for email marketers, better starts here. Why is email testing important? If your emails are already performing well, then why should you need to test them? Simple. Because they could be performing better. And of course, there may be a time that their performance starts dropping. But you won’t be armed with the knowledge of how to improve them. Here are some more reasons why you should be testing your emails: Lack of standardisation In an ideal world, all emails would be created with the same rules in mind. But this isn’t the case. There are no well-established guidelines when it comes to rendering or displaying email, meaning that different servers such as Gmail, or clients such as Outlook, all handle the HTML differently. This can result in your emails looking different from recipient to recipient. And displaying better for some than others. Mistakes happen We’re all human, and we all make mistakes. Whether that’s a typo, a broken link, or mismatched font. Unfortunately, even small mistakes can make brands and businesses look unprofessional. Simple testing techniques before every send can identify those fiddly little mistakes. And give marketers the confidence to hit send. Recipients can be unpredictable It’s simply the nature of being human. That snappy subject line may work for you. But not your recipients. Or that CTA may have generated great results a year ago. But just isn’t hitting the spot right now. As audiences and marketing evolves, so do consumers’ requirements and expectations. It can be a fast-paced world for marketers to keep up with. And the only way to ensure they are getting it right with email is to test. Spam filters There is nothing more disheartening than spending time and energy on crafting the perfect email. Just for it to be blocked by spam filters. There are many reasons why this can occur. Specific words in the subject line or copy, the ‘from’ address, the ratio of images to content, and just the way the email is coded. The rules around spam filters are regularly changing and not always easily available. So to be on the safe side, spam testing is essential. Types of testing When it comes to email testing, one size doesn’t always fit all. In fact, there are a variety of tests that marketers can conduct, depending on their goals, resource, and capabilities. Basic testing Whether or not you choose to practice the more advanced testing techniques below, basic tests are something that should be completed before every email send, and don’t require lots of additional budget or resource. Content – make sure your copy reads well and there are no typos or spelling mistakes, which can look unprofessional Links – from CTAs to your social icons, make sure they all link to the correct page Rendering – ensure that your email renders correctly on both mobile and desktop devices, and ideally on different email servers and clients. A/B testing A/B testing is one of the most common, and successful, forms of testing when it comes to email. And the process that we’ll be largely referring to in the rest of this guide. Also known as split testing. A/B testing refers to creating two versions of an email to determine which performs better. While this can include two completely different emails. We recommend only testing individual elements of each email, such as subject line or call to action. So it’s easier to identify what variable is generating a preferred result. A/B testing can be performed both manually and automatically. So is perfect no matter what level of expertise or resource the marketer has access to. Spam testing There is no exact science when it comes to getting your emails delivered. And as discussed, there are lots of variables that can affect deliverability rates, such as: Data collection Image to text ratio Trigger words in copy Sending volumes Complaints Spam testing allows marketers to quickly identify any deliverability issues before their campaigns are sent out. And offers guidelines as to how they can fix any potential issues. Manual testing Manual email testing simply refers to creating different emails, ideally with only one variable changed, and sending these out to different groups of recipients, to see which performs best. Depending on the variables being tested, this can be done on a small scale. By sending to colleagues who can then report back on their findings and thoughts. But to gain more meaningful insight, marketers should aim to send these emails out to randomly selected lists of recipients. However, when conducting this process manually, marketers will have to analyse and gain insights from the results themselves. This, however, can become difficult if the marketer wishes to scale the testing. And the number of email tests, or variables, increases. Automated testing Automated testing takes manual testing a step further. It is far more accurate, and far less time-consuming. Marketers simply have to create their two email variants, select a list, and the automation will take care of the rest. It will send out the email variants to a specific percentage of the list, identify the best-performing email, and then send out the most successful email to the remaining recipients in the list. The final result is that the majority of the list has received the best-performing email. And the marketer will be armed with the knowledge of which specific variables to use within emails moving forward. Variables to test Hopefully by now you are convinced of the importance of email testing, and you have some techniques in mind. But what variables should you be testing? There are lots of options, but here are some of the most useful: Deliverability First things first. If your email isn’t even delivered, then the rest of these variables are meaningless. As mentioned, spam filters are constantly changing and evolving. Which means if deliverability isn’t kept on top of, it could begin to drop at any moment. Testing deliverability can be done using a good email automation or AI platform, which will offer you the tools and services to conduct thorough deliverability and spam checks regularly. Subject lines One of the most commonly tested elements of emails is the subject line. After all, if your subject line doesn’t engage, the recipient won’t even open your email. Some areas to consider when testing the subject line are: Length – do your recipients engage with shorter or longer subject lines? Personalisation – does adding a first name greeting improve open rates? Promotion or offer – do your recipients prefer 10% off, or free delivery? Style – do your recipients engage more with humour or mystery? Preheader text This refers to the first line of copy in your email, and often accompanies your subject line in the email preview pane. It can serve to add context to your email, and as an extension of your subject line copy. Preheader text is overlooked by many marketers, but can be a really simple but effective tool. Consider testing the following: Including copy – does the inclusion of preheader text improve your open rate? Style – similar to subject line style, try testing some humour or intrigue. Extension of subject line – do recipients prefer an extension of the subject line, or additional information? Call to action (CTA) CTAs are another overlooked element of emails. They are often short and snappy, as there isn’t a lot of space in a CTA, so what’s the point in testing? Actually, CTA results can vary drastically. So, here are some areas to test: Copy – does a simple “Buy” resonate with your audience more than “Buy these shoes”? Colour – what CTA colour catches your recipient’s eye? Don’t just choose your brand colour because it’s consistent, test a colour that stands out. Personalisation – test tailoring your CTAs to specific product ranges to make it easier for recipients to browse Urgency – test adding urgency to your CTA, instead of “Buy” try “Buy now!”. Copy You’ve gained that all-important email open. Don’t lose your recipients’ engagement with poor copy. Instead, try testing the following elements: Length – do your recipients prefer lots of information? Or short and snappy copy? Style – are your recipients informal? Or do they prefer a more serious tone to their emails? Personalisation – test dynamic content blocks in your emails to see if recipients engage more with content tailored to them Imagery Similar to copy, imagery is an element of the email that will engage recipients and encourage them to take further action. Common image variables to test include: Image/text ratio – different ratios are cited by different email experts, with a minimum of 40% image generally recommended to pass spam filters. Outside of this, play around with the ratio and see what is most engaging. Style – do your recipients engage more with photography or illustration? Do they prefer to see people or products? Colour – while brand colours help consistency, your recipients may engage more with other colours in the spectrum. Try to find a balance. Animation – will an animation or gif be more likely to catch recipients’ attention? Timing Sending times vary drastically from industry to industry. Business to business. The only way to determine the best sending time is to test your emails with your audience. So, if you have a regular time slot for your monthly newsletter. Mix it up a little. Explore whether your recipients are more engaged on a Monday morning or a Friday afternoon. Many may be checking their emails on the weekends too. The Beginners 5 step testing plan By now you should have a clear idea of the best variables to test for your emails. Our 5 step testing plan will take you through the process of conducting this testing, and generating meaningful insights. 1. Set objectives Begin by identifying what results you would like to improve from your email marketing. For instance, open and click through rates, unsubscribe rates, website conversions, or revenue. Objectives will differ dramatically from business to business. So ensure yours reflect your own business strategic focus areas. You can even develop a hypothesis based on the areas you want to test, what you will change, and what you think the outcome will be. For example: I believe I can improve website conversions by changing the email CTA, to boost click through rates and website visits. 2. Decide what to test Now you can determine what you need to test to meet your goals and hypothesis. For instance: Changing the colour of the CTA to a prominent red. And adding urgency to the copy with an exclamation mark. At this stage you will also want to consider the amount of emails that you test. Ideally, you do not want to test on a large sample size, but still require a meaningful result. So, for smaller audiences, consider 10% of your audience for testing purposes. And for larger audiences, such as 10,000+, consider roughly 5%. 3. Set your test live How you create and set your test live will depend on the type of testing you are conducting. So may be a simpler process for some businesses than others. However, we recommend that as important as testing is, don’t let it take over your marketing activity. Testing is there to complement and improve your marketing. Not to delay it. 4. Collect results Once you have conclusive results, you can prove or disprove your hypothesis. This allows you to optimise your strategy accordingly. 5. Optimise or retest If your test has produced the result that you desired, then great news. You can start optimising your future campaigns for even better results. However, if you haven’t generated the results you were hoping for. You may have to test some more. Testing is often an ongoing process for marketers who want to continually improve their results. So, it may take two of three tests to gain any meaningful insights or gain the desired results. The Pure360 ASGARDIANS Testing Methodology For those of you that want to take their testing more seriously and need to build a business case for testing, we have created a methodology that will help To ensure testing is a regular feature in your BAU email marketing strategy we recommend that you agree on a methodology that works best for your business. Sometimes this is easier said than done. When faced with little time and often a confusion about the best place to start, creating a testing methodology can seem like having to perform wizardry! Which is why, with a little bit of magic and a touch of sorcery, we have created ASGARDIANS – the Ultimate email marketing testing framework. (For those who don’t know who the ASGARD’s are there’s a link to Wikipedia at the bottom of this article) The ASGARDIANS testing framework allows you to be a brave and powerful marketer with a reputation of respect throughout your business! It is the powerful hammer in your tactical arsenal to optimise your marketing. I could go on with the analogies! However, to be honest, there’s no magic or sorcery going on here, but there’s no harm in adding a glimmer of fun to something that is often perceived as dull. In all seriousness, this email marketing testing framework allows you to:- Form a well thought out testing logic and plan Clearly document and define the success parameters Keep an accurate record of the changes you made Easily summarise the why, how and success to your boss and the rest of the business! The ASGARDIAN Framework Tried, tested and approved by brand marketers, the framework has 9 simple steps for you to follow before you go about your testing. Aim – Outline what the aim of the test is going to be. You will also define the challenge. It can be as simple or as complicated as you need it to be. E.g. “We have a declining click rate over the last 3 months, this has led to a dip in traffic from returning customers. The aim of this test is to increase click through rates and bring in more returning customers” Success – This is where you document what success looks like. This needs to be as specific as you need it to be. E.g. “Success will be if click through rates increase from 15% to 18% and returning traffic improves by 5%” Grasp – This is your current situation analysis. This is where you also document your current starting point. It may also form your control campaign. E.g. We are currently at 15% click through rate. Assume – This is your hypothesis that you will be testing. It allows you to think about the tactics you are going to implement to improve results. Make sure you keep this singular – this is not a process for multi variant testing. We recommend that you test a change to an element or a change/addition of a theme. E.g. “We assume that by adding trust messaging we will improve results by the given success metrics.” Or “ We assume by changing the colour of the buttons on the CTA we will improve results by the given success metrics” Recipients – At this point you will be deciding on who you are going to test these changes on. Or alternatively what campaigns. E.g. “I will be testing this on my post purchase emails” or “I will be testing this on Persona 1” Data – Once you have decided who. You will need to determine how much data you will need to use to give you a statistical significant result. There is no magic answer here on what that number should be and it is best that you devise your own formula and approach. E.g. “We will be splitting the data 50:50” Interval – At this stage in the framework you are deciding on how long you will run the test for. This will be totally dependent on various factors including the frequency of your emails. E.g. “We will run the test for 4 weeks.” The next 3 steps in the framework are to summarise the planning that has taken place in the previous 6 steps. This forms the elevator pitch for your testing when presenting to the rest of the business. Ambition – This is the why. Why are we doing this? A summary of your challenges and goals. New – How and what are you going to do to make these changes. Be specific about what you are changing. It also is important to document what your control is. What are you testing against. Success – Re-iterate the success metrics and what they mean for the business Document it We highly recommend that the elements within the framework are documented. Click the image below for a downloadable version of the framework for you to fill in. Are you looking to improve your email results with testing? Great idea! Whether or not you already have fantastic email results, testing can identify areas that can be improved. At Pure360, we offer an all-in-one AI marketing platform and a Customer Success Team whose sole mission is to get you better results. So get in touch to find out how we can improve your results today.