Minimum Viable Testing: 5 things you are doing wrong when testing new marketing channels

Everyone is a growth hacker these days. Startups are popping up right, left, and centre. Series A and above funding seems to be on the rise and titles such as Head of Growth or Growth Hacker can be found anywhere. It certainly doesn’t hurt that these can be some of the best paid positions out there.

Over the last ten years marketing has changed a lot. We went from a discipline that was only really practiced in agencies at scale to a fully democratised workforce where everyone can be a marketer. Self-serve platforms such as Google and Facebook certainly contributed to that.

At the same time with the arrival of digital media we collectively went from marketing being a dark art to something much more measurable and scientific. Or at least it seemed that way. 

In reality, the availability of numbers and measurement can often be a bit of a trap as bad numbers = bad decisions. Its just simple math.

This is particularly important when it comes to growth and new channel testing. Testing new channels can be one of the most challenging but at the same time most rewarding aspects of the role of a performance marketer. It’s where the art and the science meet in our profession.

Art, because realistically you will never have all of the data you need in advance. Science, because you will have to be quite methodical in the way you prioratise, set up, and measure.

Testing new channels can be one of the most challenging but at the same time most rewarding aspects of the role of a performance marketer. It’s where the art and the science meet in our profession.

This is where the concept of Minimum Viable Test can be quite handy. 

Most will be familiar with Minimum Viable Product. This is essentially planning and releasing  the smallest set of features that allows a product to a) be functional b) test the most basic set of hypotheses around market fit.

It takes time and, more crucially, money to build and ship a feature so strategic design and ruthless prioratisation is key.

Its similar for marketing.

Minimum Viable Testing for marketing channels

MVT is the methodology that allows us to plan design setup and deploy new channel testing in the most efficient way as possible.

Minimum viable testing design should include planning around timescales, budget, as well as measurement.

Whilst most marketing teams will undertake some version of the above preparation what sets  minimum viable testing above and beyond any other planning, is how systematic the approach needs to be in order to make the maximum out of any new growth initiative.

MVT is the methodology that allows us to plan design setup and deploy new channel testing in the most efficient way as possible.

Having worked with over 100 brands and done more experiments than I care to coutnt  in the course of the past decade I quickly saw that there are some clear patterns that emerge. 

Here is a by no means exhaustive list of the most common mistakes  growth marketers make when trying to discover new channels. 

  1. Too little

This is by far the most common issue with new channel testing and for a good reason. Everyone wants to bet on a certain thing. No company wants to waste tons of money on something that might not succeed.

That said, there is a very fine line between being cautious and simply invalidating your test by spending too little on it. There is a reason we put the word “viable” right in the middle of Minimum Viable Testing.

Here is why spending the right amount is one of the most important decisions you can make.  

Statistical significance should be both the goal and key guardrail of any channel testing.  By designing a test to be statistically significant it means that your results are valid and can be used to interpret whether the channel is viable. Once you have a clear result you can decide to continue running the channel, invest more, or pause the activity.

If you don’t achieve statistical significance you might have to discard the whole test meaning that your money and effort have been wasted.

How can you make sure that statistical significance is within your reach?

It’s simple yet effective: you need to invest the right amount of money on the right goal, over the right period of time.

We will focus on the right campaign duration and optimal KPIs setting later in this piece but for the time being let’s look at budget.

In order to define the right amount you will need to invest to see statistically significant return you need to ask yourself things like what channel am I testing and what am I  measuring?

As a rule of thumb digital channels require smaller amounts of investment than offline ones.

For something like  Facebook sales based activity for example a good starting point would be around £5,000 per month when optimising for low to mid basket purchases. There are scenarios where you could invest less but this amount is a safer bet when it comes to statistical significance.

For something like Out-Of-Home on the other hand, you might be looking at a minimum of £50,000 per month per location. That is, of course, a large difference and  may potentially be a deciding factor in your channel prioritisation.

Naturally, the earlier in the user journey a KPI is, the faster you will be able to reach statistically significant results.

This is because you will always lose users in the process of getting closer to conversion. For example, you will typically have more clicks than unique users on the website. By the same logic you will have more unique users than purchases.

If you are therefore looking for statistically significant results at the purchase level it will take more budget and time. 

Here are a couple of statistical significance calculators; this one is a simple version, whilst this one has a bit more complexity.

  1. Too soon

Time matters.  One of the worst mistakes you can make is being too impatient.  

Time frames will vary by channel however in most cases you will need to run a test for at least two weeks before deciding whether you’re going to switch it off and move on.

Two weeks is really the bare minimum of time you will need before you’re able to make an early assessment. For a full channel assessment you need longer than that as most channels take at least two weeks to start optimising.

The other factor that will have an impact on your timeline is the KPI. Typically, KPIs such as brand awareness take much longer to move the needle therefore it is very unlikely that you will see any significant results early on.

Two weeks is really the bare minimum of time you will need before you’re able to make an early assessment. For a full channel assessment you need longer than that as most channels take at least two weeks to start optimising.

Going back to the notion of significance as as it has been mentioned before there’s very little point in switching off a test before  achieving significant numbers. Not only will it be a waste of time and money but more importantly if these results are later used to describe the channel as a viable option the wrong decision based on wrong data might have an impact on your business’s long-term growth

  1. Doing what everyone else is doing

“But it worked for the other guys…”

Just because a channel is working well for a competitor he doesn’t mean it’s going to work well for your business.

Before making a decision based  on your competitor set you really need to invest time in making sure your definition of competitors is indeed applicable.

Here are some factors you should look into:

Service: Are they offering exactly the same service? are there any fundamental differences that could lead to a different user behaviour and preferences? 

If the answer is yes then proceed with caution. 

Audience: Is the audience composition of your competitor  identical to yours? Are there any differences you should be exploiting. If your audience, for example, is younger than your competitors audience then maybe you should be looking at a very different channel mix.

Saturation: If 50 of your competitors are advertising on a  bid based channel, then it might be worth exploring less competitive avenues.

  1. Not setting the right KPIs

If you judge a fish by its ability to ride a bicycle then you will always be disappointed by the results.

KPI setting can be a challenge for even the most experienced marketers out there.  The aspect of KPI setting that most companies get wrong is that the KPIs should match your channel not just your business goal.

Most businesses want to create sales. This, or similar, should be the North Star metric for all of your activity.

However, when it comes to channel testing having something like sales as your primary KPI only be suitable for certain types of channels. You might be missing opportunities to grow your business by looking at the wrong places.

Fundamentally, once you start moving higher up the funnel it becomes harder to assign sales to your marketing activity.

You might be missing opportunities to grow your business by looking at the wrong places.

For activities such as display or Out Of Home metrics like reach will be more suitable

In order to help you match the right measurement to the right channel, I have created the below chart.

  1. Experts

Thats great but we dont know what we are doing.

This is one of the most challenging areas especially for new companies or decision leaders without extensive connections in the world of marketing.  Just because somebody is great in social media it does not mean that they have the knowledge to set up and test a channel like Audio or Out Of Home.

A lot of businesses do not have the right knowledge or resources to test some of the channels that could bring most of the growth.

Inevitably the question arises: Do we go out in the market to find an agency or do we try to hire a freelancer?

Neither of these are easy choices. Agencies have massive overheads and therefore might be incredibly expensive.  As a rule they also have quite a high minimum spends there for the entry barrier for an initial test may become very quickly very high. 

Just because somebody is great in social media it does not mean that they have the knowledge to set up and test a channel like Audio or Out Of Home.

Agencies are great at standardising work and creating processes but in this case you might be in need of something a bit more agile; speed is of the essence.

Freelancers on the other hand, can be notoriously difficult to source and on offline channels there may not be an option. Its the Wild West out there with a lot of people claiming that they know what they are doing with very little experience at all.

This is one of the key challenges I was facing when I was the head of performance marketing for a start up a few years ago.

 I needed to test and scale and number of channels I was not a specialist in.  I didn’t have the knowledge within my team and there was no platform I could trust to help me source the right expert.

As a result I ended up creating the service I wish I could have used and Adadot was born. 

At Adadot we do all the heavy lifting for you so that you can discover new channels with confidence.  We link you up to the best experts from all over the world and make everyone accessible through one single platform. Different disciplines come together as a single team working on your project. The benefits of a network agency without the overhead.

If you are planning to grow your business online in the near future keep in mind the above common mistakes and dont hesitate to give us a shout.

Categories:Startups

Tagged as: ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: