Have you ever had a bump in your organic traffic and then everybody jumps up and says it was something that they did last week?
Marketing says it was their outdoor campaign.
Engineering says their recent page speed improvements were the cause.
Product says they added new product review features.
SEO says Google recently made some algorithm updates.
The data scientist tells everyone to calm down because it was just seasonality.
Funny how no one jumps up when the numbers go the wrong way - the SEO people usually get the credit for that one.
It could have been all of these, but how do you know how much each contributed?
SEO Experiments!
An SEO experiment involves making a change to a random percentage of pages on your website and observing the difference in organic traffic between pages with the change and those without the change.
It’s quite different to a normal A/B test as the split is done based on pages rather than users.
Who should be running SEO experiments?
1. Your website needs to have a significant amount of traffic
If you don’t have much traffic, you’re going to be waiting a very long time to get a statistically significant result!
2. Your website needs to have a significant amount of pages with the same page template
This usually occurs when each page represents some entity: cars, houses, products, jobs, people, restaurants, places of interest, locations etc.
3. The distribution of traffic for each page can’t be too skewed to a handful of pages
If 99% of the traffic goes to only a couple of the pages from that page template, then you’re going to have a problem. There’s always going to be a skew and a long tail of traffic to less frequently visited pages but there needs to be enough pages with enough traffic.
Why should you run SEO experiments?
The reality is no one really knows how Google’s search engine works. The people who built it probably don’t understand what the deep learning model is actually doing.
This is known as the interpretability problem in the field of AI and machine learning.
Google’s search algorithm is a black box just like many machine learning models - even to its own creators.
To figure out the black box, the best companies in the world use experiments.
The main reason to run experiments is to learn. You want to learn what the effect is of your actions or changes. If you just change things on the page and hope for the best, you’ll never know if what you did actually worked. Even if the numbers did go up, it could have been due to so many other things.
If we learn what changes bump up our traffic the most, we can optimise faster and with less wasted effort implementing changes which have no effect.
Our new topic: SEO Experiments
We’ve just launched a new topic on supercooldata specifically to go through how to analyse SEO experiments.
A realistic case study of a page change SEO experiment with a realistic dataset is included in our latest topic. We go through how to analyse the data found in your company’s data warehouse including event tables such as those from Google Analytics or Segment to arrive at a conclusion as to the effectiveness of SEO improvements implemented.
If you want to rank like the best tech companies in the world particularly for long tail keywords, then you need to be running SEO experiments for almost any page changes you’re making.
Check out this insightful post by Pinterest and this Twitter thread by Lenny Rachitsky (former Airbnb Product Lead) to learn how the best tech companies do it.
To learn how to analyse SEO experiments with SQL, sign up at supercooldata.com.