Lower Head


E-Marketing Performance Blog

The "So What?" Test

A great post by Avinash Kaushik teaches us a simple test to use to identify if an analytics metric is right for your business.  It’s the “So What? Test” and it simply involves asking that question up to three times when you make an observation about a particular metric. The kind of metrics you’re looking for will measure how you’re doing with your business goals and lead to recommendations to move the dial.

Let me try and show this with a couple of my own made-up ones and one from the post.


Observation: The bounce rate on our homepage fell by 30% last quarter.

So what?

Visitors are believing more that they are in the right place because they are finding what they’re looking for.

So what?

We should implement this, this and this like we did on our homepage to some of the other top landing pages on our site.

Sweet, this works because our observation leads to action.  This metric is right for your business and can be used as a KPI (Key Performance Indicator) of how your business is doing in light of its goals.

Observation: We had 1,000 more visitors this month than last month.

So what?

More people on our site is a good thing.

So what?

More people means more conversions.

So what?

More conversions means more money.

While it may be good that you’re getting more traffic, it may not be.  First of all, this metric is not driving any action.  Second, it gives us no insight at all into why the observation happened.  Did it really lead to more money?  How do you know?  All of the extra visitors you got on your site could have all bounced cause they weren’t looking for what you were providing and now you think that what you are doing is working, but really it’s not.

Here’s my favorite one from Avinash’s post:

Observation: Our task completion rate is down five points this month to 58%.

So what?

Having indexed our performance against that of last quarter, each one percent drop causes a loss of $80,000 in revenue.

So what?

I have drilled down to the Primary Purpose Report and most of the fall is from visitors who were there to purchase on our website.  The most likely cause is the call to action on our landing pages and a reported slowness in response when people add to cart.

Cha-ching! This is where love in your boss’s/client’s heart for you lies.

A good metric will answer business questions and supply you with actionable ideas that recommend you at least experiment with doing more or less of something.

Mike Fleming

Michael Fleming

One Response to The "So What?" Test

  1. The ‘So What?’ Test reminds me of 1990’s-era management guru Peter Senge’s ‘Five Whys’ Test, which he described in his book The Fifth Discipline. Essentially, the test involves asking ‘Why?’ every time you see a problem or describe a potential cause.

    There’s oil on the factory floor. Why? A machine is leaking oil. Why? The oil pressure is too high. Why? The technician who sets the pressure was told the wrong value. Why? And so on.

    The idea is that within five Why’s, you should be able to find the root cause of the problem. Kaushik’s test is basically the same one, just projecting forwards rather than backwards.

    Tests like these are simple, but I’ve found have a tendency to only work when the dynamics of the situation are transparent and there’s little or no time-delay between causes and effects. Otherwise, one can waste a lot of time chasing shallow answers that are just plain wrong.

    As I said, though, these types of tests are easy to do, so they are good to keep in mind as potential first-pass tools, but be aware of their limits and, when necessary, be prepared to roll up your sleeves and dig into the data.