Thoughts about media in Canada

Why I don’t trust (most) case studies

case-studies image

Ever noticed there is a case study for virtually everything you can think of? According to them, everything works!

Well I have news for you: Meh…

I have seen plenty of campaigns focusing on a specific tactic that failed miserably. Yet, if I google “Tactic XYZ case study” I easily find some, from very credible sources, stating that the tactic rocks. Sometimes the case studies have virtually the same variables, the same client type, the same target market than a few campaigns you saw crash n burning.

There’s a very simple explanation to this: Even a broken clock is right twice a day.

Out of a thousand campaigns, inevitably, one will work and will show the results hoped for. A case study will be built. The industry will be misled.

Now… Since the industry heavily relies on case studies, what should you do?

Quite simple: Methodology (I know… you’re flabbergasted… I’ve almost never used this word before, except maybe here, here, here, here, here and here)

If you find a case study that is interesting: contact the person that built it and ask as many questions as possible about the methodology.

– Were there other tactics running simultaneously?

– How was the data collected?

– What was the test null hypothesis?

– Were there inconsistencies in the data?

– Was this replicated with success for at least 2 other clients?

– Etc.?

The industry is moving more and more toward data… learn to filter the good from the terrible.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: