I Tested the Best Headline Analyzers and Confirmed My Suspicions

12 min readFeb 19, 2019

Imagine for a moment that you could write headlines that were 10% better than your current ones.

That would give you 10% more visitors. 10% more readers or viewers. 10% more subscribers. And finally, 10% more buyers.

Maybe the math doesn’t check out exactly like that (it’s actually much better*), but I’m sure you will agree with me that having 10% more people pay attention to whatever you are doing would be awesome. Unless you are doing something embarrassing. In which case, feel free to refrain from putting a headline on it.

But if you do want more attention, your headline is the first place to look for improvements.

If your headlines are better, you’ll have more people paying attention to your Tweets, Facebook posts, YouTube videos, blog posts, Medium articles. You’ll have more people clicking your emails after reading the subject line. Your ads will perform better, and so will your sales pages.

In an online world where everyone is battling for attention, headlines may be the biggest leverage point anyone has.

*The final numbers are likely going to be better than 10%. If you use your improved headline skills to write better landing pages, sales pages, email subject lines, etc., you will boost attention on more levels and go way beyond a 10% improvement.

And yet…

Despite this clear connection between attention — something all creators, marketers, and business people crave — and headlines, we don’t spend that much time thinking about them.

Most of us still suck at writing headlines. I know I suck. And probably you do too.

Headlines often end up being an afterthought. A hassle that has to be completed before we can publish our latest blog post or send our email newsletter.

And that’s bad. So bad it may be the only thing separating you from success.

The question is, how do we fix it?

A simple promise

Last year I wrote a post trying to answer this very question — how to write better headlines. I aptly named the post How to Write Better Headlines Even If You Suck at Copywriting and tried my best to explain how I was helping myself write better headlines.

I now suspect part of that post is misleading.

The part that makes me anxious is about headline analyzers. Tools that promise to help us write better headlines without doing all of the hard work. ‘Just enter your headline and we’ll tell you if it’s good’-kind of tools.

The all sell an intoxicating promise: you don’t need to work on your headline skills, we’ll do it for you. It was a promise I took to heart.

But I’m starting to think I was wrong.

How good are these tools in real scenarios? It is really possible to predict the success of a headline just based on simple heuristics?

These are important questions to answer if we want to use headline analyzers for our headlines. Which is why I set out to explore them. And you are welcome to join.

Let’s start by introducing our subjects.

The 3 headline analyzers

The headline analyzers in this study are:

When bloggers, marketers, and other website owners talk about website analyzers, these three tools are what they refer to.

The analyzers all work much like you’d expect. You enter a headline, the tool performs magic and returns a score. I’ve written a brief summary for each of the tools below, but feel free to skip it if you already know them — or if you are just here for the results.

CoSchedule Headline Analyzer

CoSchedule will give your headline a score from 0 to 100. Next to the score you’ll also get an analysis of your word balance:

Below this, you’ll find information about your headline type, length, first & last words, keywords, sentiment, search preview, and email subject line preview.

The headline score may be the most upfront result of your headline analysis, but these additional features make CoSchedule stand out as one of the most popular headline tools.

Sharethrough Headline Analyzer

Your headline will also get a score from 0 to 100 from the Sharethrough analyzer.

Followed by strengths and suggestions in bullet form:

After this comes two additional scores — engagement and impression. These scores are opposing forces so you can’t maximize both. Instead, you can get a sense of where your headline is heading:

Emotional Marketing Value Headline Analyzer

The emotional marketing value (EMV) headline analyzer by AMI is not as flashy as the other two tools. Rather, it looks like something from the previous millennium. But the tool still has its place in this test as both Sharethrough and CoSchedule are built upon the idea of emotional value in words.

In other words, this is one of the fundamental measures used in headline analyzers.

The EMV tool allows you to enter a headline and receive its emotional score along with a classification:

When using the EMV tool, you’ll quickly see that it was created years ago. The layout is from another time and the tool starts to act out if you enter special characters — sometimes for no apparent reason.

It’s like an old but well-crafted car that has long outlived its generation. The engine is still humming as long as you know the correct way to operate it.

How to test a headline analyzer?

Testing headline analyzers is tricky business. You need to have lots of headlines and lots of results for these headlines.

I could have used one of my own websites as an experiment, but that would potentially skew the results — I would only have one website and I would also have too few headlines.

There are plenty of headlines to look at on the internet, but the difficult part is getting their results. How do you judge the success of one blog post versus another?

As an outsider, that is difficult.

So I had to go in another direction. I turned to a place where headlines are very important, and where results are shown publicly — I turned to Medium.

Each Medium article has a public score — the number of claps it receives. If you are writing on Medium, you want to have as many claps as you can get. Just as you hunger for likes on Facebook, re-tweets on Twitter, and whatever score Instagram has.

The experiment

In a scientific study, the experiment is always described in detail before the results are revealed. This way, doubters can replicate the experiment and draw their own conclusions.

This is at the very core of the scientific method. And as there’s still some science left in me, I’ll follow protocol in this not-so-scientific study too.

The setup:

The subjects in this study are the headline tools. We want to know how well they perform in real scenarios. To test this, I’ll enter a headline into the tool and record the response I receive (i.e., the headline score).

If the headline scores are good a predicting the success of a headline, we can rely on the analyzers. If not… well, you get the idea.

For input, I will use Medium articles for reasons explained above. I have recorded the 30 most recent articles posted by top writers in the entrepreneurship, business, productivity, self-improvement, and life lessons sections. That gave me 383 articles by 13 different writers (one writer didn’t have 30 articles to pick from).

For each of these articles, I have noted down the headline and the number of claps it has received. The claps will be used as a measure of popularity (and in turn success). This isn’t the best measure, but it’s what I have.

The results

Our goal is to show how well the 3 headline analyzers can predict how well a headline does. If they are good at this, we can use them to improve our headlines before we publish.

Stated differently, what we want to measure is how well the headline scores and the claps line up. In a word, how well they correlate.

In statistics, one way to measure this is with the correlation coefficient.

Very briefly, this is a measure between -1 and 1 that shows whether two variables move together (e.g., when one goes up the other ones goes up the same amount too). The closer to 1 this coefficient is, the more correlated the measurements are.

A score near 1 means the headline analyzers are very good at predicting how many claps a headline gets. 0 means no correlation at all. In that case, throwing dice is as accurate as the headline analyzers. A negative correlation (between 0 and -1) indicates the headline score goes up, the number of claps goes down.

I used excel to compute the correlations for me. This link provides more information about the computation (look under correlation).

Enough foreplay. Let’s get the numbers, shall we?

The average correlation between headline scores and claps for all 383 articles is…

[Suspenseful music playing]

[Drumroll starts]

0.06.

A mere zero-point-zero-six correlation coefficient.

[Drumroll stops, followed by the sound of disappointment (silence)]

Correlation — and statistics in general — is often hard to interpret. There’s no definite answer and everything depends on sample size, context, and a myriad of other factors.

Despite all of that, 0.06 is AT BEST a very weak positive correlation. More likely, it’s no correlation at all.

Here’s the full correlation table:

I have painted the important column yellow. In this, you can see the correlation between claps and the headline analyzers.

At a glance, it looks like Sharethrough is doing alright — it is at least doing much better than CoSchedule — but don’t let that difference fool you. These correlation numbers are still low.

Do the analyzers help you decide on a headline?

The correlation between headline score and claps on Medium isn’t promising. But from Thinking, Fast and Slow (great book) I learned that humans don’t relate to statistics that well. Concrete examples are much easier for us to understand, so what about a few of those?

Here’s one:

Which one would choose?

I’ll give you the correct answer in a moment. Just remember your answer.

As you’ll note, I have provided you with the headline scores for each of the headline analyzers. They don’t agree on a winner, so what you do with the scores is up to you.

Here’s another one:

The answers

The reason I can tell you which headline performed best is that these are not just random headlines picked from my imagination. They are headlines that I have used — and more, they are headlines that I have tested.

I was asking myself the same question I just asked you — which of these headlines would perform better?

And the only way to actually find out is by testing them. Which I did.

How to Generate Headline Ideas in Just 5 Minutes performed 60% better than How to Write Better Headlines Even If You Suck at Copywriting (Like Me).

Only Sharethrough thought the 5-minute headline was the better one, and only with a single point — 67 versus 66.

The Truth About Success and Happiness performed 22% better than The Happy Secret to Better Work.

And this time it was the other way around, with CoSchedule as the only believer in the first headline.

I don’t know how the results line up with your choices, but I know how they line up with the headline analyzers. And it is not looking good.

I’m starting to form a conclusion

In How to Write Better Headlines Even If You Suck at Copywriting I advocated using headline analyzers to improve your headlines.

Now I’m not so sure.

In 4 A/B email split tests I did to my subscribers (two of which you just saw above), the headline analyzers were right 6 out of 12 times — exactly half the time. And it doesn’t help to average out their answers as they have conflicting opinions on what makes a good headline — doing this made them miss the mark in 3 out of the 4 tests.

So far, the evidence doesn’t paint a great picture of the headline analyzers. Truth be told, throwing dice 383 times would have produced results close to what we are seeing. (I actually computed a couple of correlations between claps and 383 random numbers from 0 to 100. One of the cases produced a 0.08 correlation — a stronger correlation than what our headline analyzers did.)

My sample size should be taken into consideration though. 383 articles with a questionable measure (claps) isn’t the ground truth. And my examples are just that — examples.

(The random numbers experiment also shows how my numbers may be off from the true correlations. The true correlation between claps and random numbers is exactly 0 — they are independent observations and one of them can’t influence the other. If I had enough samples, my results would get closer to that. But as you can see, with just 383 samples, variance plays a big role.)

But then again. If I want to use headline analyzers to predict the success of my headlines, I want them to give me accurate readings. And what I have seen so far doesn’t indicate that.

Headlines… are hard.

In the end, I don’t think the headline analyzers are to blame. They are actually doing a good job providing useful information about headline length, composition, words, etc.

As long as you don’t give the overall score too much weight.

And that’s where our error lies, I think. We are so desperate for the easy and simple solution that we are ready to let a tool tell us if our headline is good or not.

But it’s never that simple.

First, it takes more than just a few words to create popular articles. The headline may be important, but it still needs content to back it up once it has attracted attention.

Second, headlines vary from community to community. Reddit is a great place to view how different headlines can be. Browse a few subreddits and you’ll see.

Third, there’s more to a headline that just the words it contains. The Truth About Success and Happiness may get a score of 77, but so does About and Truth Success Happiness The. In the real world, I’m quite sure they wouldn’t be as evenly matched (although I haven’t tested this). Headline analyzers can’t read and so any meaning in the headline is lost to them. In that sense, they are still quite primitive.

How to write better headlines

Headline analyzers don’t look like the golden shortcut to better headlines. They may be able to give you some pointers and tips, but that’s all.

They won’t make your headlines great, or even tell you if they good or bad.

That’s unfortunate. I was looking for a shortcut and this wasn’t it. But maybe it was my own fault all along? Maybe it’s time to look at headline writing for what it is — a skill. And skills are to be mastered before you can expect to improve your results.

There are no shortcuts, tools, or analyzers that will help you skip the process of learning. If you want to write better headlines, you need to work on this skill. It’s that simple.

Create more headlines. Test more headlines. Read more headlines.

This post was born out of a suspicion — that headline analyzers can’t accurately predict the success of a headline. And my experiments have convinced me that this suspicion was well-founded.

If you want to write better headlines, you need to learn how to. No tool will do it for you. Sorry.