Our Blog

Klout is broken


Posted by
2 December 2010
13:56
82 comments

A bit more than a month ago, I asked the question: Can you become influential on Twitter, and get a high Klout Score, merely by Tweeting a lot?

To test this, I set up an experiment, which involves four Twitter bots that automatically tweet the output of the Unix fortune command-line application.

Fortune randomly outputs mildly humorous quotes, and was often used on Unix to produce a ‘welcome message of the day’ upon login.

The four bots Tweet once every minute, once every five minutes, once every fifteen minutes and once every thirty minutes respectively. They are completely anonymous, have no avatars or custom user profiles set, and do not follow anyone.

Now, after 80 days of running the experiment (Jules Verne style), there’s a set of pretty hot data available.

The Data (the good)

Let’s start off by simply plotting the amount of followers for each bot against time:

Follower accumulation over time

(Click to enlarge)

We can clearly see from this graph (quite surprisingly), that each bot accumulated followers linearly. Also, it seems the more they tweeted, the steeper the follower accumulation rate is, without any drop off, even for the bot that tweets every minute.

This brings us to a question: Can these graphs in some way be normalized? Surely the bot that Tweets at the annoying rate of once every minute, should get fewer followers per tweet as the one that Tweets at a more acceptable once every 30 minutes?

Let’s normalize the data by plotting the amount of followers against the amount of Tweets, thereby literally measuring the amount of followers per Tweet:

Followers per Tweet

(Click to enlarge)

The scale is a bit awkward, but it seems that these bots are all more or less following the same slope, in other words, by the time the once every 30 minutes bot has tweeted as much as the once a minute bot, it will have the same amount of followers.

Let’s test this assumption, by plotting the curves over time again, including an amplification factor equal to the amount of minutes that lapse between Tweets. That means, we assume the once every 5 minutes bot would have had 5 times more followers if it Tweeted once every minute, etc:

Normalized Amount of Followers over time

(Click to enlarge)

Transient fluctuations aside, These curves really do seem to follow roughly the same path – linearly upwards.

That means, the more you Tweet, the more followers you get. Period. It doesn’t matter how often you tweet, you gain an equal amount of followers for every time you Tweet.

The Followers (the bad)

Now, on that bombshell … time for a sobering revelation:

Looking at the followers of these bots, many of them seem to be bots themselves (there are quite a few real people who attempt conversations with them, but they are in the minority). Most of these bots get triggered by keywords present in our bots’ Tweets, and then follow and retweet our bots’ Tweets. A good example is @BurroughsBot, which retweets Tweets that match the search term William Burroughs.

At this point I turned to Klout (which, incidentally, is the actual reason for setting up this experiment in the first place). Surely Klout should be able to make sense of this robotic mess (like Google does with link farms), shouldn’t it?

The Klout Scores (the ugly)

For all practical purposes though, no matter how I look at it, Klout seems to be broken.

Consider the following Klout scores, for the four bots:

Klout Score: Bot 1

Klout score for 'once a minute' bot

Klout Score: Bot 2

Klout Score for 'once every 5 minutes' bot

Klout Score: Bot 3

Klout Score for 'once every 15 minutes' bot

Klout Score: Bot 4

Klout Score for 'once every 30 minutes' bot

What’s wrong with this picture? To start off with, it should not really be possible for a bot to reach a Klout Score of 50 within 80 days merely by Tweeting random (yet entertaining) rubbish every minute, should it?

24 hours after the above klout scores were sampled, I took another set of samples, just to be sure:

Klout Score 2: Bot 1

Klout Score for 'once every minute' bot

Klout Score 2: Bot 2

Klout Score for 'once every 5 minutes' bot

Klout Score 2: Bot 3

Klout Score for 'once every 15 minutes' bot

Klout Score 2: Bot 4

Klout Score for 'once every 30 minutes' bot

Roughly the same result, except for huge fluctuations in transient metrics (see True Reach for Bot 1), which also seems a bit suspect. We can’t say for sure without knowledge of Klout’s exact algorithm.

The fact is, though, no matter how you look at it, unless Klout updates this aspect of their algorithm, in another 80 days Bot 1 could very well have the same Klout Score as @scobleizer!

Taking into account that many Twitter clients (like Hootsuite) and filter applications (like Datasift) are using Klout as a trusted way of filtering tweets, it means Klout will have to up their game on this one to stay in the game.

Or else, we might just be run by machines sooner than we think!

Update – 2010/12/: The Peerindex results are written up. Do check them out here.

If you find this post interesting, you might find our weekly Newsletter, The RAAKonteur, even more interesting. Sign up in the sidebar, top right.

Posted by

35 Comments

  • Posted by Matt Owen
    December 2, 2010 at 4:26 pm | Permalink

    Nicely done sir, I’ve been running something similar and it seems my atrocious, automated celeb news feed is more influential than I am through my personal account – a clar example of the wrong metrics being rated.

  • Posted by Joe Fernandez
    December 2, 2010 at 4:27 pm | Permalink

    Hey Adriaan,

    I am one of the cofounders and the ceo here at Klout. This is a great post, even though we get slammed :)

    A couple things:
    - Clearly there is more we can do to recognize and punish bots. This is something we are working on and I think you’ll be impressed with what we have coming. That said, this is an incredibly hard problem that even Twitter still has trouble with (judging from the clear spam bots I see following me and not disappearing).

    - The score right now is actually doing what it’s supposed to in the sense that it’s measuring engagement. Take a look at search for @burroughsbot (http://search.twitter.com/search?q=%40BurroughsBot). This account is actually getting way more engagement then it should. We do measure for noisiness but obviously we need to look at how we handle extreme cases like this.

    We have a science team working on stuff like this on a daily basis. Post like this get us really fired up so I am excited about your challenge to step our game up.

    Would love to chat sometime about how we can throw some data your way for some more independent testing.

    Thanks.

  • Posted by Adriaan Pelzer
    December 2, 2010 at 4:43 pm | Permalink

    Hi Joe,

    I can just imagine how hard a problem this must be to solve. It probably ranks way up there with the holy grail of sentiment analysis.

    If you crack this, I do believe it will give you an undisputed lead in the market. I’m looking forward to see what’s in store! Let us know as soon as anything’s public, and we’ll write a post on it.

    We’d love to take a look at any data you’d like to throw our way.

    BTW – it is great to see you, as CEO, so involved in your product!

  • Posted by Adriaan Pelzer
    December 2, 2010 at 4:47 pm | Permalink

    Thanks @Matt!

    I realized this first while filtering the public timeline for the highest Klout scores … something didn’t look quite right, partly because I misunderstood influence at the time, but also partly because of the sheer size of the “twitter spam” problem.

    It seems from Joe’s comment above that there might be some interesting updates from Klout in the near future, though

  • Posted by john burke
    December 2, 2010 at 5:24 pm | Permalink

    and this is some kind of shock? influencing is a fallacy in these terms.

  • Posted by Ed
    December 2, 2010 at 5:25 pm | Permalink

    Curious, did you start these bots with an automated program to follow others?

  • Posted by Adriaan Pelzer
    December 2, 2010 at 5:27 pm | Permalink

    No – these bots still follow 0 people. The idea was to keep them as neutral as possible.

    Also no avatar, no custom profile imagery, etc.

  • Posted by Dan Greenberg
    December 2, 2010 at 5:30 pm | Permalink

    I heard that Old Spice used Klout scores to decide which folks got personalized greetings from the Old Spice Man. “Greetings, FortuneBot1Min…”
    :-)

    BTW +1 to Joe for being a CEO personally involved in both his product and company PR.

  • Posted by Jason Douglas
    December 2, 2010 at 5:31 pm | Permalink

    Thanks for the post. I appreciate all the data and analysis behind this experiment.

    I am glad Joe responded to this post, as it shows they are truly committed to being the leader in the measurement of influence. Klout is further along than other measurement sites, but is still relatively new. That has to be recognized here.

    I’m happy that I’m more influential than any of these bots (Klout of 63) :)

    I’ll tweet this post and hope this generates more conversation. Keep up the good work, Klout and Adriaan!

  • Posted by Jenifer Olson
    December 2, 2010 at 5:34 pm | Permalink

    Excellent response, Joe. Hoping you can update the algorithms to more accurately reflect true engagement, especially since more and more employers seem to be using Klout scores to help them identify good job prospects. Thanks!

  • Posted by Adriaan Pelzer
    December 2, 2010 at 5:38 pm | Permalink

    It seems we’re all agreed Joe@Klout couldn’t have handled this any better! It happens when you’re truly into what you’re doing.

    @jason: Thanks – the more conversation the better.

    @dan: wouldn’t an Old Spice mention classify as tampering with the experiment? ;)

  • Posted by Mike Handy
    December 2, 2010 at 8:43 pm | Permalink

    wow… this one hurts the ego… lol thats pretty amazing… most frustrating score on Klout is the true reach measure…. I could be that the 1 time per min bot is an outlier… but with the other 2 Im ok with them though not thrilled.

  • Posted by Liz
    December 3, 2010 at 12:23 am | Permalink

    I think one of your accounts followed me. They were gibberish sounding Tweets. I actually ReTweeted some because they were so nonsensical.

  • Posted by Gary Arndt
    December 3, 2010 at 3:06 am | Permalink

    I’ve noticed that Klout can’t even get basic data right. I am listed on close to 4,000 Twitter lists, yet it has me being on zero lists. This isn’t a number that has to be calculated. It is a number that is sitting on my Twitter page ready to be grabbed.

    I’ve noticed the same problem for many accounts, where the Twitter list data is wrong, as is mutual follower data or even the topic summary.

    The decision to include personal Facebook accounts, but not fan pages, also makes no sense. Many people just use Facebook for friends and family. Being influential with your mom and high school friends really doesn’t mean anything in the big scheme of things. Fan pages are public and would make much more sense.

    Also, if Klout is to be a measure of online influence, why not include Feedburner, Quantcast, Alexa, Compete, and other data. Some people like Seth Godin aren’t on Twitter, yet he has an enormous amount of online influence.

  • Posted by Adriaan Pelzer
    December 3, 2010 at 10:12 am | Permalink

    @Liz – our accounts don’t follow anyone, to make sure the experiment doesn’t get tainted by the “Follow Back” syndrome. So, if something is following you, it’s not one of our drones – shoot it down! :)

    @Gary – I didn’t even notice this. Could the lists issue be due to update frequency? I mean, did you get onto all those lists in quite a short time?

  • Posted by Adriaan Pelzer
    December 3, 2010 at 10:38 am | Permalink

    We’re busy running Peerindex against the same bots, for a possible comparative piece. Do check back in a week or two (or however long it takes for Peerindex to analyze the accounts)

  • Posted by Azeem
    December 3, 2010 at 2:28 pm | Permalink

    Hi Adrian

    Great work — and good job.

    It might take us a while to get to your bots, we’re pushing a platform upgrade.

    We do actively extract spam bots through a series of spam filters, but our approach to spam detection is in some cases after the fact.

    That is – we’ll find and index an account, unless it is hugely and obviously spammy. And then post indexing we’ll review acounts, and punish the spammers.

    Based on the process you’ve described, I *think* (and there are a few too many variables to be certain) that we will catch these as spammers; or even if we don’t catch them as spammers, they may end up with low-ish PeerIndexs.
    Let’s see how it turns out :)

    I see you are based in Hackney, if I can tempt you out for a coffee one day, I’d be happy to meet up with you. We’re just moving to Old Street (i know, i know) so perhaps that end of town in mid December?
    best wishes
    Azeem

  • December 3, 2010 at 5:54 pm | Permalink

    Adriaan solid post on what we have been seeing for sometime. @InfiniGraph took a very different approach to social rank taking in context of relevance. Our most recent post talks to this at http://bit.ly/bklBvJ Influencer, Content Intelligence and Social Engagement -

    @chasemcmichael

  • Posted by Liz DeLoach
    December 3, 2010 at 6:39 pm | Permalink

    Adrian and Joe,

    I’m a social media consultant and I would like to write a blog post for my site about this. Its thrust would be how to respond when your business or product get slammed – quoting the productive dialog between the two of you. Great stuff – but I wanted to ask first before I use. Plus, if you’ll provide me your twitter user names, I’d love to follow you both there.

    As an aside, Joe, my current Klout rating is Explorer – seems accurate given my understanding of its explanation. :) Pretty good – working to make it even better. And FYI, my twitter is @lizdeloach. Thanks for considering!

    Liz

  • Posted by Gary Arndt
    December 4, 2010 at 12:51 am | Permalink

    @adrian no. I’ve been on Twitter since early 2007. I have over 22,00 tweets. I contacted Klout and they said there was nothing they could do because it is a Twitter API problem, yet there doesn’t seem to be an issue with most other accounts.

  • Posted by sven
    December 5, 2010 at 4:28 am | Permalink

    I have always had a huge problem with Klout — they purposefully hide any statistically sound baseline or methodology just to sell a number (just like credit scores or morningstar ratings). Even if they want to keep their methods a secret, they can’t even really explain what the score is attempting to measure. When someone actually steps up to the task of correlating reality with Klout scores, the results aren’t pretty. This article is one example, another is the following:

    https://seomoz.box.net/shared/otzi9bf204

    It pains the author to even try to correlate Klout scores with CTR.

    Since Klout is present here, why don’t you shed some light before selling any more snake oil?

  • Posted by TheBusyFool
    December 6, 2010 at 10:58 am | Permalink

    The trouble with any published algorithm is that it’ll get gamed instantly – and that would make its output LESS reliable, not more. As long as it takes spammers and bots a while to catch up with the algorithm-meisters (and the morphed algorithms actually work, of course), we have a chance of getting a true(ish) reading.

  • Posted by Hitesh Mehta
    December 7, 2010 at 9:44 am | Permalink

    Somehow even I find that Klout’s not giving the accurate data, there is complete mismatch on the Most influential topics which it gives under my account @HiteshMehta Or at least i do not understand how they have calculated this. 90% of my tweets are about #design #socialmedia #ux #usability #marketing and none of these are captured under my account.

  • Posted by Martijn Linssen
    December 7, 2010 at 2:05 pm | Permalink

    Hi Adriaan,

    great experiment! And you as well have managed for Joe to come out here and comment – well done

    Klout is a very unreliable product and service, the anomalies you noted overnight are quite common.
    http://www.martijnlinssen.com/2010/06/why-i-have-doubt-about-klout.html, http://www.martijnlinssen.com/2010/06/get-your-act-together-klout.html, http://www.martijnlinssen.com/2010/11/why-i-think-klout-is-krap.html and http://www.martijnlinssen.com/2010/11/klout-nail-coffin-who-cares.html are my four posts on noticing extreme irregularities in Klout scores

    Basically, Klout is Krap. They can’t handle the volume anymore, and the so-called daily scoring they claim to do today is either fake or even more broken than the product itself

    Klout is really great in Marketing, but that has taken over the quality of their product and service, which are lagging miles behind today

    I guess Joe c.s. is just hoping Virgin or their other Perks’ partners aren’t too internet-savvy and miss most of the discussions going on about the obvious very low quality and consistency of the Klout product

    Thanks for KloutBashing Adriaan – we need to keep the 1.0 people out of this 2.0 world

  • Posted by Olin Hyde
    December 8, 2010 at 6:44 pm | Permalink

    Great analysis. Deeply disturbing that bots are more likely to generate influence and followers over actual people. The inference is that bots can move group opinion and be used as an effective PR and marketing weapon. Many thanks for posting.

  • Posted by Paula Lee Bright
    December 9, 2010 at 2:27 pm | Permalink

    Nicely played, sir!

    Adriaan, this is a marvelous bit of analysis, and it’s clear enough that even this novice and avoider of analytics understood it!

    Thank you for the insight. I’ve been wondering why people I only talked with once were my big influencers!

    Both my selves (one a teacher, one a person without the constraints of being a teacher) have all kinds of things that don’t make any sense. This week, @Child
    WillRead was an influencer of tons of people…but I hadn’t posted to these people for a week!

    I dunno. We’ll watch it over time, I guess, but for now, this was a real relief to read. I don’t feel so strange about my mad, mad, mad mad scores. ;)

  • Posted by Adriaan Pelzer
    December 9, 2010 at 2:50 pm | Permalink

    @Paula,

    Do check your accounts on PeerIndex too (peerindex.net). They have a different approach.

    At this point I don’t think either klout or peerindex have the magic approach, but they both do many things right, and the things they do right differ from each other.

    That’s a very good thing.

    Also check out my latest post, checking these bots against peerindex.

  • Posted by Dharmesh Shah
    December 17, 2010 at 9:15 pm | Permalink

    Awesome post!

    I love folks that actually do real experiments and share the data.

    If you have a minute, would love to run your 4 bots through Twitter Grader (http://TwitterGrader.com). Likely won’t fare any better than Klout, but I’d be curious.

    Thanks.

  • Posted by Kim William
    December 23, 2010 at 8:00 pm | Permalink

    Nicely played, sir! Adriaan, this is a marvelous bit of analysis, and it’s clear enough that even this novice and avoider of analytics understood it! Thank you for the insight. I’ve been wondering why people I only talked with once were my big influencers! Both my selves (one a teacher, one a person without the constraints of being a teacher) have all kinds of things that don’t make any sense. This week, @Child WillRead was an influencer of tons of people…but I hadn’t posted to these people for a week! I dunno. We’ll watch it over time, I guess, but for now, this was a real relief to read. I don’t feel so strange about my mad, mad, mad mad scores. ;)

  • January 13, 2011 at 4:05 pm | Permalink

    congrats for this very interesting study!

  • Posted by Angus Fox
    January 18, 2011 at 9:14 am | Permalink

    Since you presented this at Devnest I have found it again. It is a great piece of analysis.

    Angus

  • Posted by Michael Hodson
    February 16, 2011 at 6:47 pm | Permalink

    really great experiment — thanks for knocking this out of the park for all of us.

  • Posted by james White
    June 13, 2011 at 11:33 am | Permalink

    This is a great idea, i have been working on a similar project to analyse the improvement of are talents and how well marketing is working in the company.

    Keep up the good work!!!

  • Posted by Sophia Guo
    June 17, 2011 at 9:17 pm | Permalink

    At eXpertMeme.com, we strive to measure influence with the most meaningful and robust algorithm, so that the influence score could be something really informative and valuable to users. We welcome you to TEST it out. And send me emails if you found any glitches, we will address it ASAP.

  • Posted by Courtney Bolton
    October 7, 2011 at 6:15 am | Permalink

    yes, this is excellent. thank you.

47 Trackbacks

Leave a Comment

Your email is never shared. Required fields are marked *