Klout is broken

A bit more than a month ago, I asked the question: Can you become influential on Twitter, and get a high Klout Score, merely by Tweeting a lot?
To test this, I set up an experiment, which involves four Twitter bots that automatically tweet the output of the Unix fortune command-line application.
Fortune randomly outputs mildly humorous quotes, and was often used on Unix to produce a ‘welcome message of the day’ upon login.
The four bots Tweet once every minute, once every five minutes, once every fifteen minutes and once every thirty minutes respectively. They are completely anonymous, have no avatars or custom user profiles set, and do not follow anyone.
Now, after 80 days of running the experiment (Jules Verne style), there’s a set of pretty hot data available.
The Data (the good)
Let’s start off by simply plotting the amount of followers for each bot against time:
We can clearly see from this graph (quite surprisingly), that each bot accumulated followers linearly. Also, it seems the more they tweeted, the steeper the follower accumulation rate is, without any drop off, even for the bot that tweets every minute.
This brings us to a question: Can these graphs in some way be normalized? Surely the bot that Tweets at the annoying rate of once every minute, should get fewer followers per tweet as the one that Tweets at a more acceptable once every 30 minutes?
Let’s normalize the data by plotting the amount of followers against the amount of Tweets, thereby literally measuring the amount of followers per Tweet:
The scale is a bit awkward, but it seems that these bots are all more or less following the same slope, in other words, by the time the once every 30 minutes bot has tweeted as much as the once a minute bot, it will have the same amount of followers.
Let’s test this assumption, by plotting the curves over time again, including an amplification factor equal to the amount of minutes that lapse between Tweets. That means, we assume the once every 5 minutes bot would have had 5 times more followers if it Tweeted once every minute, etc:
Transient fluctuations aside, These curves really do seem to follow roughly the same path – linearly upwards.
That means, the more you Tweet, the more followers you get. Period. It doesn’t matter how often you tweet, you gain an equal amount of followers for every time you Tweet.
The Followers (the bad)
Now, on that bombshell … time for a sobering revelation:
Looking at the followers of these bots, many of them seem to be bots themselves (there are quite a few real people who attempt conversations with them, but they are in the minority). Most of these bots get triggered by keywords present in our bots’ Tweets, and then follow and retweet our bots’ Tweets. A good example is @BurroughsBot, which retweets Tweets that match the search term William Burroughs.
At this point I turned to Klout (which, incidentally, is the actual reason for setting up this experiment in the first place). Surely Klout should be able to make sense of this robotic mess (like Google does with link farms), shouldn’t it?
The Klout Scores (the ugly)
For all practical purposes though, no matter how I look at it, Klout seems to be broken.
Consider the following Klout scores, for the four bots:
What’s wrong with this picture? To start off with, it should not really be possible for a bot to reach a Klout Score of 50 within 80 days merely by Tweeting random (yet entertaining) rubbish every minute, should it?
24 hours after the above klout scores were sampled, I took another set of samples, just to be sure:
Roughly the same result, except for huge fluctuations in transient metrics (see True Reach for Bot 1), which also seems a bit suspect. We can’t say for sure without knowledge of Klout’s exact algorithm.
The fact is, though, no matter how you look at it, unless Klout updates this aspect of their algorithm, in another 80 days Bot 1 could very well have the same Klout Score as @scobleizer!
Taking into account that many Twitter clients (like Hootsuite) and filter applications (like Datasift) are using Klout as a trusted way of filtering tweets, it means Klout will have to up their game on this one to stay in the game.
Or else, we might just be run by machines sooner than we think!
Update – 2010/12/: The Peerindex results are written up. Do check them out here.
If you find this post interesting, you might find our weekly Newsletter, The RAAKonteur, even more interesting. Sign up in the sidebar, top right.
Posted by Adriaan Pelzer
35 Comments
Nicely done sir, I’ve been running something similar and it seems my atrocious, automated celeb news feed is more influential than I am through my personal account – a clar example of the wrong metrics being rated.
Hey Adriaan,
I am one of the cofounders and the ceo here at Klout. This is a great post, even though we get slammed
A couple things:
- Clearly there is more we can do to recognize and punish bots. This is something we are working on and I think you’ll be impressed with what we have coming. That said, this is an incredibly hard problem that even Twitter still has trouble with (judging from the clear spam bots I see following me and not disappearing).
- The score right now is actually doing what it’s supposed to in the sense that it’s measuring engagement. Take a look at search for @burroughsbot (http://search.twitter.com/search?q=%40BurroughsBot). This account is actually getting way more engagement then it should. We do measure for noisiness but obviously we need to look at how we handle extreme cases like this.
We have a science team working on stuff like this on a daily basis. Post like this get us really fired up so I am excited about your challenge to step our game up.
Would love to chat sometime about how we can throw some data your way for some more independent testing.
Thanks.
Hi Joe,
I can just imagine how hard a problem this must be to solve. It probably ranks way up there with the holy grail of sentiment analysis.
If you crack this, I do believe it will give you an undisputed lead in the market. I’m looking forward to see what’s in store! Let us know as soon as anything’s public, and we’ll write a post on it.
We’d love to take a look at any data you’d like to throw our way.
BTW – it is great to see you, as CEO, so involved in your product!
Thanks @Matt!
I realized this first while filtering the public timeline for the highest Klout scores … something didn’t look quite right, partly because I misunderstood influence at the time, but also partly because of the sheer size of the “twitter spam” problem.
It seems from Joe’s comment above that there might be some interesting updates from Klout in the near future, though
and this is some kind of shock? influencing is a fallacy in these terms.
Curious, did you start these bots with an automated program to follow others?
No – these bots still follow 0 people. The idea was to keep them as neutral as possible.
Also no avatar, no custom profile imagery, etc.
I heard that Old Spice used Klout scores to decide which folks got personalized greetings from the Old Spice Man. “Greetings, FortuneBot1Min…”
BTW +1 to Joe for being a CEO personally involved in both his product and company PR.
Thanks for the post. I appreciate all the data and analysis behind this experiment.
I am glad Joe responded to this post, as it shows they are truly committed to being the leader in the measurement of influence. Klout is further along than other measurement sites, but is still relatively new. That has to be recognized here.
I’m happy that I’m more influential than any of these bots (Klout of 63)
I’ll tweet this post and hope this generates more conversation. Keep up the good work, Klout and Adriaan!
Excellent response, Joe. Hoping you can update the algorithms to more accurately reflect true engagement, especially since more and more employers seem to be using Klout scores to help them identify good job prospects. Thanks!
It seems we’re all agreed Joe@Klout couldn’t have handled this any better! It happens when you’re truly into what you’re doing.
@jason: Thanks – the more conversation the better.
@dan: wouldn’t an Old Spice mention classify as tampering with the experiment?
wow… this one hurts the ego… lol thats pretty amazing… most frustrating score on Klout is the true reach measure…. I could be that the 1 time per min bot is an outlier… but with the other 2 Im ok with them though not thrilled.
I think one of your accounts followed me. They were gibberish sounding Tweets. I actually ReTweeted some because they were so nonsensical.
I’ve noticed that Klout can’t even get basic data right. I am listed on close to 4,000 Twitter lists, yet it has me being on zero lists. This isn’t a number that has to be calculated. It is a number that is sitting on my Twitter page ready to be grabbed.
I’ve noticed the same problem for many accounts, where the Twitter list data is wrong, as is mutual follower data or even the topic summary.
The decision to include personal Facebook accounts, but not fan pages, also makes no sense. Many people just use Facebook for friends and family. Being influential with your mom and high school friends really doesn’t mean anything in the big scheme of things. Fan pages are public and would make much more sense.
Also, if Klout is to be a measure of online influence, why not include Feedburner, Quantcast, Alexa, Compete, and other data. Some people like Seth Godin aren’t on Twitter, yet he has an enormous amount of online influence.
@Liz – our accounts don’t follow anyone, to make sure the experiment doesn’t get tainted by the “Follow Back” syndrome. So, if something is following you, it’s not one of our drones – shoot it down!
@Gary – I didn’t even notice this. Could the lists issue be due to update frequency? I mean, did you get onto all those lists in quite a short time?
We’re busy running Peerindex against the same bots, for a possible comparative piece. Do check back in a week or two (or however long it takes for Peerindex to analyze the accounts)
Hi Adrian
Great work — and good job.
It might take us a while to get to your bots, we’re pushing a platform upgrade.
We do actively extract spam bots through a series of spam filters, but our approach to spam detection is in some cases after the fact.
That is – we’ll find and index an account, unless it is hugely and obviously spammy. And then post indexing we’ll review acounts, and punish the spammers.
Based on the process you’ve described, I *think* (and there are a few too many variables to be certain) that we will catch these as spammers; or even if we don’t catch them as spammers, they may end up with low-ish PeerIndexs.
Let’s see how it turns out
I see you are based in Hackney, if I can tempt you out for a coffee one day, I’d be happy to meet up with you. We’re just moving to Old Street (i know, i know) so perhaps that end of town in mid December?
best wishes
Azeem
Adriaan solid post on what we have been seeing for sometime. @InfiniGraph took a very different approach to social rank taking in context of relevance. Our most recent post talks to this at http://bit.ly/bklBvJ Influencer, Content Intelligence and Social Engagement -
@chasemcmichael
Adrian and Joe,
I’m a social media consultant and I would like to write a blog post for my site about this. Its thrust would be how to respond when your business or product get slammed – quoting the productive dialog between the two of you. Great stuff – but I wanted to ask first before I use. Plus, if you’ll provide me your twitter user names, I’d love to follow you both there.
As an aside, Joe, my current Klout rating is Explorer – seems accurate given my understanding of its explanation.
Pretty good – working to make it even better. And FYI, my twitter is @lizdeloach. Thanks for considering!
Liz
@adrian no. I’ve been on Twitter since early 2007. I have over 22,00 tweets. I contacted Klout and they said there was nothing they could do because it is a Twitter API problem, yet there doesn’t seem to be an issue with most other accounts.
I have always had a huge problem with Klout — they purposefully hide any statistically sound baseline or methodology just to sell a number (just like credit scores or morningstar ratings). Even if they want to keep their methods a secret, they can’t even really explain what the score is attempting to measure. When someone actually steps up to the task of correlating reality with Klout scores, the results aren’t pretty. This article is one example, another is the following:
https://seomoz.box.net/shared/otzi9bf204
It pains the author to even try to correlate Klout scores with CTR.
Since Klout is present here, why don’t you shed some light before selling any more snake oil?
The trouble with any published algorithm is that it’ll get gamed instantly – and that would make its output LESS reliable, not more. As long as it takes spammers and bots a while to catch up with the algorithm-meisters (and the morphed algorithms actually work, of course), we have a chance of getting a true(ish) reading.
Somehow even I find that Klout’s not giving the accurate data, there is complete mismatch on the Most influential topics which it gives under my account @HiteshMehta Or at least i do not understand how they have calculated this. 90% of my tweets are about #design #socialmedia #ux #usability #marketing and none of these are captured under my account.
Hi Adriaan,
great experiment! And you as well have managed for Joe to come out here and comment – well done
Klout is a very unreliable product and service, the anomalies you noted overnight are quite common.
http://www.martijnlinssen.com/2010/06/why-i-have-doubt-about-klout.html, http://www.martijnlinssen.com/2010/06/get-your-act-together-klout.html, http://www.martijnlinssen.com/2010/11/why-i-think-klout-is-krap.html and http://www.martijnlinssen.com/2010/11/klout-nail-coffin-who-cares.html are my four posts on noticing extreme irregularities in Klout scores
Basically, Klout is Krap. They can’t handle the volume anymore, and the so-called daily scoring they claim to do today is either fake or even more broken than the product itself
Klout is really great in Marketing, but that has taken over the quality of their product and service, which are lagging miles behind today
I guess Joe c.s. is just hoping Virgin or their other Perks’ partners aren’t too internet-savvy and miss most of the discussions going on about the obvious very low quality and consistency of the Klout product
Thanks for KloutBashing Adriaan – we need to keep the 1.0 people out of this 2.0 world
Great analysis. Deeply disturbing that bots are more likely to generate influence and followers over actual people. The inference is that bots can move group opinion and be used as an effective PR and marketing weapon. Many thanks for posting.
Nicely played, sir!
Adriaan, this is a marvelous bit of analysis, and it’s clear enough that even this novice and avoider of analytics understood it!
Thank you for the insight. I’ve been wondering why people I only talked with once were my big influencers!
Both my selves (one a teacher, one a person without the constraints of being a teacher) have all kinds of things that don’t make any sense. This week, @Child
WillRead was an influencer of tons of people…but I hadn’t posted to these people for a week!
I dunno. We’ll watch it over time, I guess, but for now, this was a real relief to read. I don’t feel so strange about my mad, mad, mad mad scores.
@Paula,
Do check your accounts on PeerIndex too (peerindex.net). They have a different approach.
At this point I don’t think either klout or peerindex have the magic approach, but they both do many things right, and the things they do right differ from each other.
That’s a very good thing.
Also check out my latest post, checking these bots against peerindex.
Awesome post!
I love folks that actually do real experiments and share the data.
If you have a minute, would love to run your 4 bots through Twitter Grader (http://TwitterGrader.com). Likely won’t fare any better than Klout, but I’d be curious.
Thanks.
Nicely played, sir! Adriaan, this is a marvelous bit of analysis, and it’s clear enough that even this novice and avoider of analytics understood it! Thank you for the insight. I’ve been wondering why people I only talked with once were my big influencers! Both my selves (one a teacher, one a person without the constraints of being a teacher) have all kinds of things that don’t make any sense. This week, @Child WillRead was an influencer of tons of people…but I hadn’t posted to these people for a week! I dunno. We’ll watch it over time, I guess, but for now, this was a real relief to read. I don’t feel so strange about my mad, mad, mad mad scores.
congrats for this very interesting study!
Since you presented this at Devnest I have found it again. It is a great piece of analysis.
Angus
really great experiment — thanks for knocking this out of the park for all of us.
This is a great idea, i have been working on a similar project to analyse the improvement of are talents and how well marketing is working in the company.
Keep up the good work!!!
At eXpertMeme.com, we strive to measure influence with the most meaningful and robust algorithm, so that the influence score could be something really informative and valuable to users. We welcome you to TEST it out. And send me emails if you found any glitches, we will address it ASAP.
yes, this is excellent. thank you.
47 Trackbacks
[...] Klout is Broken (on social media influence measurement models) [...]
Are there any published algorithms about how to estimate influence in social networks?... I just read this post http://www.wewillraakyou.com/2010/12/klout-is-broken/ . I think it explains perfectly why I am asking for an published algorithm instead of an implementation of a non published algorithm...
[...] We've managed (on our first try) to build a Twitter bot that, within 80 days, managed to get a Klout score of 50, and 336 Twitter followers. Should this be possible? We think not, unless Klout is broken. Read more >> [...]
[...] with Klout over the last few months, but have become increasingly sceptical of the results. I saw this post earlier today, which does a great job of explaining the problem. It’s also worth reading [...]
[...] Klout is broken [...]
[...] Klout is broken – RAAK | Digital & Social Media Agency London RT @graemewood: awesome bit of research - I'd always suspected Klout wasn't all that, but @adriaan_pelzer has quantified it http://bit … (tags: via:packrati.us) [...]
[...] news! An interesting discussion has sprung up around this post from Raak entitled “Klout is Broken.” (If you haven’t read that post, read it first, then come [...]
[...] to determine your score based on your twitter usage. The problem, however, is that statistics are game able (and as well all know, 88% of stats are made up!). Plus, the definition of [...]
[...] a particular channel would likely not be considered an influencer, however there have been enough analyses by people showing how important frequency is to your Klout score. I’d much rather contribute a smaller [...]
[...] last week’s tech post I focused on Twitter influence meassuring service Klout’s inability to stop bots from [...]
[...] and Klout, is that there seems to be some hub bub on the interwebs with the math specifically, how Klout can’t detect robots well and Peer Index uses a “real” metric, which I assume addresses in some way tweet [...]
[...] first instance concerns the weapon of choice for many – Twitter. Adrian Pelzer set out to study whether merely tweeting a lot is a way to gain popularity on Twitter, and increase your Klout [...]
[...] tweeted computer generated quotes every minute and achieved a Klout score of 51 within 80 days. Read the article but please take note of the comment by Joe Fernandez one of the co-founders of Klout who has a [...]
[...] The comments are worth reading too.This post points out some issues: http://www.wewillraakyou.com/201... Again, check the comments.Klout would be the first to admit that the score is a continual work in [...]
Which tool is the more accurate to calculate influence on Social Media: Klout, PeerIndex, other?... Great question, and I think it gets at a larger definition of "influence." For social (public) influence on internet services, Klout and PeerIndex and others do a great job of taking an algorithmic approach to measuring the visibility, reach, and clu...
[...] ook graag dat de metingen van online invloed soms vreemde scores opleveren. Zo is uitgebreid in een artikel beschreven dat een aantal aangemaakte accounts op Twitter, met alleen automatisch verstuurde [...]
[...] of like if you buy your twitter followers… I really think the Klout is broken article is spot on and shows a worrying thing that you can do with [...]
[...] ook graag dat de metingen van online invloed soms vreemde scores opleveren. Zo is uitgebreid in een artikel beschreven dat een aantal aangemaakte accounts op Twitter, met alleen automatisch verstuurde [...]
[...] do I buy Klout score? There are plenty of examples of how Klout overvalues Tweet frequency, and it reserves the “up and to the right” part of their charts for more general [...]
[...] Pelzer hat auf RAAK ein interessantes Experiment beschrieben, das Sie kennen sollten, um einschätzen zu können, wie robust ein Klout Score ist. Er [...]
[...] Tummelplatz für die »Scorekeepers«. Das wäre auch leicht erklärbar, gilt Klout doch besonders anfällig für Manipulationen. PeerIndex scheint hier robuster zu sein. Das ist auch erklärbar. [...]
[...] any more accurate than Klout or other systems out there? The other month the guys from RAAK created a fake Twitter account to (successfully) game Klout – Raak then tried Peerindex in the same experiment and found it did better…but still [...]
[...] They don’t measure true influence. RAAK did several tests looking at PeerIndex and Klout. The key test was Klout, as that seems to be the standard right now. They found that the bot that tweeted the most had a [...]
[...] They don’t measure true influence. RAAK did several tests looking at PeerIndex and Klout. The key test was Klout, as that seems to be the standard right now. They found that the bot that tweeted the most had a [...]
[...] a été prouvé comme étant brisé à l’aide de tests, comme quoi rien n’est [...]
Has the Sunday Times Social list got greater credibility than @peerindex or @klout? #stsl #socialmedia... Ged, first you say Klout and PeerIndex have serious deficiencies without substantiating it, then try to denigrate them because they as you say they have a "following in the social media/online tech/digerati/Williamsburg, Brooklyn/Shoreditch London cre...
[...] the end of last year, I proved that bots who tweet regularly, but non-interactively, can get very high Klout scores. I repeated this experiment against Peerindex, with somewhat better [...]
[...] weighted in favor of quantity - in fact, this fascinating study goes a long way toasted proving it: http://wewillraakyou.com/2010/12...The secret sauce is, and will always be, value. To the right audience, even content published every [...]
[...] There’s no doubt that the scales of social media are currently weighted in favor of quantity – in fact, this fascinating study goes a long way towards proving it. [...]
[...] Bots have been able to achieve a high Klout Score. [...]
[...] same thing as measuring your influence. For example. a really great blog post by Raak explains how a bot on twitter can eventually have the same Klout score as @scobelizer in 160 days. Clearly there is some work to do [...]
[...] Klout is Broken by Adrian Pelzer If you enjoyed this article, please consider sharing it! [...]
Klout Scores Are Hilarious... Let’s get straight to the point: This is funny. Klout scores should not make people in an uproar over who is left off lists, and we definitely should not be debating the validity of such a service to determine one’s influence…or for lack of a better wo...
[...] more: “Klout is broken” (Link) / “How Charlie Sheen Broke Klout” [...]
[...] it gets more complicated. Even accounts that Tweet can be non human. We ourselves have created bots that Tweet and that have attracted large follower counts; even from what we can tell are real people (as [...]
[...] ratio,as well as number of tweet,retweets,etc. You can game Klout though. For instance just Tweeting a lot can get you a Klout score of 50 (I do mean a lot ) and here’s another post about gaming Klout.6. #ff lots of peopleFollow [...]
[...] Yes, it seems to me that there's lots wrong with Klout. Its sophisticated-yet-still-crude-measurement-of-social-activity-with-a-bias towards Twitter means that noisy bots, troll and comedy accounts are all classed as being more influential in general terms than they really are. And you can game it. Still. [...]
[...] Yes, it seems to me that there’s lots wrong with Klout. Its sophisticated-yet-still-crude-measurement-of-social-activity-with-a-bias towards Twitter means that noisy bots, troll and comedy accounts are all classed as being more influential in general terms than they really are. And you can game it. Still. [...]
[...] de les entrades més comentades és l’escrita per Adriaan Pelzer titulada ‘Klout is Broken‘. En aquesta entrada explica l’experiment que va dur a terme durant 80 dies amb 4 bots [...]
[...] of many services that purport to measure influence online. It’s hardly flawless, as we have shown ourselves, but it’s a better measure than Twitter followers, for [...]
[...] werden ohne ihr Einverständnis erfasst und ‘bewertet’), sowie die Verlässlichkeit und Manipulierbarkeit der Berechnung werden sowohl Klout Score als auch Peer Index zunehmend in Lösungen zur Social [...]
[...] ohne ihr Einverständnis erfasst und ‘bewertet’), sowie die Verlässlichkeit und Manipulierbarkeit der Berechnung werden sowohl Klout Score als auch Peer Index zunehmend in Lösungen zur Social [...]
[...] scores award quantity over quality. In his post, “Klout is Broken“ Adriaan Pelzer found that a person can obtain a high Klout score simply by tweeting a lot. [...]
[...] aunque son de hace más de un año muestran las pruebas que hicieron con “Bots” para engañar a Klout y Peerindex. No dejes de leer los comentarios en respuesta al post de los fundadores de ambos Joe [...]
[...] to grow numbers. If you haven’t seen it, you should totally go check out the analysis on RAAK, where they were testing out some theories on [...]
[...] d’une étude menée par l’agence RAAK (spécialisée en Social Media) , les résultat ont démontré [...]
[...] to grow numbers. If you haven’t seen it, you should totally go check out the analysis on RAAK, where they were testing out some theories on [...]
Leave a Comment