Our Blog

Peerindex is not broken, but it's not perfect


Posted by
9 December 2010
13:10
17 comments

The story so far …

In last week’s tech post I focused on Twitter influence measuring service Klout’s inability to stop bots from acquiring high klout scores. Klout responded positively, and got involved in the conversation.

Joe Fernandez, CEO of Klout, raised the philosophical question of whether giving bots a high influence score is a good or a bad thing. Shouldn’t it be possible for a bot to get a high Klout score? Is it not possible for a bot to have a high influence level?

To recap, the test probes in this experiment are four bots that Tweet random slightly humorous quotes once every minute, once every five minutes, once every 15 minutes and once every 30 minutes respectively.

Testing Peerindex

So, this week, as promised, I registered my four little soldiers on peerindex.net, and within 24 hours the scores were in. As predicted by Azeem Azhar, Peerindex’s CEO, the bots scored reassuringly low. (See his comment on the previous post).

Let’s look at a screen shot of the first bot (the one with the highest score, which reached a score of 52 on Klout)

Bot 1 Peerindex

(Click to enlarge)

Quite interesting, isn’t it? At a first glance, it really seems like they got it spot-on. Except for one thing … the realness count (indication of whether the account is a human or a bot) is 75% – quite high for a bot, especially since, according to their score documentation, they start the score at 50%, and increase it only based on clues that the account is human.

Let’s look at the other bots’ realness count:

Bot 2 Realness

5 minute bot

Bot 3 Realness

15 minute bot

Bot 4 Realness

30 minute bot

All of them score 75% – an indication that this metric is, as it should be, based (directly or indirectly) on Tweet content. Since the bots Tweet from the same set of quotes, this makes sense. It’s not as reassuring as our first impression though …

Then, while looking at one of my other accounts, I saw something quite disturbing. A highly acclaimed South African author and singer/songwriter I’m following, Koos Kombuis, has a peerindex of 0 …

Koos Kombuis Peerindex

Click to enlarge

… and a klout score of 58! His large klout score makes a lot more sense, given his celebrity profile and the size of his actively participating online audiences.

Koos Kombuis Klout

Koos also seems to have only a 35% likelyhood to be a human. Koos, are you still in there?

This had me quite puzzled for a while, until, Tuesday at #DevNest, Peerindex’s Product Manager, Simon Cast, revealed that a large part of their metric is based on the links you include in your Tweets. None of my bots ever tweet any links, and Koos Kombuis also rarely tweets links.

This actually means that testing Peerindex against these bots doesn’t mean much – a proper test will be to set up a set of bots that tweet automated content and links, and test Peerindex against those.

Let’s take a moment to investigate the philosophy behind ignoring linkless Tweets, though …

I believe, by discounting Tweets that don’t contain any links, Peerindex is ignoring a large part of what is important on Twitter. Koos Kombuis, for instance, uses Twitter for widely followed creative projects, like his most recent, #Twitterdawn, a novel written entirely by means of Twitter.

Stephen Fry had a very interesting feed to follow last year, as he followed the transfer of four Northern White Rhino’s from a zoo in the Czech Republic to a nature reserve in Kenya. During this period, Fry tweeted a lot, and his feed mostly consisted of news that he was witnessing first hand, in other words, linkless.

Having said this, in a recent Cinch interview by Robert Scoble, Azeem@Peerindex sums it up quite well:

… we have these obvious monsterous failings, which I’m quite happy to talk about … there’s a guy who’s won the nobel prize in chemistry, and his Peerindex is 0 … that doesn’t reflect the real world … you have a guy like Clay Shirky, he blogs once a month, a million people read his blogposts. he doesn’t have as high a Peerindex as somebody who Tweets his stuff out … these are huge holes in our data index and in our analytics … we’re a startup, we’re going to get better at doing that, but right now, today, if you’re trying to find topical authority for people on Peerindex, it’s not a bad place to start …

Both Klout and Peerindex are busy with pioneering work. In a way, they are to Social Networks what Google was to the Web in 1998.

There are obvious flaws in their respective approaches, not because of negligence or stupidity, but because what they’re trying to do is dangerously close to linguistically perfect artificial intelligence. It’s never been achieved before.

Posted by

3 Comments

  • Posted by Azeem
    December 9, 2010 at 1:50 pm | Permalink

    Hi adriaan

    Thanks for putting us through our paces. The realness score is relatively new and something is obviously funky with this.

    This kind of testing is really important. Lets clarify the Koos scenario …. For a subset of users, we won’t have fully indexed them. We add tens of thousands of users a day and it does take us a while to index those, particularly ones which may not coincide actively with

    As i mentioned to Scoble, we are at the eary stages of this journey, and starting to understand the best approaches. Our focus (today) is getting the fat middle right, that is the ordinary people who are working to establish a reputation. We are bound to have inconsistencies at this point, and as yu point them out, we will figure out how to squish them!

    Azeem

    The

  • Posted by Adriaan Pelzer
    December 9, 2010 at 2:36 pm | Permalink

    Hi Azeem,

    Understood – I also understand the pressures involved in rolling out massive changes on live services. You guys must’ve had quite an interesting few weeks!

    I do like the new interface – the radar display is a very nice touch. I did find a few issues on the new interface – my regions shows China only – one of my sources show as the shortening service I use in Tweetdeck, so the redirects are not being followed to their ultimate destination …

    Anyway – these are smallish issues … I also played with the API – it’s great!

  • Posted by Ralph Savage
    May 15, 2011 at 1:48 pm | Permalink

    I’m really impressed by the analysis here. Particularly the idea that Both Klout and Peerindex are “dangerously close to linguistically perfect artificial intelligence.” Very keen to hear more.

14 Trackbacks

  • [...] Update – 2010/12/: The Peerindex results are written up. Do check them out here. [...]

  • [...] This week, as promised, we registered our four little twitterbot soldiers on peerindex.net, and within 24 hours the scores were in. Read more >> [...]

  • [...] that an acceptable influencer algorithm will be built than one that does sentiment analysis. But our tests showed the current leaders Klout and Peerindex are not without their problems. [...]

  • [...] matter as much as the quantity. Of course a percentage of the followers were other bots. While PeerIndex, though it kept the rating of the bots low, gave these bots a 75% realness score, which meant that [...]

  • By Quora on January 13, 2011 at 9:26 am

    What is the best alternative to Klout?... Klout is much better than metrics like Edeleman's Tweetlevel, because it does not over emphasise followers numbers, a number which can be meaningless in the extreme. (Some people follow others simply in the knowledge that they will be followed back.) ...

  • By Quora on January 13, 2011 at 4:14 pm

    What are some critical perspectives on influence metrics such as Klout and PeerIndex?... First off. We think Klout and PeerIndex are the leading two services thus far. But Klout has two main issues as we see it. The one is that it's a general score. Does somebody with a high ranking have influence across topics? Secondly Klout has issues ...

  • [...] Twitter – puff the real people using twitter well. If PeerIndex can’t do this then I agree that Peerindex is broken, or alternatively their claim to show us where the real action is going on in twitter isn’t their [...]

  • [...] erklärbar, gilt Klout doch besonders anfällig für Manipulationen. Peer­Index scheint hier robuster zu sein. Das ist auch erklärbar. Peer­Index bindet wie z. B. auch Back­Type Blogs in die [...]

  • By What’s your best influence evaluator? « sideworks on February 25, 2011 at 5:48 pm

    [...] Often cited as a reference by other bloggers and Twitter users, PeerIndex seems to invest a lot of energy in catchy graphics, but my current knowledge of algorithms and the fact that I do not really know the Twitter subject I picked prevent me from giving you a more in depth perspective on how it works. But I consider that these platforms should be understandable by every kind of audience so company owners looking for figures on these “tweet thingies they don’t understand” finally make sense. And with PeerIndex, I definitely feel comfortable with explaining the Topic Fingerprint graph but I’m still a little confused as to how one reads the Component one. I registered with PeerIndex yesterday, hoping to get more information. What also caught my attention was PeerIndex’s effort to evaluate the possibility your Twitter account might be a bot. Interesting, but still very much in testing phase apparently. [...]

  • [...] The question is, is Peerindex really any more accurate than Klout or other systems out there?  The other month the guys from RAAK created a fake Twitter account to (successfully) game Klout – Raak then tried Peerindex in the same experiment and found it did better…but still wasn’t perfect. [...]

  • By - CONSTRUCTION MARKETING UK BLOG on April 20, 2011 at 6:59 am

    [...] – puff the real people using twitter well. If PeerIndex can’t do this then I agree that Peerindex is broken, or alternatively their claim to show us where the real action is going on in twitter isn’t their [...]

  • [...] tweet regularly, but non-interactively, can get very high Klout scores. I repeated this experiment against Peerindex, with somewhat better [...]

  • [...] tweet regularly, but non-interactively, can get very high Klout scores. I repeated this experiment against Peerindex, with somewhat better results. Obvisouly, the next target should be their list, should it [...]

  • [...] de hace más de un año muestran las pruebas que hicieron con “Bots” para engañar a Klout y Peerindex. No dejes de leer los comentarios en respuesta al post de los fundadores de ambos Joe Fernandez y [...]

Leave a Comment

Your email is never shared. Required fields are marked *