Posts filed under ‘Uncategorized’
I just pushed a new feature out, that I think might be kinda interesting. Basically, you can see how people’s scores have/are changing. Are their scores increasing or decreasing, and how quickly? Who’s hot right now?
If you go over to Twitterank right now, you’ll see a new item in the nav bar. For those of you too lazy to actually go there, here’s what it looks like:
The new Rapid Risers feature shows Twitter users whose Twitteranks have increased the most between the last two times their scores were calculated. Right now, it looks like this:
At the top of the list right now is @stephenfry, and the little green number shows you that his score has shot up 335 points per day! Actually, that’s a lie. The maximum score you can get is 300 and there aren’t any negative scores (yet) so clearly his score can’t have gone up by 335. Rather, 335 is the score increase extrapolated out 24 hours. If you want to find out what really happened, click on his user name and you’ll see:
As you can see, apparently his score went up 94 points between 28 hours ago and 21 hours ago. That’s basically a 100% increase. Way to go Stephen. Now, as to why his score went up suddenly, I don’t know. Based on how the algorithm works, I can tell you that a bunch of people suddenly started talking to him, but why that happened, I don’t know.
So why is this interesting?
I think it’s interesting because Twitterank is a measure of “engagedness” and interestingness. If someone’s score suddenly goes up, it means they’ve suddenly become more engaging or interesting. This could also be interesting to the increasing number of businesses and celebrities on Twitter who are trying to directly reach out to and engage the masses. Twitterank, perhaps for the first time, offers them a way to quantify how well they’re doing.
Anyway. I recently was roused out of slumbering funemployment of the non-coding kind, and started working on a major revision for Twitterank. I just pushed it live to twitterank.com, so let me tell you about the new stuff… in bullet list form. Mmm bullet lists.
- No More Passwords! – If you have more of a long term memory than I do, you may recall all the brouhaha Twitterank stirred up when it was first launched. It’s a scam! They’z steelin’ ur passwurds!! OMG! Yeah. Well, in case you had any lingering doubts about Twitterank’s intentions, you can stop being so frackin’ paranoid now. The new version will no longer require your password. Instead, it’ll use this thing called OAuth that Twitter has since adopted (yay), which allows me to make authenticated requests on your behalf, without you giving me your password. So yeah, you can keep your stinkin’ password. I don’t want it anyway.
- New Algorithm – Speaking of authentication, the new algorithm works using the public search APIs which don’t require authentication. This means 2 things. First, you won’t see inconsistent scores like you did in the old version. In the old version, there were actually 2 different algorithms being used, one that used your password, and one that didn’t. Depending on which algorithm was used, you might’ve seen inconsistent scores. That doesn’t happen any more. Secondly, since the new algorithm works great with the public APIs, Twitteranks can be calculated for any Twitter user. And that’s what I’ve been doing. The old version had about 90k users in the DB. The new version has something like 2.5 million, and I’m constantly adding people as I come across them. One unfortunate side-effect of the new algorithm is that a lot of people will have lower scores than the old algorithm that used your passwords, because the search API returns less data than the authenticated APIs. But hey, in the grand scheme of things, that’s not a huge deal.
- Developer APIs – If you’re a developer and you think Twitteranks might be useful for something, check out the nifty new APIs. I’m still not entirely sure what it’ll be good for, but try it out and let me know. It seems like developers are looking for a good way to weed out spammers and SEO/marketers. While that wasn’t the original goal of Twitterank, I’m considering incorporating some anti-spammer filters to the scoring algorithm, so we might get there eventually.
- Frequent Updates – In the old version, scores were only re-calculated if someone looked up their score. In the new version, scores will be re-calculated periodically. Currently, users whose scores are in the top 10,000 will have their scores re-calculated every day or so. That means the Top 50 list will change at least once a day, which should make things a little more interesting.
And… I guess that’s it. Maybe it wasn’t such a huge update. Oh, but wait! I have one last thing! I really do, but it’s not done yet. So I’ll announce it when it’s done.
As I’ve mentioned elsewhere, my goal with Twitterank was to experiment with a slightly different way of quantifying Twitter users, specifically, using @ replies. Quite a few people have asked why we can’t just use follow counts or the number of posts. My answer has been something along the lines of “there are other signals”, which is another way to say, if we use less obvious signals, we might get less obvious (and potentially better) results. Let me illustrate with a few examples…
Earlier today, I got a tweet from @erikvb who asked: “how is it that your secwet alrowithm calculates 224.941 for @google if they don’t have any tweets at all?” Indeed, @google is not following anyone, only has a few hundred followers, and has no tweets. Yet, it’s ranked in the top 20. Weird, huh?
Of course, if you look at all the @ replies that @google receives, you can see that the Twitterank algorithm is behaving as expected. Now, this is weird and surprising, but does that mean twitterank is “wrong”? I’d say “no”. In fact, it’s actually doing exactly what it’s supposed to do: it’s telling us something we might not have otherwise known. It’s actually telling us (or rather twitterers everywhere are telling us) how influential @google could be, if it were a real twitterer with real tweets. It’s like going to a crowded party and pin pointing influential individuals without even looking at them directly or them saying anything. It’s quite powerful stuff.
Here’s another example I stumbled across. Some of you may know (or know of/about) @caterina, who is probably most famous as the co-founder of Flickr. She has close to 3000 followers on Twitter, and you might think someone like her would be influential in twitterverse. As it turns out, her score is currently 59.35 (72.79 percentile, 1.2% confidence), which is high, but not high for someone with so many followers. It actually makes sense if you look at her tweets. She’s obviously not a heavy twitter user, and her tweets are somewhat cryptic, at least to a casual passerby. But how would she stack up against @joetheplumber? He only has 530 followers and 44 updates. Take a guess, then go find out. Were you surprised?
All this is mildly amusing, but Twitterank itself actually isn’t very interesting. Someone’s twitterank is metadata, not data. It’s a spice, not a main course. So, Twitterank still needs to find a main course to spice up, but that’ll come later.
The original version of Twitterank required a user name and password in order to make an authenticated web service request to Twitter and retrieve data that isn’t otherwise accessible. Understandably, many people were reluctant to enter their passwords in a new 3rd party site, and so far, only a small fraction of twitterers have been twitteranked.
The good news is, I have enough twitteranks at this point, that I can start calculating twitteranks for most twitter users, using only publicly accessible data and previously calculated twitteranks. In other words, you can now get any Twitter user’s twitterank, without any passwords. Take for example, @ev, one of the founders of Twitter. He’s so far been cautious enough to not get a twitterank (or he probably doesn’t care), but wouldn’t you want to know what his score would be?
Now you can find out. Check out these screen shots:
Notice that under the score, it says “(30.04% confidence)”. Since the version of the Twitterank algorithm which works without a password uses limited public data, there’s a certain amount of guessing involved. Generally speaking, the higher the confidence, the closer the score should be to what Twitterank would calculate if it had access to the user’s password and can retrieve more data.
In any case, head over to twitterank and start looking up the twitteranks of all your friends who’ve resisted the urge so far. Make sure you let them know how they score too 🙂
I’m about to go to bed, because I really really need sleep. Today was pretty quiet, fortunately, and by tomorrow, most of Twitterdom will have forgotten about Twitterank. Thank goodness for short attention spans. So this evening, all you’re getting is a redesign (thanks @redct), and a new Tweet for those of you with scores in the upper 80th percentile (BTW, the latest version says “XX% of twitterers” instead of “XX% of you”, because the latter wording makes your ego seem more inflated than it should be).
Also, I experimented with a Twitterank algorithm that uses an API that doesn’t require authentication, but some crucial information is lost in that case, and the scores are 1) low and 2) less meaningful, which is no good. I have another idea I’ll try tomorrow, but for now it’s back to the old algorithm.
If I were to do this over again, I probably would’ve done 2 things differently. Firstly, I would’ve come up with an algorithm that works with Twitter’s Search API, which doesn’t require authentication. Secondly, I would’ve handled posting to twitter differently, which would’ve sacrificed virality, but after today, I’m not so sure insane rapid growth is really worth it. So… sorry, Twitterverse. I’ll do better next time. Promise.
Also, one last thought before I go to bed (damn, it’s almost 6am here): it doesn’t seem like Twitter rate-limited me, even though I’m certain I went over the 100 requests per 60 minute limit (by orders of magnitude, in fact). I’m not sure if that’s a bug or a feature since I haven’t heard a peep from them… but if anyone at Twitter is listening: thanks guys!
You can now also see the top 50 users with the highest Twitterank scores. What does it mean? Are they any good? It’s hard to say.