Who’s a rising star on Twitter?

I just pushed a new feature out, that I think might be kinda interesting. Basically, you can see how people’s scores have/are changing. Are their scores increasing or decreasing, and how quickly? Who’s hot right now?

If you go over to Twitterank right now, you’ll see a new item in the nav bar. For those of you too lazy to actually go there, here’s what it looks like:

The new Rapid Risers feature shows Twitter users whose Twitteranks have increased the most between the last two times their scores were calculated. Right now, it looks like this:

At the top of the list right now is @stephenfry, and the little green number shows you that his score has shot up 335 points per day! Actually, that’s a lie. The maximum score you can get is 300 and there aren’t any negative scores (yet) so clearly his score can’t have gone up by 335. Rather, 335 is the score increase extrapolated out 24 hours. If you want to find out what really happened, click on his user name and you’ll see:

As you can see, apparently his score went up 94 points between 28 hours ago and 21 hours ago. That’s basically a 100% increase. Way to go Stephen. Now, as to why his score went up suddenly, I don’t know. Based on how the algorithm works, I can tell you that a bunch of people suddenly started talking to him, but why that happened, I don’t know.

So why is this interesting?
I think it’s interesting because Twitterank is a measure of “engagedness” and interestingness. If someone’s score suddenly goes up, it means they’ve suddenly become more engaging or interesting. This could also be interesting to the increasing number of businesses and celebrities on Twitter who are trying to directly reach out to and engage the masses. Twitterank, perhaps for the first time, offers them a way to quantify how well they’re doing.

Major Update!

So… it’s been a while. How’ve you been? Me? I’ve been swell. I quit my job. Been chillin’. Trying to buy some land. Y’know, the usual.

Anyway. I recently was roused out of slumbering funemployment of the non-coding kind, and started working on a major revision for Twitterank. I just pushed it live to twitterank.com, so let me tell you about the new stuff… in bullet list form. Mmm bullet lists.

• No More Passwords! – If you have more of a long term memory than I do, you may recall all the brouhaha Twitterank stirred up when it was first launched. It’s a scam! They’z steelin’ ur passwurds!! OMG! Yeah. Well, in case you had any lingering doubts about Twitterank’s intentions, you can stop being so frackin’ paranoid now. The new version will no longer require your password. Instead, it’ll use this thing called OAuth that Twitter has since adopted (yay), which allows me to make authenticated requests on your behalf, without you giving me your password. So yeah, you can keep your stinkin’ password. I don’t want it anyway.
• New Algorithm – Speaking of authentication, the new algorithm works using the public search APIs which don’t require authentication. This means 2 things. First, you won’t see inconsistent scores like you did in the old version. In the old version, there were actually 2 different algorithms being used, one that used your password, and one that didn’t. Depending on which algorithm was used, you might’ve seen inconsistent scores. That doesn’t happen any more. Secondly, since the new algorithm works great with the public APIs, Twitteranks can be calculated for any Twitter user. And that’s what I’ve been doing. The old version had about 90k users in the DB. The new version has something like 2.5 million, and I’m constantly adding people as I come across them. One unfortunate side-effect of the new algorithm is that a lot of people will have lower scores than the old algorithm that used your passwords, because the search API returns less data than the authenticated APIs. But hey, in the grand scheme of things, that’s not a huge deal.
• Developer APIs – If you’re a developer and you think Twitteranks might be useful for something, check out the nifty new APIs. I’m still not entirely sure what it’ll be good for, but try it out and let me know. It seems like developers are looking for a good way to weed out spammers and SEO/marketers. While that wasn’t the original goal of Twitterank, I’m considering incorporating some anti-spammer filters to the scoring algorithm, so we might get there eventually.
• Frequent Updates – In the old version, scores were only re-calculated if someone looked up their score. In the new version, scores will be re-calculated periodically. Currently, users whose scores are in the top 10,000 will have their scores re-calculated every day or so. That means the Top 50 list will change at least once a day, which should make things a little more interesting.

And… I guess that’s it. Maybe it wasn’t such a huge update. Oh, but wait! I have one last thing! I really do, but it’s not done yet. So I’ll announce it when it’s done.

twitterankiness

As I’ve mentioned elsewhere, my goal with Twitterank was to experiment with a slightly different way of quantifying Twitter users, specifically, using @ replies. Quite a few people have asked why we can’t just use follow counts or the number of posts. My answer has been something along the lines of “there are other signals”, which is another way to say, if we use less obvious signals, we might get less obvious (and potentially better) results. Let me illustrate with a few examples…

Earlier today, I got a tweet from @erikvb who asked: “how is it that your secwet alrowithm calculates 224.941 for @google if they don’t have any tweets at all?” Indeed, @google is not following anyone, only has a few hundred followers, and has no tweets. Yet, it’s ranked in the top 20. Weird, huh?

Of course, if you look at all the @ replies that @google receives, you can see that the Twitterank algorithm is behaving as expected. Now, this is weird and surprising, but does that mean twitterank is “wrong”? I’d say “no”. In fact, it’s actually doing exactly what it’s supposed to do: it’s telling us something we might not have otherwise known. It’s actually telling us (or rather twitterers everywhere are telling us) how influential @google could be, if it were a real twitterer with real tweets. It’s like going to a crowded party and pin pointing influential individuals without even looking at them directly or them saying anything. It’s quite powerful stuff.

Here’s another example I stumbled across. Some of you may know (or know of/about) @caterina, who is probably most famous as the co-founder of Flickr. She has close to 3000 followers on Twitter, and you might think someone like her would be influential in twitterverse. As it turns out, her score is currently 59.35 (72.79 percentile, 1.2% confidence), which is high, but not high for someone with so many followers. It actually makes sense if you look at her tweets. She’s obviously not a heavy twitter user, and her tweets are somewhat cryptic, at least to a casual passerby. But how would she stack up against @joetheplumber? He only has 530 followers and 44 updates. Take a guess, then go find out. Were you surprised?

All this is mildly amusing, but Twitterank itself actually isn’t very interesting. Someone’s twitterank is metadata, not data. It’s a spice, not a main course. So, Twitterank still needs to find a main course to spice up, but that’ll come later.

twitterank for the rest of you

The original version of Twitterank required a user name and password in order to make an authenticated web service request to Twitter and retrieve data that isn’t otherwise accessible. Understandably, many people were reluctant to enter their passwords in a new 3rd party site, and so far, only a small fraction of twitterers have been twitteranked.

The good news is, I have enough twitteranks at this point, that I can start calculating twitteranks for most twitter users, using only publicly accessible data and previously calculated twitteranks. In other words, you can now get any Twitter user’s twitterank, without any passwords. Take for example, @ev, one of the founders of Twitter. He’s so far been cautious enough to not get a twitterank (or he probably doesn’t care), but wouldn’t you want to know what his score would be?

Now you can find out. Check out these screen shots:

screen shot of new form to get anyone's twitterank

screen shot of @ev's estimated twitterank

Notice that under the score, it says “(30.04% confidence)”. Since the version of the Twitterank algorithm which works without a password uses limited public data, there’s a certain amount of guessing involved. Generally speaking, the higher the confidence, the closer the score should be to what Twitterank would calculate if it had access to the user’s password and can retrieve more data.

In any case, head over to twitterank and start looking up the twitteranks of all your friends who’ve resisted the urge so far. Make sure you let them know how they score too 🙂

Quick update

I’m about to go to bed, because I really really need sleep. Today was pretty quiet, fortunately, and by tomorrow, most of Twitterdom will have forgotten about Twitterank. Thank goodness for short attention spans. So this evening, all you’re getting is a redesign (thanks @redct), and a new Tweet for those of you with scores in the upper 80th percentile (BTW, the latest version says “XX% of twitterers” instead of “XX% of you”, because the latter wording makes your ego seem more inflated than it should be).

Also, I experimented with a Twitterank algorithm that uses an API that doesn’t require authentication, but some crucial information is lost in that case, and the scores are 1) low and 2) less meaningful, which is no good. I have another idea I’ll try tomorrow, but for now it’s back to the old algorithm.

if I were to do it over again…

If I were to do this over again, I probably would’ve done 2 things differently. Firstly, I would’ve come up with an algorithm that works with Twitter’s Search API, which doesn’t require authentication. Secondly, I would’ve handled posting to twitter differently, which would’ve sacrificed virality, but after today, I’m not so sure insane rapid growth is really worth it. So… sorry, Twitterverse. I’ll do better next time. Promise.

Also, one last thought before I go to bed (damn, it’s almost 6am here): it doesn’t seem like Twitter rate-limited me, even though I’m certain I went over the 100 requests per 60 minute limit (by orders of magnitude, in fact). I’m not sure if that’s a bug or a feature since I haven’t heard a peep from them… but if anyone at Twitter is listening: thanks guys!

One last thing… top 50

You can now also see the top 50 users with the highest Twitterank scores. What does it mean? Are they any good? It’s hard to say.

Also, here’s a graph showing the distribution of scores. It looks about as you’d expect… Lots of users with relatively low scores, and a few with very high scores.

My side of the story on ZDNet

… is up: Twitterank Creator Speaks

The main thing I wanted to get at was that this was a small, casual hack, that blew up. The Twitterank algorithm isn’t particularly exciting, and it clearly needs more work. When I made the thing, asking for passwords seemed suboptimal, but not a huge deal since I figured most people would just be turned away by it. We live and learn, I guess.

Anyway, like most internet fads, Twitterank will also eventually fade away. But before that happens, something might come out of it. Or not. We’ll see.

New Features!!!

I’ll never convince everybody that this is not a phishing site, but one thing I can do is to try and make it better. So, I’m staying up late to bring you new features. Yay!

Prettier URLs

You can see your (and other people’s) Twitterank by creating URLs like this:

http://twitterank.com/view/%5Btwitterid%5D
e.g. http://twitterank.com/view/t_rank


Where do I stand?

I got a ton of feedback asking “What’s the number mean? Is this good or bad?” That’s my bad. I used the word “rank” but didn’t actually give you a ranking. That’s changed now. Under your score, you’ll see text that says “aprox. XXX percentile”. For example @t_rank currently has a score of 86.41, and “aprox. 83.12 percentile”. That means t_rank has a score higher than approximately 83.12% of twitterers that Twitterank knows about. This ranking is pretty fluid, and will change depending on your score and the distribution of scores in the system.

Some follow up…

Thanks to everyone who posted a comment to the previous entry. Since it seemed like there were common questions, I’ll just answer them all in a single post.

Are you a phishing site? Are you going to steal my account? etc..etc..

No, I am not a phisher. I don’t even store your password. Your password gets used once to calculate your Twitterank, and is never stored on disk or any other permanent storage device. Having said that, people do need to be more careful about giving away their account information. I’m not evil, but the next guy might be.

How can we verify that you aren’t storing our password?

I don’t have a good answer, but I’d be happy to do whatever I can to help convince people that this isn’t a phishing operation. I know that the people who know me will vouch for my character and integrity, but I’m also open to showing a trusted 3rd party the innards of the system. If you have any suggestions, please leave a comment.

Why do you need my password to begin with?

There’s some data I use (but not store) that I need to calculate your Twitterank. There are ways for Twitter to make that data available without requiring you to give out your password to 3rd party sites (Facebook, Yahoo! and others have such systems) but Twitter doesn’t yet offer those options to developers. As soon as Twitter adds more secure authentication mechanisms, I’ll switch to that. (*Give or take up to a few days it’ll take to change the code.)

My score is x… is that good?

There’s no good or bad, per se, but a higher score means you are more active and prominent on Twitter. As you and your friends use Twitter more you should see your scores increase.

What kind of “ranking” is this?

I’m sorry the name is misleading. It’s not really a “ranking” even though the name implies it. The name is an homage to PageRank, an algorithm developed by Google’s founders, and is considered to have been part of the secret to their search engine’s success.

Please delete my account from your system.

Again, I don’t have your password, so I can’t abuse your account. But if you really really want to be removed, leave a comment and I’ll delete you from the system. (But please understand that I’m doing this for fun, and have a day job, so I can’t promise to respond instantly. I’ll do my best though.)