Ad verba per numeros

Are Tweets-per-second a meaningful metric?

Thursday, September 19, 2013, 11:53 PM
A couple of weeks ago my colleague @odo asked my opinion about this opinion piece (in Spanish) about the new Tweets-per-second (TPS) record and the purported uselessness of such a metric.

In that piece the journalist argues that TPS is a mostly meaningless metric for two main reasons: on one hand the chat can come from a very vocal minority and, on another hand, there surely must exist (the journalist argument) events with much wider impact than the broadcast of "Castle in the Sky". He cites as examples bin Laden's death or an iPhone presentation.

I have nothing to argue about the vocal minorities argument; it's a real problem when trying to use Twitter to "pulse" public opinion and, in fact, there are some literature about this (see Mustafaraj et al. 2011)

However, the second argument should be justified and to that end I conducted some back-of-the-envelope research using Google Trends. As you probably know, that tool allows you to compare search volumes for different queries filtering by country (and even region), and also at different time slices.

Therefore, I compared queries for "Castle in the Sky", Osama bin Laden, Superbowl, Barack Obama and iPhone. Neither Superbowl or Obama were mentioned in the original newspaper piece but I think they are fair examples.

I've limited my research to Japan since, as explained in Twitter's blog, the new TPS record was due to Japanese users. Hence, I used queries in Japanese. After some Wikipedia research and trying different approaches (e.g. Osama bin Laden, Laden, and bin Laden) I eventually employed the following queries:

  • 天空の城 (Castle in the Sky)
  • ビンラデン (bin Laden)
  • スーパーボウル (Superbowl)
  • オバマ (Obama)
  • iphone
Given that bin Laden's death took place on May 2011 and the TPS record took place on August 2013, I took as time frame from April 2011 to September 2013.

In the following graphs you can check how "Castle in the Sky" is extremely popular in Japan, greatly surpassing bin Laden's death (a meager 7% of the peak number of queries for CitS) and Superbowl (52% of the peak number of queries for CitS). Certainly, Obama's victory in 2012 was much more important: the CitS broadcasting responsible for the TPS record achieved just 69% of the queries that got Obama. However, there is something that seems to be really important in Japan, much more than Obama: the iPhone. CitS peaks at 3% of iPhone queries!

So, in short, impactful events may be uninteresting for Internet users (either Googlers or Twitterers) and, of course, number of queries or TPS do not reflect the importance of the event but the interest or attention that event got from the Internet population.

Now, going back to the usefulness or uselessness of TPS as a metric, many of you may remember past TPS records (this link is for those of you that cannot remember them) and are wondering how this new record compares to previous ones. The truth is that they are simply not comparable for a number of reasons:

  1. Twitter user base was different at each record. First and foremost, larger each time.
  2. Events driving to each TPS record were pretty different even when being of worldwide interest. For instance, Superbowl and FIFA World Cup are followed by different populations with a different approach to Twitter.
  3. Twitter architecture and infrastructure was different each time. In other words, TPS records are IMHO a signal of Twitter's performance at peak time more than a way to compare moments of worldwide-awe.
So, to sum up. TPS provide clues but not the whole story, specially when you compare apples to pears (i.e. different events); however, I think it can be an interesting metric to use when evaluating the "performance" of a single event.

As usual you can reach me at @PFCdgayo.


Back Next