In a rush to drag typical “horse race” style political reporting (with all of its associated baggage) into the social media age, many journalists and the news organizations they work for are far too frequently falling into some dangerous traps.
They’re using the lowest-hanging fruit of social media metrics – things like Twitter followers, Facebook likes, mentions and tweets per minute – to track who is “winning” or “losing” the race on social media.
By various accounts, the Romney/Ryan campaign is winning. By others, it’s Obama/Biden. But none of these accounts (at least none that I’ve read) does anything more than really just scratch the surface, using metrics that, while easy to obtain, tell us very little of any substance.
Obama has more Facebook fans! (he’s been president for over three years and has a huge and very active base from his run in 2008…)
Michelle Obama’s speech at the DNC had more “tweets per minute” (that’s TPM, for all the cool kids, according to Twitter’s @gov account) than Romney’s the week previous (I…don’t even know where to start with this one!)
— Twitter Government (@gov) September 6, 2012
Lest you think this phenomenon is restricted to just the tech press, bloggers and the like, even The Washington Post (with their Mention Machine) is, unfortunately, falling into a very similar trap as are many other mainstream news organizations.
I tweeted a few weeks ago that all news organizations ought to let their social media editor (or, if they don’t have one, at least the most tech savvy person in their organization) have an edit of any story that mentions social media to save themselves some embarrassment when their more tech savvy readers see that article in the paper or hear about it on TV.
The same goes for reporting that relies on dubious metrics like “which candidate has the most Twitter followers”. There’s probably someone in your organization who can help and if not, there’s always friendly folks on social media who might also be willing to lend a hand.
How To Fix It
I’ll start off with a few tips to hopefully improve some of this reporting but feel free to reach out any time. Seriously, even if you just want to talk through how best to measure something and get to a significant result, I’d be happy to talk with you.
Ok, here goes: four reasons why the social media metric du jour may not be what it seems:
Twitter followers and Facebook fans can be bought. This is probably the simplest and most common critique of using these metrics for anything really. Since fans and followers can be bought (pretty easily, Google it) these numbers tend to not tell us much at all other than which candidate is best at getting Facebook fans or Twitter followers. (See what I did there?) And besides, if this is the metric you’re going to use, then it’s obviously Eminem/Rihanna 2012! (calling it right now, neither major party ticket stands a chance of racking up that many Facebook fans before November).
“Tweets per minute” or “Mentions” are only a measure of activity. They don’t tell us how many of those tweets said positive things or how many were negative. How many were checkins or other “neutral” tweets that tell us pretty much nothing. Unlike the first measure, these numbers MIGHT (in some cases) tell us something about enthusiasm, or perhaps which side of the political spectrum is more active on Facebook or Twitter, but what they absolutely do not tell us is which side is “winning”.
Sentiment is hard to measure. When you see a reported metric that says X% of tweets were positive and Y% were negative, be deeply skeptical. I know I am, and here’s why: measuring sentiment (well) is still very difficult, time-consuming and expensive. The tools we have are also not nearly as accurate as we would like, so they tend to result in a lot of false positives or negatives. For example: “that speech was sick!” To a computer, that’s negative, but we all know better…right? Now, some tools are getting better at this, but to please the statisticians, you’d likely have to go through and manually code each and every tweet or status update, by hand, and use your human brain (note: still subjective) to better determine the sentiment of each post before you could tally up the final percentage.
These metrics (often) come from dubious (single) sources. Is Twitter really a disinterested party? Should we trust the numbers they report? Do we know anything about their methodology? Could we verify their numbers independently? Is there another source for this data? I have no doubt that they have sophisticated tools for monitoring and analyzing activity on their network, but I’m equally sure they are selective in what they choose to share with the public. They also have a vested interest in making Twitter seem like it’s the driving the political conversation (same for Facebook) when this may or may not actually be the case. I would love for someone to do that story. They also want campaigns to spend more on advertising. So just keep all of that in mind.
I’m going to leave it there for today, but I’m curious about a couple of things:
- What metrics do you find valuable and how do you use them in your reporting?
- What do you think of these “horse race” type stories and how do you think we can make them more meaningful (or is that even possible)?