TwittElection at OpenTech

Last Saturday was OpenTech. It was as great as it always is and I’ll write more about what I saw later. But I gave a talk about TwittElection in the afternoon and I thought it might be useful to publish my slides here along with a brief summary of what I said.

• I started with a couple of screenshots of what TwittElection is. There’s basically a main page which shows how many days are left until the general election and a page for every constituency which has a widget displaying a Twitter list for all of the candidates in that constituency.
• Why did I do it? Well I love elections. I have vague memories of one (or perhaps both) of the 1974 general elections and I have closely followed every general election since then. In the 90s I was occasionally  one of those annoying people who ask you for your voter number as you’re leaving the polling station and in 2005 I worked all night to make sure that the results on the Guardian web site were up to date.
• I love Twitter too. Who doesn’t?
• In 2010 I created a site that monitored the candidates in my local constituency. It wasn’t just Twitter (which was far less important back then) but any kind of web feed that they produced. That’s easy enough to do for one constituency, but it’s a bit more of a challenge for 650.
• The technology for the system was pretty simple. It was the data that was going to be a lot trickier.
• Just as I was considering the project, Twitter made a couple of changes which made my life substantially easier. Firstly they increased the number of Twitter lists that each user could create from 20 to 1000 (I needed 650). An secondly, they removed the restriction that Twitter list widgets were tightly associated with a specific list. Under the old system, I would have needed to create 650 individual widgets. Under the new system, I could create one widget and pass it a list ID in order to display any of my 650 lists.
• I wrote the code in Perl. I made a throwaway remark about it being the “programming languages of champions”. Someone in the audience tweeted that quote and it’s been retweeted rather a lot.
• I hosted the site on Github Pages in case it got too popular. This was a ridiculous thing to be worried about.
• I used Bootstrap (of course) and small amounts of various Javascript libraries.
• The data was harder. We have 650 constituencies and each one will have about six candidates. That means I’ll be looking for data about something like 4,000 candidates. And there’s no official centralised source for this data.
• Back in November I asked my Twitter followers if they knew of anyone who was collecting lists of candidates and Sam Smith put me in touch with the Democracy Club.
• At the time, the Democracy Club were just building a new version of YourNextMP – a crowd-sourced list of candidates. It did all that I needed. Which made me very happy. [Note: My talk followed one from the Democracy Club which went into this in far more detail.]
• So with data from YNMP and my code, the site was build.
• And it worked pretty well. There were a few bugs (including one that was pointed out by a previous speaker in the same session) but they all got fixed quickly.
• I became an expert in Twitter error codes.
• 403 and 429 are the codes that Twitter returns when you make more API requests than you are allowed to. There are two ways to deal with Twitter’s rate limits. You can keep a careful count of your requests and stop before you hit the limits. Or you can keep going until you get one of these codes back at which point you stop. The second option is far simpler. I took the second option. [Note: At this point I forgot to mention that the rate limits were so, well…, limiting that when I got my first complete data dump from YNMP, it took almost two days to build all of the Twitter lists.]
• 108 means you’re trying to do something with a user that doesn’t exist. Basically, you’ve got the username wrong. Sometimes this is because there’s a typo in the name that YNMP has been given. Sometimes it’s because the user has changed their Twitter username and YNMP doesn’t know about the change yet. One common cause for the latter is when MPs changed their Twitter usernames to remove “MP” whilst the campaign was in progress and legally, there were no MPs. [Note: One of the YNMP developers spoke to me afterwards and admitted that they should have handled Twitter usernames better – for example, they could have stored the ID (which is invariant) rather than the username (which can change).]
• Error 106 means that the user has blocked you and therefore you can’t add that user to a Twitter list. This seems like strange behaviour given that candidates are presumably using Twitter to publicise their opinions as widely as possible.
• The first time I was blocked it was @glenntingle, the UKIP candidate for Norwich North.
• I wondered why he might be blocking me. A friend pointed out that he might be embarrassed by his following habits. It turned out that of the 700 people he followed on Twitter, all but about a dozen of them were young women posting pictures of themselves wearing very little.
• There was some discussion of this amongst some of my friends. This was apparently noticed by Mr Tingle who first protected his tweets and then deleted his account.
• I’m not sure how good I feel about hounding a candidate off of Twitter.
• Another UKIP candidate, @timscottukip, also blocked me. And I heard of another who was running his account in protected mode.
• Some users didn’t understand crowd-sourcing. Every constituency page included a link to the associated page on YNMP along with text asking people to submit corrections there. But I still got a lot of tweets pointing out errors in my lists.
• 72% of candidates were on Twitter.
• Results by party were mixed. 100% of the SNP candidates were on Twitter, but only 51% of UKIP candidates (or perhaps I couldn’t see the others as they were blocking me!)
• Was it worth it? Well, only 1000 or so people visited the site over the course of the campaign.
• I haven’t yet seen if I can get any stats on people using the raw Twitter lists rather than looking at my web site.
• I need to rip out all of the information that is specific to that particular election and encourage people to use the code for other elections. YNMP is based on software called PopIt and I think my code could be useful wherever that is used.
• There are 1790 days until the next UK general election (as of Saturday 13th June 2015).