TwittElection at OpenTech

Last Saturday was OpenTech. It was as great as it always is and I’ll write more about what I saw later. But I gave a talk about TwittElection in the afternoon and I thought it might be useful to publish my slides here along with a brief summary of what I said.

TwittElection from Dave Cross
  • I started with a couple of screenshots of what TwittElection is. There’s basically a main page which shows how many days are left until the general election and a page for every constituency which has a widget displaying a Twitter list for all of the candidates in that constituency.
  • Why did I do it? Well I love elections. I have vague memories of one (or perhaps both) of the 1974 general elections and I have closely followed every general election since then. In the 90s I was occasionally¬† one of those annoying people who ask you for your voter number as you’re leaving the polling station and in 2005 I worked all night to make sure that the results on the Guardian web site were up to date.
  • I love Twitter too. Who doesn’t?
  • In 2010 I created a site that monitored the candidates in my local constituency. It wasn’t just Twitter (which was far less important back then) but any kind of web feed that they produced. That’s easy enough to do for one constituency, but it’s a bit more of a challenge for 650.
  • The technology for the system was pretty simple. It was the data that was going to be a lot trickier.
  • Just as I was considering the project, Twitter made a couple of changes which made my life substantially easier. Firstly they increased the number of Twitter lists that each user could create from 20 to 1000 (I needed 650). An secondly, they removed the restriction that Twitter list widgets were tightly associated with a specific list. Under the old system, I would have needed to create 650 individual widgets. Under the new system, I could create one widget and pass it a list ID in order to display any of my 650 lists.
  • I wrote the code in Perl. I made a throwaway remark about it being the “programming languages of champions”. Someone in the audience tweeted that quote and it’s been retweeted rather a lot.
  • I hosted the site on Github Pages in case it got too popular. This was a ridiculous thing to be worried about.
  • I used Bootstrap (of course) and small amounts of various Javascript libraries.
  • The data was harder. We have 650 constituencies and each one will have about six candidates. That means I’ll be looking for data about something like 4,000 candidates. And there’s no official centralised source for this data.
  • Back in November I asked my Twitter followers if they knew of anyone who was collecting lists of candidates and Sam Smith put me in touch with the Democracy Club.
  • At the time, the Democracy Club were just building a new version of YourNextMP – a crowd-sourced list of candidates. It did all that I needed. Which made me very happy. [Note: My talk followed one from the Democracy Club which went into this in far more detail.]
  • So with data from YNMP and my code, the site was build.
  • And it worked pretty well. There were a few bugs (including one that was pointed out by a previous speaker in the same session) but they all got fixed quickly.
  • I became an expert in Twitter error codes.
  • 403 and 429 are the codes that Twitter returns when you make more API requests than you are allowed to. There are two ways to deal with Twitter’s rate limits. You can keep a careful count of your requests and stop before you hit the limits. Or you can keep going until you get one of these codes back at which point you stop. The second option is far simpler. I took the second option. [Note: At this point I forgot to mention that the rate limits were so, well…, limiting that when I got my first complete data dump from YNMP, it took almost two days to build all of the Twitter lists.]
  • 108 means you’re trying to do something with a user that doesn’t exist. Basically, you’ve got the username wrong. Sometimes this is because there’s a typo in the name that YNMP has been given. Sometimes it’s because the user has changed their Twitter username and YNMP doesn’t know about the change yet. One common cause for the latter is when MPs changed their Twitter usernames to remove “MP” whilst the campaign was in progress and legally, there were no MPs. [Note: One of the YNMP developers spoke to me afterwards and admitted that they should have handled Twitter usernames better – for example, they could have stored the ID (which is invariant) rather than the username (which can change).]
  • Error 106 means that the user has blocked you and therefore you can’t add that user to a Twitter list. This seems like strange behaviour given that candidates are presumably using Twitter to publicise their opinions as widely as possible.
  • The first time I was blocked it was @glenntingle, the UKIP candidate for Norwich North.
  • I wondered why he might be blocking me. A friend pointed out that he might be embarrassed by his following habits. It turned out that of the 700 people he followed on Twitter, all but about a dozen of them were young women posting pictures of themselves wearing very little.
  • There was some discussion of this amongst some of my friends. This was apparently noticed by Mr Tingle who first protected his tweets and then deleted his account.
  • I’m not sure how good I feel about hounding a candidate off of Twitter.
  • Another UKIP candidate, @timscottukip, also blocked me. And I heard of another who was running his account in protected mode.
  • Some users didn’t understand crowd-sourcing. Every constituency page included a link to the associated page on YNMP along with text asking people to submit corrections there. But I still got a lot of tweets pointing out errors in my lists.
  • 72% of candidates were on Twitter.
  • Results by party were mixed. 100% of the SNP candidates were on Twitter, but only 51% of UKIP candidates (or perhaps I couldn’t see the others as they were blocking me!)
  • Was it worth it? Well, only 1000 or so people visited the site over the course of the campaign.
  • I haven’t yet seen if I can get any stats on people using the raw Twitter lists rather than looking at my web site.
  • I need to rip out all of the information that is specific to that particular election and encourage people to use the code for other elections. YNMP is based on software called PopIt and I think my code could be useful wherever that is used.
  • There are 1790 days until the next UK general election (as of Saturday 13th June 2015).

The People’s Pamphlet

Update: Ok, yes, we admit it. It was an April Fool’s joke. Well most of it was. I’m not really going to be taking a month off to live in a camper van with Tim and Sim-O (though I’m sure it would have been fun!)

But the wiki really exists. And we really want your help to create a pamphlet that we can distribute to the voters of Mid-Bedfordshire.

I expect that Tim and Sim-O will also be coming clean about now. Here are the full details from Tim.

Hopefully you’ll have seen this morning’s posts by Tim and Sim-O about our new project aiming to bring the politics of accountability to the good burghers of Mid-Narnia. Their MP, Nadine Dorries, is famous for avoiding questions that she doesn’t want to answer so we’re going to to our best to ensure that the Mid-Narnians get the answers they deserve during the election campaign. Tim is in charge of high level strategy, Sim-O has sorted the wheels and I’m the project geek.

A project like this has a few interesting challenges for a geek. Firstly I had to hack a GPS system so that it would guide us through the back of the wardrobe. But secondly, and more importantly, I had to come up with a wiki.

“A wiki?”, I hear you cry, “What would a political campaign want with a wiki?” And I’m glad you asked. Because I’m going to tell you. You see, this isn’t just any old political campaign. No, this is Politics 2.0. We’ll be using the power of Social Media. We’ll be crowd-sourcing some of the campaign’s contents [Is that enough buzzwords, Tim?]

We all have our own ideas for what questions Mad Nad should be answering. Personally, I’d like to ask how many foetuses she saw ripping holes in their mothers’ stomachs whilst she was a nurse. But we need to realise that what’s important to us might not be import to the people of Mid-Narnia. Hence the need for the wiki. This afternoon we’ll be throwing it open for people to suggest questions for Ms Dorries. Once we have broad agreement on the contents of the “people’s pamphlet” we’ll lock the page and print copies of the pamphlet to be distributed in Narnia.

But a wiki is a dangerous thing. Particularly on a contentious subject like this. We need to be sure that everyone who contributes is doing so constructively. So we’ve put some measures in place to try and minimise the amount of vandalism. We’re using a standard installation of MediaWiki to which we added the Confirm Accounts extension. This means that only registered account holders will be able to edit the wiki. And we’ll only being handing out accounts to people with confirmed email addresses. So if anyone starts being stupid, we’ll know exactly where to send our strongly-worded emails of rebuke.

However, it seemed to me that this might not be enough. And late last night I had another idea which I was up until 3am implementing. I’ve written another extension which increases security even more. Now you’ll only be able to edit the wiki if you have a webcam attached to your computer. And the webcam will take photos of you whilst you are editing. The photos will be uploaded to a secure server in Switzerland where they will only be accessed in case of a dispute over the authorship of particular changes. I’m sure I don’t need to emphasise the importance of remaining fully clothed whenever editing the wiki.

Still a few wrinkles to iron out – but once I’m happy with it I’ll be releasing the source code under an open source licence.

Looking forward to seeing some of you in Mid-Narnia over the next few weeks.

social media

Rage Against The X Factor

Well done everyone. We really showed ’em. We showed The Man that he can’t mess with Da Kidz.

But, actually, we really just doubled the amount of money that Sony BMG made out of this year’s christmas number one. And, despite clear instructions to the contrary in the song’s lyrics, we did exactly what They told us to do.

The truth is, of course, somewhere between the two extremes.

I know it’s only a christmas number one and doesn’t mean anything at all, but I couldn’t help feeling really happy when I heard the news last night. It really felt like people were telling Cowell “enough is enough”. We’ve had enough of the anodyne tripe that is pushed out year after year by X Factor, Pop Idol or whatever it’s called this week. There are still some people out there who want to hear real music. Of course, it’s not many people. And it took a co-ordinated internet campaign to get us all pointing in the same direction. And I have little doubt that this is only a blip in the chart’s inexorable decline into mediocrity. This time next year, I’d love to look back on a year of interesting and intelligent number one records. But I know that’s not going to happen,

If nothing else, we can surely chalk this up as another in this year’s impressive list of social media triumphs. 2009 must go down as the year when social media grew up and started to achieve something. We’ve proven time and time again this year that social media can be a force for change. But “with great power comes great responsibility” and all that. We must do all we can to ensure that our new power is used to make positive changes.

We’re going to have a general election next year. And pretty much anyone you speak to seems to think that the result is a foregone conclusion. But those of us who remember the disappointment of 1992 know that the result of a general election is never a foregone conclusion.

If we can get people out buying Rage Against The Machine records, then surely we can also get people to think about their vote. Maybe we can even persuade some of the 40% who didn’t vote last time to get out and make a difference.

Of course the problem with this plan is that the christmas number one is something that people actually care about. Most people don’t care at all about politics or who is running the country.

And if that’s not a sad indictment of the state of British society, then I don’t know what it.

Update: Charlie Brooker, as always, has interesting things to say on the subject.


Who Is To Blame?

Last night I dreamt that the BNP had won seats in the European Parliament. This morning I woke up and it was true. Across the country, 6% of the electorate (well, ok, 6% of the 35% who could be bothered to vote) had decided that they were best represented by racists. In two regions the percentage was high enough for them to win a seat.

It’s difficult to know who to blame for this. Certainly the Labour government have a lot to answer for. If they hadn’t done all they could to alienate the electorate over the last few years then there wouldn’t have been the need for people to vote against them. And it’s not just the Labour Party, of course, every MP who has been abusing the expenses system and destroying the public’s trust in politics has to take a share of the blame. Then there’s the right wing press. They don’t explicitly support the BNP, of course, but papers like the Sun, the Mail and the Standard (“sorry”, my arse) have been slowly but surely creating an environment where the BNP’s poisonous attitudes have moved from being completely unacceptable to something that “middle England” discusses over the dinner table. People who didn’t vote were also to blame. Low turnouts favour minor and extreme parties. Every vote that isn’t cast increases the power of votes that are cast.

And then there are the voters. There’s a load of nonsense talked about the BNP vote being a protest vote and that the people who voted for them not being racists. I’m afraid that doesn’t really bear any kind of scrutiny. There were plenty of protest parties to choose from. Just because you want to give the Labour Party an electoral kicking, that doesn’t lead inexorably to voting to the BNP. There are only two reasons why you would vote for the BNP. Firstly you’re the kind of racist dickhead who agrees with their policies. Or, alternatively, you thought you wanted to vote against Labour and didn’t bother to research the policies of the party you decided to vote for. In either case, you’re a moron.

Just before I went to bed last night, Nick Griffin (the leader of the BNP) was being interviewed by the BBC. What an odious little toad of a man. He was on the defensive throughout the interview. He obviously knows that his opinions are completely offensive to all rational people so he spends all of his time trying to find increasingly bizarre ways to defend them. He claimed that one reason why the BNP only allows white people to join is so that they can use race discrimination legislation against employers who try to sack employees who are found to be members of the party. Every time he opens his mouth, sane people just want to slap him.

I can understand why the BNP want to be a whites-only organisation (it’s because they’re racists) but I don’t understand why UK electoral rules allow it. They want to be seen as a legitimate politcal party. So why can’t we pass a law saying that all UK political parties have to reach certain standards of equality. You know, basic stuff like not discriminating on the basis of gender, race or sexuality. Seems obvious to me.

Griffin also likes to harp on about the “indigenous people”. He really needs a lesson in history. Perhaps someone should send him a copy of Homo Britannicus. The UK doesn’t have any indigenous people. Modern humans arrived in the UK from Europe less than 30,000 years ago. Maybe we should try to send Griffin back to the home of his ancestors. Mind you, it’ll be pretty crowded there as we all have our roots in Ethiopia.

Before the election there was a lot of discussion of the BNP on Twitter. The “#theBNPareTwats” meme got a lot of use. And yet it appeared to achieve nothing. But that’s not really surprising, is it? Twitter is largely an echo chamber. You follow (and are followed by) your friends and people who like what you write. The BNP discussions were largely people who were never going to vote BNP telling other people who were never going to vote BNP not to vote BNP. The chances of any of that witty repartee reaching and converting people who were going to vote BNP was close to zero. So perhaps we’re to blame a bit too. Instead of doing our bit to exchange insults about the BNP on Twitter, we should have been out there knocking on doors and explaining our point of view to people who don’t share it. Perhaps shouting about things on Twitter (and, I’ll admit with slight embarrassment, on blogs) isn’t the best way to change things.

It’s too late now though. There’s nothing we can do[1]. Four the next five years, two regions in the north of England will be represented by racists. We can hope that people saw these elections as unimportant and that they won’t vote the same way in the next General Election. But can we be sure of that? Perhaps we’d better consider doing some real campaigning next time.

Because the thought of BNP supporters in the House of Commons is far too grim to contemplate.

[1] Well, we can (and should) sign Hope Not Hate’s “Not in My Name” petition, but it’s not going to change anything.


Happy Elections

Four years ago I drew a comparison between the US Presidential Election result and the 1992 British General Election. This year the comparison seems even more apt. The footage we saw of the celebrations in Chicago and elsewhere in the US as Obama was confirmed as the winner brought back memories of how we all felt in the UK on the night of May 1st 1997.

I just hope that Barack Obama doesn’t turn into Tony Blair.