Categories
tech

Opentech 2015

It’s three weeks since I was at this year’s Opentech conference and I haven’t written my now-traditional post about what I saw. So let’s put that right.

I got there rather later than expected. It was a nice day, so I decided that I would walk from Victoria station to ULU. That route took me past Buckingham Palace and up the Mall. But I hadn’t realised that the Trooping of the Colour was taking place which made it impossible to get across the Mall and into Trafalgar Square. Of course I didn’t realise that until I reached the corner of St James Park near the Admiralty Arch. A helpful policeman explained what was going on and suggested that my best bet was to go to St James Park tube station and get the underground to Embankment. This involved walking most of the way back through the park. And when I got to the tube station it was closed. So I ended up walking to Embankment.

All of which meant I arrived about forty minutes later than I wanted to and the first session was in full swing as I got there.

So what did I see?

Being Female on the Internet – Sarah Brown

This is the talk I missed most of. And I had really wanted to see this talk. As I arrived she was just finishing her talk, and the audio doesn’t seem to be on the Opentech web site.

Selling ideas – Vinay Gupta

I think I didn’t concentrate on this as much as I should have. It was basically a talk about marketing – which is something that the geek community needs to get better at. Vinay illustrated his talk with examples from his Hexayurt project.

RIPA 2 – Ian Brown

Ian talked about potential changes to the Regulation of Investigatory Powers Act. It was all very scary stuff. The slides are online.

The 3rd year of Snowdenia — Caroline Wilson Palow

Caroline talked about Ed Snowden’s work and the way it is changing the world.

Privacy: I do not think that word means what you think it means — Kat Matfield

Kat has been doing research into how end users view privacy on the web. It’s clear that people are worried about their privacy but that they don’t know enough about the subject in order to focus their fear (and anger) at the right things.

The State of the Network Address — Bill Thompson

Bill thinks that many of the world’s woes are caused by people in power abusing the technological tools that geeks have build. And he would like us to do more to prevent them doing that.

The State of Data — Gavin Starks

Gavin works for the Open Data Institute. It’s his job to help organisations to release as much data as possible and to help the rest of us to make as much use of that data as possible. He talked about the problems that he sees in this new data-rich world.

Using data to find patterns in law — John Sheridan

John is using impressive text parsing and manipulation techniques to investigate the UK’s legislation. It sounds like a really interesting project.

Scenic environments, healthy environments? How open data offers answers to this age-old question. — Chanuki Seresinhe

The answer seems to be yes :-)

I stood as a candidate, and… — James Smith

James stood as a candidate in this year’s general election, using various geek tools to power his campaign. He talked through the story of his campaign and tried to encourage others to try the same thing in the next election.

Democracy Club — Sym Roe

The Democracy Club built an number of tools and web sites which built databases of information about candidates in the recent election – and then shared that data with the public. Sym explained why and how these tools were built.

The Twitter Election? — Dave Cross

This was me. I’ve already written up my talk.

Election: what’s next

This was supposed to follow my talk. Bill Thompson had some ideas to start the discussion and suggested that anyone interested retired to the bar. I put away my laptop and various other equipment and the set off to find them. But I failed, so I went home instead.

Yet another massively successful event. Thanks, as always, to all of the speakers and organisers.

Categories
tech

TwittElection at OpenTech

Last Saturday was OpenTech. It was as great as it always is and I’ll write more about what I saw later. But I gave a talk about TwittElection in the afternoon and I thought it might be useful to publish my slides here along with a brief summary of what I said.

TwittElection from Dave Cross
  • I started with a couple of screenshots of what TwittElection is. There’s basically a main page which shows how many days are left until the general election and a page for every constituency which has a widget displaying a Twitter list for all of the candidates in that constituency.
  • Why did I do it? Well I love elections. I have vague memories of one (or perhaps both) of the 1974 general elections and I have closely followed every general election since then. In the 90s I was occasionally  one of those annoying people who ask you for your voter number as you’re leaving the polling station and in 2005 I worked all night to make sure that the results on the Guardian web site were up to date.
  • I love Twitter too. Who doesn’t?
  • In 2010 I created a site that monitored the candidates in my local constituency. It wasn’t just Twitter (which was far less important back then) but any kind of web feed that they produced. That’s easy enough to do for one constituency, but it’s a bit more of a challenge for 650.
  • The technology for the system was pretty simple. It was the data that was going to be a lot trickier.
  • Just as I was considering the project, Twitter made a couple of changes which made my life substantially easier. Firstly they increased the number of Twitter lists that each user could create from 20 to 1000 (I needed 650). An secondly, they removed the restriction that Twitter list widgets were tightly associated with a specific list. Under the old system, I would have needed to create 650 individual widgets. Under the new system, I could create one widget and pass it a list ID in order to display any of my 650 lists.
  • I wrote the code in Perl. I made a throwaway remark about it being the “programming languages of champions”. Someone in the audience tweeted that quote and it’s been retweeted rather a lot.
  • I hosted the site on Github Pages in case it got too popular. This was a ridiculous thing to be worried about.
  • I used Bootstrap (of course) and small amounts of various Javascript libraries.
  • The data was harder. We have 650 constituencies and each one will have about six candidates. That means I’ll be looking for data about something like 4,000 candidates. And there’s no official centralised source for this data.
  • Back in November I asked my Twitter followers if they knew of anyone who was collecting lists of candidates and Sam Smith put me in touch with the Democracy Club.
  • At the time, the Democracy Club were just building a new version of YourNextMP – a crowd-sourced list of candidates. It did all that I needed. Which made me very happy. [Note: My talk followed one from the Democracy Club which went into this in far more detail.]
  • So with data from YNMP and my code, the site was build.
  • And it worked pretty well. There were a few bugs (including one that was pointed out by a previous speaker in the same session) but they all got fixed quickly.
  • I became an expert in Twitter error codes.
  • 403 and 429 are the codes that Twitter returns when you make more API requests than you are allowed to. There are two ways to deal with Twitter’s rate limits. You can keep a careful count of your requests and stop before you hit the limits. Or you can keep going until you get one of these codes back at which point you stop. The second option is far simpler. I took the second option. [Note: At this point I forgot to mention that the rate limits were so, well…, limiting that when I got my first complete data dump from YNMP, it took almost two days to build all of the Twitter lists.]
  • 108 means you’re trying to do something with a user that doesn’t exist. Basically, you’ve got the username wrong. Sometimes this is because there’s a typo in the name that YNMP has been given. Sometimes it’s because the user has changed their Twitter username and YNMP doesn’t know about the change yet. One common cause for the latter is when MPs changed their Twitter usernames to remove “MP” whilst the campaign was in progress and legally, there were no MPs. [Note: One of the YNMP developers spoke to me afterwards and admitted that they should have handled Twitter usernames better – for example, they could have stored the ID (which is invariant) rather than the username (which can change).]
  • Error 106 means that the user has blocked you and therefore you can’t add that user to a Twitter list. This seems like strange behaviour given that candidates are presumably using Twitter to publicise their opinions as widely as possible.
  • The first time I was blocked it was @glenntingle, the UKIP candidate for Norwich North.
  • I wondered why he might be blocking me. A friend pointed out that he might be embarrassed by his following habits. It turned out that of the 700 people he followed on Twitter, all but about a dozen of them were young women posting pictures of themselves wearing very little.
  • There was some discussion of this amongst some of my friends. This was apparently noticed by Mr Tingle who first protected his tweets and then deleted his account.
  • I’m not sure how good I feel about hounding a candidate off of Twitter.
  • Another UKIP candidate, @timscottukip, also blocked me. And I heard of another who was running his account in protected mode.
  • Some users didn’t understand crowd-sourcing. Every constituency page included a link to the associated page on YNMP along with text asking people to submit corrections there. But I still got a lot of tweets pointing out errors in my lists.
  • 72% of candidates were on Twitter.
  • Results by party were mixed. 100% of the SNP candidates were on Twitter, but only 51% of UKIP candidates (or perhaps I couldn’t see the others as they were blocking me!)
  • Was it worth it? Well, only 1000 or so people visited the site over the course of the campaign.
  • I haven’t yet seen if I can get any stats on people using the raw Twitter lists rather than looking at my web site.
  • I need to rip out all of the information that is specific to that particular election and encourage people to use the code for other elections. YNMP is based on software called PopIt and I think my code could be useful wherever that is used.
  • There are 1790 days until the next UK general election (as of Saturday 13th June 2015).