Brighton SEO

Last Friday, I was in Brighton for the Brighton SEO conference. It was quite a change for me. I’ve been going to technical conferences for about twenty years or so, but the ones I go to tend to be rather grass-roots affairs like YAPC or Opentech. Even big conferences like FOSDEM have a very grass-roots feel to them.

Brighton SEO is different. Brighton SEO is a huge conference and there is obviously a lot of money sloshing around in the SEO industry. I’ve been to big technical conferences like OSCON, but tickets for conferences like that are expensive. Brighton SEO is free for most attendees. They must have lots of very generous sponsors.

The conference took place at the Brighton Centre. The people I was staying with in Brighton asked how much of the centre the conference took up. Turns out the answer was “all of it”. Not bad for a conference that started out as a few friends meeting in a pub just a few years ago.

The conference day is broken up into four sessions. It was easy enough to choose sessions that sounded useful to me. I’ve only really been looking into SEO since the start of the year and I’m more interested in the technical side of SEO. I don’t have much time for things like content marketing and keyword tracking (although I’m sure they have their place).

So I started in a session about Javascript and Frameworks. This began with 

This was followed by Emily Grossman talking about Progressive Web Apps – which are basically web sites bundled up to look like smartphone apps. I plan to try this out with a couple of my sites soon.

The final talk in this session was David Lockie on Using Open Source Software to Speed Up Your Roadmap. I’ve used pretty much nothing but open source software for the last thirty years so I needed no convincing that he was advocating a good approach.

A quick coffee break and then the second session started. I chose a session on Onsite SEO. I was amused to see that even after only eight months of working on SEO, I could pick a session that was too basic for me.

The session started with Chloé Bodard on SEO quick wins from a technical check. This was interesting because it’s close to a service that I’m thinking of offering to clients. But I learned very little.

Chloé was followed by Sébastien Monnier with a talk entitled How Google Tag Manager Can Save Your SEO. Earlier this year I was involved in discussions where a client was talking about using Google Tag Manager. Another developer and I managed to persuade them that it was a bad idea as GTM inserts data into the page using Javascript and the right approach was to ensure that the correct data was inserted into the page as it was first built. It was gratifying to hear Sébastien (who is a former Google employee) say that (and I’m paraphrasing) “GTM is really a tool for SEOs to work around bad developers”.

The final talk in the session was Aysun Akarsu and On the Road to HTTPS Worldwide. This was a good talk, but it would have been far more useful to me before we moved ZPG’s three major web sites to https earlier this year.

It was then lunch and with some ZPG colleagues I wandered off to sample some of Brighton’s excellent food.

For the first session in the afternoon, I chose three talks on Technical SEO. We started with Peter Nikolow with Quick and Dirty Server-Side Hacks to Improve Your SEO. To be honest, I think Peter misjudged his audience. I was following the conference hashtag on Twitter and there were a lot of people saying that his talk was going over their head. It didn’t go over my head, but I thought that some of his server-side knowledge looked a little dated.

Then there was Dominic Woodman with a talk entitled Advanced Site Architecture – Testing architecture & keyword/page groupings. There was a lot of good stuff in this talk and I need to go back over the slides in a lot more detail.

The session ended with Dawn Anderson talking about Generational Cruft in SEO – There is Never a ‘New Site’ When There’s History. A lot of this talk rang very true for me. In fact just the week before, I had been configuring a web site to return 410 responses when Google and Bing came looking for XML sitemaps that had been switched off two years ago.

For the fourth and final session, I chose the talks on Crawl and Indexation. This session began with Chris Green giving a talk called Robots: X, Meta & TXT – The Snog, Marry & Avoid of the Web Crawling World. The title was slightly cringe-making, but there was some good content about using the right tools to ensure that pages you don’t want crawled don’t end up in Google’s index.

I think I wass getting tired by this point. I confess that I don’t remember much about François Goube’s How to Optimise Your Crawl Budget. I’m sure it was full of good stuff.

There was no chance of dozing off during Cindy Krum’s closing talk Understanding the Impact of Mobile-First Indexing (the link goes to the slides for a slightly older version of the talk). This was a real wake-up call about how Google’s indexing will change over the next few years.

I had a great time at my first Brighton SEO. I wonder how much of that is down to the fact that for probably the first time this millennium I was at a conference and not giving a talk. But I’m already thinking about a talk for the next Brighton SEO conference.

Many thanks to all of the organisers and speakers. I will be back.

Twitter’s Early Adopters

You’ll be seeing that tweet a lot over the next few days. It’s the first ever public tweet that was posted to the service we now know as Twitter. And it was sent ten years ago by Jack Dorsey, one of Twitter’s founders.

Today, Twitter has over a hundred million users, who send 340 million tweets a day (those numbers are almost certainly out of date already) but I thought it would be interesting to look back and look at Twitter’s earliest users.

Every Twitter user has a user ID. That’s an integer which uniquely identifies them to the system. This is a simple incrementing counter[1]. You can use a site like MyTwitterID to get anyone’s ID given their Twitter username. It’s worth noting that you can change your username, but your ID is fixed. When I registered a new account last week, I got an ID that was eighteen digits long. But back in 2006, IDs were far shorter. Jack’s ID, for example, is 12. That’s the lowest currently active ID on the system. I assume that the earlier numbers were used for test accounts.

Using the Twitter API you can write a program that will give you details of a user from their ID. Yesterday I wrote a simple program to get the details of the first 100,000 Twitter users (the code is available on Github). The results from running the program are online. That’s a list of all of the currently active Twitter users with an ID less than 100,000.

The first thing you’ll notice is that there are far fewer than you might expect. The API only returns details on currently active users. So anyone who has closed their account won’t be listed. I expected that perhaps 20-25% of accounts might fall into that category, but it was much higher than that.

There are 12,435 users in the file. That means that 87,500 of the first 100,000 Twitter accounts are no longer active. That was such a surprise to me that I assumed there was a bug in my program. But I can’t find one. It really looks like almost 90% of the early Twitter users are no longer using the service.

The dates that the account were created range from Jack‘s on 21st March 2006 to Jeremy Hulette (ID 99983 – the closest we have to 100,000) exactly nine months later on 21st December 2006.  I guess you could get a good visualisation of Twitter’s early growth by plotting ID against creation date – but I’ll leave that to someone else.

My file also contains location. But it’s important to note that I’m getting the location that is currently associated with that account – not the original location (I wonder if Twitter still have that information). I know a large number of people who were in London when they joined Twitter by who are now in San Francisco, so any conclusions you draw from the location field are necessarily sketchy. But bearing that in mind, here are some “firsts”.

  • First non-Californian: rabble (ID 22, PDX & MVD)
  • First non-America: florian (ID 38, Berlin)
  • First Brit: blaine (ID 246, London)

That last one seems a little high to me. I might have missed someone earlier who didn’t put “UK” in their location.

So who’s on the list? Is there anyone famous? Not that I’ve seen yet. Oh, there are well-known geeks on the list. But no-one you’d describe as a celebrity. No musicians, no actors, no politicians, no footballers or athletes. I may have missed someone – please let me know if you spot anyone.

Oh, and I’m on the list. I’m at number 14753. I signed up (as @davorg) at 11:30 on Wednesday 22nd November 2006. I suspect I’m one of the first thousand or so Brits on the list – but it’s hard to be sure of that.

Anyway, happy birthday to Twitter. I hope that someone finds this data interesting. Let me know what you find.

[1] Actually, there’s a good chance that this is no longer the case – but it was certainly true back in 2006.

Writing Books (The Easy Bit)

Last night I spoke at a London Perl Mongers meeting. As part of the talk I spoke about a toolchain that I have been using for creating ebooks. In this article I’ll go into a little more detail about the process.

Basically, we’re talking about a process that takes one or more files in some input format and (as easily as possible) turns them into one or more output formats which can be described as “ebooks”. So before we can decided which tools we need, we should decide what those various file formats should be.

For my input format I chose Markdown. This is a text-based format that has become popular amongst geeks over the last few years. Geeks tend to like text-based formats more than the proprietary binary formats like those produced by word processors. This is for a number of reasons. You can read them without any specialised tools. You’re not tied down to using specific tools to create them. And it’s generally easier to store them in a revision management system like Github.

For my output formats, I wanted EPUB and Mobipocket. EPUB is the generally accepted standard for ebooks and Mobipocket is the ebook format that Amazon use. And I also wanted to produce PDFs, just because they are easy to read on just about any platform.

(As an aside, you’ll notice that I said nothing in that previous paragraph about DRM. That’s simply because nice people don’t do that.)

Ok, so we know what file formats we’ll be working with. Now we need to know a) how we create the input format and b) how we convert between the various formats. Creating the Markdown files is easy enough. It’s just a text file, so any text editor would do the job (it would be interesting to find out if any word processor can be made to save text as Markdown).

To convert our Markdown into EPUB, we’ll need a new tool. Pandoc describes itself as “a universal document converter”. It’s not quite universal (otherwise that would be the only tool that we would need), but it is certainly great for this job. Once you have installed Pandoc, the conversion is simple:

pandoc -o your_book.epub title.txt --epub-metadata=metadata.xml --toc --toc-depth=2

There are two extra files you need here (I’m not sure why it can’t all be in the same file, but that’s just the way it seems to be). The first (which I’ve called “title.txt”), contains two lines. The first line has the title of your book and the second has the author’s name. Each line needs to start with a “%” character. So it might look like this:

% Your title
% Your name

The second file (which I’ve called “metadata.xml”) contains various pieces of information about the book. It’s (ew!) XML and looks like this:

<metadata xmlns:dc="">
<dc:title id="main">Your Title</dc:title>
<meta refines="#main" property="title-type">main</meta>
<dc:creator opf:file-as="Surname, Forename" opf:role="aut">Forename Surname</dc:creator>
<dc:publisher>Your name</dc:publisher>
<dc:date opf:event="publication">2015-08-14</dc:date>
<dc:rights>Copyright ©2015 by Your Name</dc:rights> </metadata>

So after creating those files and running that command, you’ll have an EPUB file. Next we want to convert that to a Mobipocket file so that we can distribute our book through Amazon. Unsurprisingly, the easiest way to do that is to use a piece of software that you get from Amazon. It’s called Kindlegen and you can download it from their site. Once it is installed, the conversion is as simple as:

kindlegen perlwebbook.epub

This will leave you with a file called “” which you can upload to Amazon.

There’s one last conversion that you might need. And that’s converting the EPUB to PDF. Pandoc will make that conversion for you. But it does it using a piece of software called LaTeX which I’ve never had much luck with. So I looked for an alternative solution and found it in Calibre. Calibre is mainly an ebook management tool, but it also converts between many ebook formats. It’s pretty famous for having a really complex user interface but, luckily for us, there’s a command line program called “ebook-convert” – which we can use.

ebook-convert perlwebbook.epub perlwebbook.pdf

And that’s it. We start with a Markdown file and end up with an ebook in three formats. Easy.

Of course, that really is the easy part. There’s a bit that comes before (actually writing the book) and a bit that comes after (marketing the book) and they are both far harder. Last year I read a book called Author, Publisher, Entrepreneur which covered these three steps to a very useful level of detail. Their step two is rather different to mind (they use Microsoft Word if I recall correctly) but what they had to say about the other steps was very interesting. You might find it interesting if you’re thinking of writing (and self-publishing) a book.

I love the way that ebooks have democratised the publishing industry. Anyone can write and publish a book and make it available to everyone through the world’s largest book distribution web site.

So what are you waiting for? Get writing. If you find my toolchain interesting (or if you have any comments on it) then please let me know.

And let me know what you’ve written.

Financial Account Aggregation

Three years ago, I wrote a blog post entitled Internet Security Rule One about the stupidity of sharing your passwords with anyone. I finished that post with a joke.

Look, I’ll tell you what. I’ve got a really good idea for an add-on for your online banking service. Just leave the login details in a comment below and I’ll set it up for you.

It was a joke because it was obviously ridiculous. No-one would possibly think it was a good idea to share their banking password with anyone else.

I should know not to make assumptions like that.

Yesterday I was made aware of a service called Money Dashboard. Money Dashboard aggregates all of your financial accounts so that you can see them all in one convenient place. They can then generate all sorts of interesting reports about where your money is going and can probably make intelligent suggestions about things you can do to improve your financial situation. It sounds like a great product. I’d love to have access to a system like that.

There’s one major flaw though.

In order to collect the information they need from all of your financial accounts, they need your login details for the various sites that you use. And that’s a violation of the Internet Security Rule One. You should never give your passwords to anyone else – particularly not passwords that are as important as your banking password.

I would have thought that was obvious. But they have 100,000 happy users.

Of course they have have a page on their site telling you exactly how securely they store your details. They use “industry-standard security practices”, their application is read-only “which means it cannot be used for withdrawals, payments or to transfer your funds”. They have “selected partners with outstanding reputations and extensive experience in security solutions”. It all sounds lovely. But it really doesn’t mean very much.

It doesn’t mean very much because at the heart of their system, they need to log on to your bank’s web site pretending to be you in order to get hold of your account information. And that means that no matter how securely they store your passwords, at some point they need to be able to retrieve them in plain text so they can use them to log on to your banks web site. So there must be code somewhere in their system which punches through all of that security and gets the string “pa$$word”. So in the worst case scenario, if someone compromises their servers they will be able to get access to your passwords.

If that doesn’t convince you, then here’s a simpler reason for not using the service. Sharing your passwords with anyone else is almost certainly a violation of your bank’s terms and conditions. So if someone does get your details from Money Dashboard’s system and uses that information to wreak havoc in your bank account – good luck getting any compensation.

Here, for example, are First Direct’s T&Cs about this (in section 9.1):

You must take all reasonable precautions to keep safe and prevent fraudulent use of any cards, security devices, security details (including PINs, security numbers, passwords or other details including those which allow you to use Internet Banking and Telephone Banking).

These precautions include but are not limited to all of the following, as applicable:


  • not allowing anyone else to have or use your card or PIN or any of our security devices, security details or password(s) (including for Internet Banking and Telephone Banking) and not disclosing them to anyone, including the police, an account aggregation service that is not operated by us

Incidentally, that “not operated by us” is a nice piece of hubris. First Direct run their own account aggregation service which, of course, they trust implicitly. But they can’t possibly trust anybody else’s service.

I started talking about this on Twitter yesterday and I got this response from the @moneydashboard account. It largely ignores the security aspects and concentrates on why you shouldn’t worry about breaking your bank’s T&Cs. They seem to be campaigning to get T&Cs changed so allow explicit exclusions for sharing passwords with account aggregation services.

I think this is entirely wrong-headed. I think there is a better campaign that they should be running.

As I said above, I think that the idea of an account aggregation service is great. I would love to use something like Money Dashboard. But I’m completely unconvinced by their talk of security. They need access to your passwords in plain text. And it doesn’t matter that their application only reads your data. If someone can extract your login details from Money Dashboard’s systems then they can do whatever they want with your money.

So what’s the solution? Well I agree with one thing that Money Dashboard say in their statement:

All that you are sharing with Money Dashboard is data; data which belongs to you. You are the customer, you should be telling the bank what to do, not the other way around!

We should be able to tell our banks to share our data with third parties. But we should be able to do it in a manner that doesn’t entail giving anyone full access to our accounts. The problem is that there is only one level of access to your bank account. If you have the login details then you can do whatever you want. But what if there was a secondary set of access details – ones that could only read from the account?

If you’ve used the web much in recent years, you will have become familiar with this idea. For example, you might have wanted to give a web app access to your Twitter account. During this process you will be shown a screen (which, crucially, is hosted on Twitter’s web site, not the new app) asking if you want to grant rights to this new app. And telling you which rights you are granting (“This app wants to read your tweets.” “This app wants to tweet on you behalf.”) You can decide whether or not to grant that access.

This is called OAuth. And it’s a well-understood protocol. We need something like this for the finance industry. So that I can say to First Direct, “please allow this app to read my account details, but don’t let them change anything”. If we had something like that, then all of these problems will be solved. The Money Dashboard statement points to the Financial Data and Technology Association – perhaps they are the people to push for this change.

I know why Money Dashboard are doing what they are doing. And I know they aren’t the only ones doing it (Mint, for example, is a very popular service in the US). And I really, really want what they are offering. But just because a service is a really good idea, shouldn’t mean that you take technical short-cuts to implement it.

I think that the “Financial OAuth” I mentioned above will come about. But the finance industry is really slow to embrace change. Perhaps the Financial Data and Technology Association will drive it. Perhaps one forward-thinking bank will implement it and other bank’s customers will start to demand it.

Another possibility is that someone somewhere will lose a lot of money through sharing their details with a system like this and governments will immediately close them all down until a safer mechanism is in place.

I firmly believe that systems like Money Dashboard are an important part of the future. I just hope that they are implemented more safely than the current generation.


Opentech 2015

It’s three weeks since I was at this year’s Opentech conference and I haven’t written my now-traditional post about what I saw. So let’s put that right.

I got there rather later than expected. It was a nice day, so I decided that I would walk from Victoria station to ULU. That route took me past Buckingham Palace and up the Mall. But I hadn’t realised that the Trooping of the Colour was taking place which made it impossible to get across the Mall and into Trafalgar Square. Of course I didn’t realise that until I reached the corner of St James Park near the Admiralty Arch. A helpful policeman explained what was going on and suggested that my best bet was to go to St James Park tube station and get the underground to Embankment. This involved walking most of the way back through the park. And when I got to the tube station it was closed. So I ended up walking to Embankment.

All of which meant I arrived about forty minutes later than I wanted to and the first session was in full swing as I got there.

So what did I see?

Being Female on the Internet – Sarah Brown

This is the talk I missed most of. And I had really wanted to see this talk. As I arrived she was just finishing her talk, and the audio doesn’t seem to be on the Opentech web site.

Selling ideas – Vinay Gupta

I think I didn’t concentrate on this as much as I should have. It was basically a talk about marketing – which is something that the geek community needs to get better at. Vinay illustrated his talk with examples from his Hexayurt project.

RIPA 2 – Ian Brown

Ian talked about potential changes to the Regulation of Investigatory Powers Act. It was all very scary stuff. The slides are online.

The 3rd year of Snowdenia — Caroline Wilson Palow

Caroline talked about Ed Snowden’s work and the way it is changing the world.

Privacy: I do not think that word means what you think it means — Kat Matfield

Kat has been doing research into how end users view privacy on the web. It’s clear that people are worried about their privacy but that they don’t know enough about the subject in order to focus their fear (and anger) at the right things.

The State of the Network Address — Bill Thompson

Bill thinks that many of the world’s woes are caused by people in power abusing the technological tools that geeks have build. And he would like us to do more to prevent them doing that.

The State of Data — Gavin Starks

Gavin works for the Open Data Institute. It’s his job to help organisations to release as much data as possible and to help the rest of us to make as much use of that data as possible. He talked about the problems that he sees in this new data-rich world.

Using data to find patterns in law — John Sheridan

John is using impressive text parsing and manipulation techniques to investigate the UK’s legislation. It sounds like a really interesting project.

Scenic environments, healthy environments? How open data offers answers to this age-old question. — Chanuki Seresinhe

The answer seems to be yes :-)

I stood as a candidate, and… — James Smith

James stood as a candidate in this year’s general election, using various geek tools to power his campaign. He talked through the story of his campaign and tried to encourage others to try the same thing in the next election.

Democracy Club — Sym Roe

The Democracy Club built an number of tools and web sites which built databases of information about candidates in the recent election – and then shared that data with the public. Sym explained why and how these tools were built.

The Twitter Election? — Dave Cross

This was me. I’ve already written up my talk.

Election: what’s next

This was supposed to follow my talk. Bill Thompson had some ideas to start the discussion and suggested that anyone interested retired to the bar. I put away my laptop and various other equipment and the set off to find them. But I failed, so I went home instead.

Yet another massively successful event. Thanks, as always, to all of the speakers and organisers.

TwittElection at OpenTech

Last Saturday was OpenTech. It was as great as it always is and I’ll write more about what I saw later. But I gave a talk about TwittElection in the afternoon and I thought it might be useful to publish my slides here along with a brief summary of what I said.

  • I started with a couple of screenshots of what TwittElection is. There’s basically a main page which shows how many days are left until the general election and a page for every constituency which has a widget displaying a Twitter list for all of the candidates in that constituency.
  • Why did I do it? Well I love elections. I have vague memories of one (or perhaps both) of the 1974 general elections and I have closely followed every general election since then. In the 90s I was occasionally  one of those annoying people who ask you for your voter number as you’re leaving the polling station and in 2005 I worked all night to make sure that the results on the Guardian web site were up to date.
  • I love Twitter too. Who doesn’t?
  • In 2010 I created a site that monitored the candidates in my local constituency. It wasn’t just Twitter (which was far less important back then) but any kind of web feed that they produced. That’s easy enough to do for one constituency, but it’s a bit more of a challenge for 650.
  • The technology for the system was pretty simple. It was the data that was going to be a lot trickier.
  • Just as I was considering the project, Twitter made a couple of changes which made my life substantially easier. Firstly they increased the number of Twitter lists that each user could create from 20 to 1000 (I needed 650). An secondly, they removed the restriction that Twitter list widgets were tightly associated with a specific list. Under the old system, I would have needed to create 650 individual widgets. Under the new system, I could create one widget and pass it a list ID in order to display any of my 650 lists.
  • I wrote the code in Perl. I made a throwaway remark about it being the “programming languages of champions”. Someone in the audience tweeted that quote and it’s been retweeted rather a lot.
  • I hosted the site on Github Pages in case it got too popular. This was a ridiculous thing to be worried about.
  • I used Bootstrap (of course) and small amounts of various Javascript libraries.
  • The data was harder. We have 650 constituencies and each one will have about six candidates. That means I’ll be looking for data about something like 4,000 candidates. And there’s no official centralised source for this data.
  • Back in November I asked my Twitter followers if they knew of anyone who was collecting lists of candidates and Sam Smith put me in touch with the Democracy Club.
  • At the time, the Democracy Club were just building a new version of YourNextMP – a crowd-sourced list of candidates. It did all that I needed. Which made me very happy. [Note: My talk followed one from the Democracy Club which went into this in far more detail.]
  • So with data from YNMP and my code, the site was build.
  • And it worked pretty well. There were a few bugs (including one that was pointed out by a previous speaker in the same session) but they all got fixed quickly.
  • I became an expert in Twitter error codes.
  • 403 and 429 are the codes that Twitter returns when you make more API requests than you are allowed to. There are two ways to deal with Twitter’s rate limits. You can keep a careful count of your requests and stop before you hit the limits. Or you can keep going until you get one of these codes back at which point you stop. The second option is far simpler. I took the second option. [Note: At this point I forgot to mention that the rate limits were so, well…, limiting that when I got my first complete data dump from YNMP, it took almost two days to build all of the Twitter lists.]
  • 108 means you’re trying to do something with a user that doesn’t exist. Basically, you’ve got the username wrong. Sometimes this is because there’s a typo in the name that YNMP has been given. Sometimes it’s because the user has changed their Twitter username and YNMP doesn’t know about the change yet. One common cause for the latter is when MPs changed their Twitter usernames to remove “MP” whilst the campaign was in progress and legally, there were no MPs. [Note: One of the YNMP developers spoke to me afterwards and admitted that they should have handled Twitter usernames better – for example, they could have stored the ID (which is invariant) rather than the username (which can change).]
  • Error 106 means that the user has blocked you and therefore you can’t add that user to a Twitter list. This seems like strange behaviour given that candidates are presumably using Twitter to publicise their opinions as widely as possible.
  • The first time I was blocked it was @glenntingle, the UKIP candidate for Norwich North.
  • I wondered why he might be blocking me. A friend pointed out that he might be embarrassed by his following habits. It turned out that of the 700 people he followed on Twitter, all but about a dozen of them were young women posting pictures of themselves wearing very little.
  • There was some discussion of this amongst some of my friends. This was apparently noticed by Mr Tingle who first protected his tweets and then deleted his account.
  • I’m not sure how good I feel about hounding a candidate off of Twitter.
  • Another UKIP candidate, @timscottukip, also blocked me. And I heard of another who was running his account in protected mode.
  • Some users didn’t understand crowd-sourcing. Every constituency page included a link to the associated page on YNMP along with text asking people to submit corrections there. But I still got a lot of tweets pointing out errors in my lists.
  • 72% of candidates were on Twitter.
  • Results by party were mixed. 100% of the SNP candidates were on Twitter, but only 51% of UKIP candidates (or perhaps I couldn’t see the others as they were blocking me!)
  • Was it worth it? Well, only 1000 or so people visited the site over the course of the campaign.
  • I haven’t yet seen if I can get any stats on people using the raw Twitter lists rather than looking at my web site.
  • I need to rip out all of the information that is specific to that particular election and encourage people to use the code for other elections. YNMP is based on software called PopIt and I think my code could be useful wherever that is used.
  • There are 1790 days until the next UK general election (as of Saturday 13th June 2015).

First Direct Update

Earlier in the week I talked about my concerns with First Direct’s new password policy. I got an email from them about this, but it really wasn’t very reassuring.

But I kept digging. And on Thursday I got a bit more information from “^GD” on the @firstdirecthelp twitter account. It still doesn’t answer all of my questions, but I think we’re a lot closer to the truth. Here’s what I was told.

The obvious question that this raises is why, then, do they limit the length of the passwords. I asked and got this (three-tweet) reply.

To which, I replied

And got the response

I thought that “as a business we are satisfied” rather missed the point. And told them so.

I got no response to that. And @brunns got no response when he tried to push them for more details about how the passwords are stored.

So, to summarise what we know.

  • First Direct say they store the passwords “encrypted”, but it’s unclear exactly what that means
  • It was a business decision to limit the length of the passwords, but we don’t know why that was considered a good idea
  • It still appears that First Direct believe that security by obscurity is an important part of their security policy

I haven ‘t really been reassured by this interaction with First Direct. I felt that the first customer support agent I talked to tried to fob me off with glib truisms, but “^GD” tried to actually get answers to my questions – although his obvious lack of knowledge in this area meant that I didn’t really get the detailed answers that I wanted.

I’m not sure that there’s anything to be achieved by pushing this any further.

First Direct Passwords

I’ve been a happy customer of First Direct since a month or so after they opened, almost twenty-five years ago.

One of the things I really liked about them was that they hadn’t followed other banks down the route of insisting that you carried a new code-generating dongle around so that you can log into their online banking. But, of course, it was only a matter of time before that changed.

A couple of weeks ago I got a message from them telling me that Secure Key was on its way. And yesterday when I logged on to my account I was prompted to choose the flavour of secure key that I wanted to use. To be fair to them they have chosen a particularly non-intrusive implementation. Each customer gets three options:

  1. The traditional small dongle to carry around with you
  2. An extension to their smartphone app
  3. No secure key at all

If you choose the final option then you only get restricted (basically read-only) access to your account through their web site. And if you choose one of the first two options, you can always log on without  the secure key and get the same restricted access.

I chose the smartphone option. I already use their Android app and I pretty much always have my phone with me.

Usually when you log on to First Direct’s online banking you’re asked for three random characters from your password. Under the new system, that changes. I now need to log on to my smartphone app and that will give me a code to input into the web site. But to get into the smartphone app, I don’t use the old three character login. No, I needed to set up a new Digital Secure Password – which I can use for all of my interactions in this brave new world.

And that’s where I think First Direct have slipped up a bit.

When they asked my for my new password, they told me that it needed to be between 6 and 10 characters long.

Those of you with any knowledge of computer security will understand why that worries me. For those who don’t, here’s a brief explanation.

Somewhere in First Direct’s systems is a database that stores details of their customers. There will be a table containing users which has a row of data for each person who logs in to the service. That row will contain information like the users name, login name, email address and (crucially) password. So when someone tries to log in the system find the right row of data (based on the login name) and compares the password in that row with the password that has been entered on the login screen. If the two match then the person is let into the system.

Whenever you have a database table, you have to worry about what would happen if someone managed to get hold of the contents of that table. Clearly it would be a disaster if someone got hold of this table of user data – as they would then have access to the usernames and passwords of all of the bank’s users.

So, to prevent this being a problem, most rational database administrators will encrypt any passwords stored in database tables. And they will encrypt them in such a way that it is impossible (ok, that’s overstating the case a bit – but certainly really really difficult) to decrypt the data to get the passwords back. They will probably use something called a “one-way hash” to do this (if you’re wondering how you check a password when it’s encrypted like this then I explain that here).

And these one-way hashes have an interesting property. No matter how long the input string is, the hashed value you get out at the other end is the same length. For example, if you’re using a hashing algorithm called MD5, every hash you get out will be thirty-two characters long.

Therefore, if you’re using a hashing algorithm to protect your users’ passwords, it doesn’t matter how long the password is. Because the hashed version will always be the same length. You should therefore encourage your users to make their passwords as long as they want. You shouldn’t be imposing artificial length restrictions on them.

And that’s why people who know about computer security will have all shared my concerns when I said that First Direct imposed a length restriction on these new passwords. The most common reason for a maximum length on a password is that the company is storing passwords as plain text in the database. With all the attendant problems that will cause if someone gets hold of the data.

I’m not saying for sure that First Direct are doing that. I’m just saying that it’s a possibility and one that is very worrying. If that’s not the case I’d like to know what other reason they have for limiting the password’s length like this.

I’ve send them a message asking for clarification. I’ll update this post with any response that I get.

Update (17 July): I got a reply from First Direct. This is what they said.

Thank you for your message dated 16-Jul-2014 regarding the security of your password for your Digital Secure Key.

Ensuring the security of our systems is, and will continue to be, our number one priority.

All the details that are sent to and from the system are encrypted using high encryption levels. As long as you keep your password secret, we can assure you that the system is secure. As you will appreciate, we cannot provide further details about the security measures used by Internet Banking, as we must protect the integrity of the system.

Our customers also have a responsibility to ensure that they protect their computers by following our common-sense recommendations.  Further information can be found by selecting ‘security’ from the bottom menu on our website,

Please let us know if you have any further questions, and we’ll be happy to discuss.

Which isn’t very helpful and doesn’t address my question. I’ve tried explaining it to them again.

National Rail Travel Alert

This is the text of a National Rail travel alert email that I received this morning.

Problems have been reported which may affect your journey between Balham (BAL) and Shepherd’s Bush (SPB)

More details of this disruption can be found here:

To see how this disruption affects your journey and to get alternative options planned for you, please use the Online Journey Planner

Alternatively, for up to date information for your station, use the Live Departure Boards.

Prefer to get in touch by phone? Call TrainTracker on 0871 200 49 50 (10p per min, mobiles higher) or text your journey details to 84950 to use TrainTracker Text

You can manage your alerts by visiting:

Don’t forget, you can also follow us on Twitter or Find us on Facebook for the latest rail travel news

Please do not reply to this email as it is sent from an unmonitored address. If you need to contact us, you can do so here:

Can you spot the obvious idiocy here?

It’s an HTML email. That’s obvious from the links that appear in it. Links to things like the Online Journey Planner and the Live Departure Boards. But there are a couple of links that are written as plain text URLs – ones that you can’t just click on. And one of them is the most important link in the email – the link to the full information about the problems.

In order to read whatever is on the other end of that link, you’d need to copy it and paste it into the location bar in your browser. That’s simple enough, of course, on a desktop computer. But surely one of the important use cases for these alerts is people standing on a platform trying to work out what’s going on with their train – in which case they’d almost certainly be using a smartphone. And copy and paste isn’t the easiest of things to do on a smartphone.

Someone in the National Rail Travel Alerts department is more than a little confused about how URLs in email work.

Free Web Advice: Marvel

It’s been a few years since I wrote a “free web advice” piece, but I got really annoyed by the Marvel web site this morning.

About a year ago I subscribed to Marvel Unlimited – a plan that gave me access to all of Marvel’s digital comics for about £40 a year. This morning, I got an email from them saying that my subscription was about to be renewed but that my credit card had expired so I should log on to my account and update my credit card details.

I went to log on and found that I had forgotten my password. So I used the “forgotten password” link expecting to get an email containing a link I could use to reset my password. Instead, I got an email that contained both my username and my password in plain text. If Marvel are able to send my password to me, then they must be storing everyone’s password in a readable format. It’s astonishing that a company the size of Marvel don’t understand just what an incredibly stupid idea that is. And sending both my username and password in the same email just compounds their error.

So that’s strike one – storing plain text passwords.

Having recovered my password, I was able to log on and found the page where I could give them my credit card details. But it looked like this:

Marvel Credit Card Maintenance Page

If you look closely, you’ll see that three fields – credit card type, expiration date and country – have captions, but no way to enter the required data. I’ve tried this page in both Firefox and Chrome and get the same results in both. I expect I’ll have to dig out a PC running Windows and try it on Internet Explorer as well.

I didn’t actually notice the missing fields at first. I just filled in the fields I could see and submitted the form. At that point I got an error pointing out what was missing. It’s interesting to note that the credit card type isn’t marked as required on the form (there’s no red asterisk next to it) but the error I got complained that it wasn’t filled it.

So that’s strikes two and three.
Strike two – always ensure that your web pages work on all the popular browsers.
Strike three – always mark your required data inputs accurately.

At that point I gave up trying to give money to Marvel. I poked around the site for a while to find a contact form. When I found it, it had the same problems as the credit card form – most of the input fields didn’t appear. Luckily, the contact page also gave an email address (that’s a really good idea that most web sites don’t follow). So I used that to report the problems. I’ll update this post if I get a response.

Interestingly, on my account page I was also given the option to upgrade my account. Apparently Marvel and I disagree on the meaning of the word “unlimited”. It’s not clear to me what extra benefits I could expect.

Update (four months later): Somehow, Marvel managed to renew my subscription, even though I never managed to update my credit card details. But bizarrely, this evening (over four months after writing to them) I got a reply from Marvel’s customer support. It said this:

Thank you for contacting Marvel’s Online Support services. We apologize for the delay in getting back to you. We see that you were able to renew your subscription, after contacting us. If you have any further questions, please do not hesitate to contact us. Thanks again for contacting Marvel.

Four months to reply to a simple customer support message must be some kind of record.