July 2008 Archives

When will web sites start to be careful with people's passwords? Oh, I know that a few sites get it right, but it seems to me that the vast majority still don't have a clue what they are doing. Here is today's example.

I got an email this morning from a company called RAM (that's Research and Analysis of Media). Somehow they knew that I was an (occasional) Observer reader and they were inviting me to join a panel that would (as I understand it) answer occasional surveys about the Observer. It sounded like a good cause, so I signed up. As part of that process I gave them both a username and a password. They immediately confirmed my sign-up by sending them both back to me in an email.

That is, of course, a serious cause for concern, but there's a slim chance that they aren't storing my password in an accessible form in their database. The mail might have been generated from the data in the web form I filled in. However, an hour or so later I got another mail from them telling my how to log into me account and including my username and password. In fact, that one email contained all of the information needed to log into my account (web site address, username and password). So they have established themselves as a company who can't be trusted with your password.

On the off-chance that they wouldn't be sending me any more mails containing those details, I thought I'd try to return at least a small amount of security to my account by changing my password. Except that there is apparently no way to change your password from within your account. By this stage they are breaking records for password stupidity.

I've contacted them about the problems and send them a link to my basic guide to password handling article. I'll let you know if I get any response. I hope their surveys are constructed with a little more thought then their web site.

Update: I heard back from them about not being able to change my password. You can do that in an "update profile" screen. Not sure why I didn't spot that last week. Nothing from them about the password storage issues though.

When I first read about Drizzle last week I assumed it was some kind of joke. It turns out that it isn't, so it just makes me a little depressed.

Drizzle is[1] a cut-down version of MySQL. MySQL is the database server than has been known to make grown database designers cry because of its rather cavalier approach to the relational database model. For much of its existence, MySQL didn't have many of the features that make databases useful - features like triggers, stored procedures and (specially) referential integrity. It was very fast, but it really couldn't be described as a relational database. It was just a data store with a vaguely SQL-like query languge. Over the last few years, MySQL has added more and more of these features and recent versions are very nearly like a real database[2].

Of course, adding these more advanced features is going to effect the performance of the database. If you're checking referential integrity on each insertion, then things are bound to get slower. Or are they? I'm not sure that's the case. If you don't use the referential integrity checking in the database then you have to duplicate that functionality in your application code. But checking referential integrity is a core feature of a database system. That code is (hopefully) going to be well written. In most cases it's likely to be better than your code. So trusting the database to check referential integrity is likely to make your application seem slower until you turn off your own hand-crafted referential integrity code. Trust your database to do the work. It's what it's best at.

But I think the damage has been done. There is a generation of database designers (and I use the term in a very loose sense) who don't know about "advanced" database features like referential integrity, triggers and views. They don't know about them because they learned database design whilst using an older version of MySQL which didn't have those features. Or they learned from someone who learned whilst using an older version of MySQL. We have systems like Activerecord which encourage the developer to ignore the advanced features of the database and to use it as a dumb data store.

So this generation of database designers don't see the need for these features and therefore don't use them. They see them as an unnecessary waste of resources which just slow down their application. In my opinion, if they learned about these features then they wouldn't think like that, but I'm not writing this entry just to plug my Advanced Databases for Beginners course.

I don't deny for a second that many applications (mostly web sites as far as I can see) run really successfully without using these features. But as a database designer it makes me nervous. It's like programming without a safety net. Why take unnecessary risks with your business data?

So this is the perceived need that Drizzle addresses. A need for a dumb data store that doesn't do any of the advanced things that a database should do. It's like we're turning back the clock to MySQL 3.x. And that's why Drizzle depresses me. It looks to me like a dumbed down tool for people who haven't taken the time to learn their craft.  I spent far too much time working with MySQL 3.x. It always ended badly?

Am I missing something here? Perhaps I'm a "database dinosaur" who just doesn't understand the new trends. Please point me at at articles that could enlighten me.

But until you do, my databases will all include referential intergrity, triggers, views and stored procedures.

[1] Or, more accurately, will be. It's a work in progress.
[2] Athough you still have to entertain doubts about a database system which documents an option (and one that is off by default) which makes it 'behave like a “traditional” SQL database system'.

For most of the last year, I've been working behind a corporate firewall which blocks most social networking sites. It's therefore only in the last month or so that I've been able to use Twitter all day every day.

It seems to me that many of Twitter's users have slightly distorted the sites original purpose. It was originally intended to be used for posting brief "I'm doing this" messages, But many people seem to be using it to hold conversations with their friends. It's become a sort of "non-instant messaging". Interestingly, the site's developers noticed this change and added features (like replies) which made it easier to use the site in this way.

But there are still places where the site's origins are obvious. On anyone's profile page you can see two numbers listing the number of people that person is following and the number of people who follow that person. But actually the Twitterverse doesn't break down into two sets like that. There is a more interesting set of three numbers. For most people their sets of followers and followees aren't disjoint sets. There is another set of people who both follow you and are being followed by you. Let's call them your peers.

So we have three sets of people. The people who you follow but who don't follow you in return (people you think are interesting but who don't think you are interesting enough to follow), your peers and the people who follow you but who you don't follow in return (people who think you are interesting but who you don't think are interesting enough to follow). There's probably a whole cyber-sociology paper in analysing the ratios between the sizes of those three groups for different types of people.

But the important thing is that you can only carry on a conversation with people in your peer group. It remind me of the old Frost Report sketch about class differences. The people higher than you in the food chain don't listen to what you say. A few times I've missed things that people said to me because I'm not following them and simply adding "@davorg" to your message doesn't add it to my home page (think of the spam potential if it did).

I get round this by using Twitter Search (previously Summize) to search for messages to me. Actually I go a step further than that and have a feed from that query in Bloglines. Is that a common solution to the problem? What do other people do? Is it a problem that you've noticed?

Another, related, issue is how do you move up the hierarchy? Is there an etiquette for contacting people who you follow but who don't follow you? Can you just send them a direct message saying "hey I'm interesting, follow me"? And is anyone being inclusive and automatically following anyone who follows them?

Oh and what does Twitter have that Pownce, Jaiku or identi.ca don't have? Is it just the number of users? Will we ever see a big move from Twitter to identi.ca like the MySpace to Facebook move of last year?

Update: hanakomu points out (on twitter of course) that if someone replies to you then the message appears in your 'replies' tag whether or not you're following to them. Also, people get a mail when you follow them - but I think that's probably optional.

A while ago, I set up Planet Westminster - a pretty simple site that simply aggregates all of the MPs' blogs that I could find. It was largely created to scratch a personal itch. I wanted a simple way to subscribe to all MPs' blogs in my feed reader. And that's really how I use it most of the time. I just read it in Bloglines rarely bother to look at the site (which explains why I haven't fixed the character-encoding problems that are obvious to anyone visiting the site).

But I had a look at it today. And I tweaked a couple of presentation problems. As part of the process, I ran the software which aggregates the feeds by hand a couple of times. And that showed me one interesting issue that I had previously missed. The program displays an error when it can't find the feed that it's looking for. It's currently generating eleven "missing feed" errors. That's out of thirty-six feeds that I currently monitor. Perhaps a couple of those could be put down to temporary network glitches, but that's potentially over a quarter of the (small number of) blogging MPs who have either given up on blogging or have moved their feeds without putting redirection in place (that's starting to become quite a regular topic round these parts).

At one point it looked like MPs might start blogging in reasonable numbers. We'd broken the 5% barrier. It would be a shame if they decided if it was a waste of their time and started to abandon it.

The errors I'm getting are as follows (with links to the missing web feeds) . If any of these are your MP, then perhaps you'd investigate what's going on and report back. One of them is my MP, Martin Linton, so I'll start by investigating him.

• Stephen Crabb
• Jon Cruddas
• Mark Lazarowicz
• Martin Linton
• Kerry McCarthy
• Maria Miller
• Austin Mitchell
• David Miliband
• Bob Spink
• Shaun Woodward
• Derek Wyatt

Update: Having looked into it a bit further, I see that many of the problems are down to people moving their web feeds without putting redirection in place. Obviously I don't blame the MPs for this, but it indicates how little their "tech support" people know about how this stuff works.

A few of the blogs have closed down though. And it's interesting to note that in a couple of places a blog feed has been replaced by a news feed.

I need to put aside some time to do some more research into this in order to ensure that the date I have is up to date. And this is exactly the kind of information that PoliticalWeb is supposed to provide.

My family have lived in the same part of Essex for over two hundred years. The earliest record I can find is a Thomas Cross who was born in Little Clacton in 1789. Thomas was my great, great, great, great grandfather. The family moved to Great Clacton soon afterwards and then (once it was created in 1871) to Clacton-on-Sea.

Coming forward a couple of  generations from Thomas, we find my great, great grandfather James Cross. He was born in Great Clacton in 1844. By all accounts, James loved boats and loved the sea. When the Clacton-on-Sea life boat service was set up in 1878 he signed up very quickly and soon became the second coxswain on the boat.

Over the following years James was involved in many sea rescues. One of the most famous was in 1881 when the french lugger Madeline was wrecked on Gunfleet Sands. The Clacton life boat saved the lives of the sixteen crewmembers. This was big news at the time and the French Government presented James and the coxswain with gold medals. The tankard in the picture dates from the same year and we've always assumed that it is associated with the same event. The tankard has been in our branch of the family for as long as my father can remember. We assume that the medal is in some other branch of the family. There's a photo of it on this page. I should try to track down that part of the family. The engraving on the tankard says:

I've come across the "password antipattern" twice today. And I had different reactions to it each time. I thought it was worth trying to work out why that was.

Let's start by explaining what I mean by the "password antipattern". There are many bad ways to handle users' passwords which I've discussed at length before, but there's one that's become depressingly common in recent months. And that's the one which encourages users to give their password for some site to some completely different site in order to facilitate interaction between the two sites.

The most common example of this is when you're trying to work out which of your friends are already on a new social networking site and the site offers to find them for you if you give it (for example) your Gmail username and password. At a time when phishing attacks are on the increase, it's terrible that popular sites are encouraging users to treat their passwords in such a cavalier fashion.

There's a simple rule to follow with passwords:

Never share your password with anyone else

It's a simple as that. Your password is used to identify you to the system you are logging on to. It's your secret. If you share it with anyone else then you can no longer be sure that you are the only person who can log into that system as you.

And it's generally completely unnecessary. There are better ways for you to authorise one site to access your details from another. Flickr gets it right. As does Fire Eagle. Asking the user for a password is laziness.

So what were the two cases I came across today?

The first was Moo's offer of 50 free business cards for LinkedIn users. As part of this process, obviously Moo have to satisfy themselves that you are a LinkedIn user. And they did that by asking for your LinkedIn username and password so they could try to log in as you.

In this case, I confess that I relaxed my rules and gave them the information. It's amazing what I'll do for a few free business cards. But I thought carefully about it before I did it. In the end, two factors persuaded me that it was a risk worth taking. Firstly, I knew that this was a one-off occurance. Moo just needed to verify my membership of LinkedIn once. There was no reason at all for them to store my password. Secondly, I trust the people at Moo. I've met a couple of them and many of them are good friends of my friends. The first reason persuaded me that there was no reason for them to store my password and the second persuaded me that they weren't the kind of people who would store my password anyway.

So, yes, I broke my own rules. But it was a considered decision. And if I wanted to I could even go and change my LinkedIn password now. That won't effect my Moo order at all.

The other case was rather different. I had seen a few people in my Twitter stream using twitterfeed to post automated messages to Twitter and I decided to investigate further. The idea is pretty simple. Twitterfeed subscribes to an RSS feed (perhaps of your blog posts) and each time a new entry is found in this feed, it posts a new message to Twitter using your username.

It's that "your username" that scares me. Twitterfeed does this by logging in as you and posting a message. Twitter has an API, but currently the only way to authenticate as a user is to log in using their username and password. Twitter accept that this is a shortcoming and say in their documentation that they are working on an improved method.

But currently, the only way for twitterfeed to post a message to your Twitter stream is by knowing your username and password. So in order to use twitterfeed to need to give them your details and they need to store them in their database. And they need your password in plain text in order to log in as you - so they can't store it using a one-way encryption of any kind. So a third party company has your Twitter login details in their database. It's not a one-off thing as happened with Moo, they need to store your details for as long as you're using their system,.

So I stopped my investigation of using twitterfeed. I'm certainly not sharing my login details with a random third party. And I strongly recommend that you don't either. I'm currently investigating using my own tool using the Perl module Net::Twitter. But solutions like that aren't going to be useful to everyone. Web-based services like twitterfeed are going to appeal to a lot of people.

How do we educate users to be more careful with their passwords? Or don't we care? It is, after all, only a microblogging system. Does it matter if someone else gets hold of your details and starts sending messages as you?

I just saw that Russell T Davies is speaking at the National Theatre. I'm a big fan of his work, so I decided to buy tickets. I wandered over to the NT web site and found the event I was looking for. I added a couple of tickets to my shopping basket and went to check out.

The site told me that I have twenty-five minutes to complete my purchase or else my tickets will be made available for anyone else to reserve. It also asked me to log in to complete the transaction.

I start to set up a new user account in order to log in. The system tells me that my email address is being used by an existing account. I must have set one up the last time I bought tickets from them (which was several years ago).

I try a few likely username and password combinations. None of them work. My twenty-five minutes is ticking away. I click on the "forgotten password" link. The site promises to send me my login details.

I wait. And wait. And wait.

I check my "potential spam" folder. I check my "definitely spam" folder. No sign of the promised email. My twenty-five minutes is almost up.

With a couple of minutes to go I have a brilliant idea and register with another email address. Somehow I manage to go through the procedure within the time remaining and successfully purchase the tickets.

Instantly after registering as a new user, I get a "thank you for your registration" mail. The promised password details email is still missing somewhere. There's apparently an email receipt en route to me too. I don't hold out too much hope.

Why do web sites make this so difficult?

Oops. Busted.

Earlier this year, I wrote a mild rant about web sites who change their RSS feeds without redirecting them and thereby losing a number of readers.

Last night mou commented on that entry pointing out that I'd done something very much like that myself. For the last two months, I haven't been publishing a new index.rdf feed.

I strongly suspect that the date of the last new version of that file coincides with the date that I installed a new version of Movable Type and reset all of the templates to the defaults. By default, current versions of MT don't seem to publish RSS feeds. They just publish an Atom version (atom.xml).

That's no excuse though. I knew about that problem. Previously I'd worked around it by installing an RSS template from an older version of MT. I might do that again when I have some spare time to think about it. But in the meantime I've taken the easiest option and created a symbolic link from atom.xml to index.rdf. Hopefully that'll work in the short term.

Apologies to anyone who was subscribed to the RSS feed and who, no doubt, thinks that I've dropped off the face of the world. I'm sorry that you'll suddenly have two months worth of my nonsense to plough through this morning.

It might be a good time to mention the other feeds that I set up recently.  There's one contains all of my long-form writing from this and other blogs, one that has shorter items from various microblogging platforms and then there's the feed from planet davorg which contains everything.

Ooh, ooh. ooh. Much excitement. Dr Horrible is coming. This is going to be fabulous.

Update: Well, it might be fabulous. I can't be sure. It seems that you can only view it from within the USA. Bugger.

Obscure bits of religious  dogma are causing a bit of a ridiculous argument over in the USA. It seems that crackers need to be treated with the right level of respect if you don't want the might of the Catholic League coming after you.

It's not just any old cracker, of course. Oh no. It needs to be a cracker that has gone through the mystical transformation process that turns an ordinary cracker into the body of the messiah.

I don't know if you realise this, but all over the world catholics believe that during the communion service, the crackers and wine literally turn into the body and blood of Jesus. Of course, it's still really well disguised as crackers and wine, but that doesn't matter to the catholics. What matters is what they believe. Which is that as part of of the communion they are literally consuming the body and blood of Jesus. I suppose that explains why the ten commandments contain no injunction against cannibalism.

Anyway, this story begins when a Florida student called Webster Cook decided that instead of eating the cracker he was given he would instead walk out of church with it. This decision didn't go down well with local catholics. A representative of the local diocese described it as a hate crime. Fearing repercussions, Cook returned the cracker.

The story was picked up on Tuesday by PZ Myers, the Minnesota biology professor who wirtes the Pharyngula blog. Myers, quite rightly, had a little bit of a laugh at the expense of the catholic church before making a more serious point:

I find this all utterly unbelievable. It's like Dark Age superstition and malice, all thriving with the endorsement of secular institutions here in 21st century America. It is a culture of deluded lunatics calling the shots and making human beings dance to their mythical bunkum.

He then takes it a step further:

Can anyone out there score me some consecrated communion wafers? There's no way I can personally get them — my local churches have stakes prepared for me, I'm sure — but if any of you would be willing to do what it takes to get me some, or even one, and mail it to me, I'll show you sacrilege, gladly, and with much fanfare.

And that has done nothing at all to calm the situation down. In a follow-up post, Myers catalogues the hate mail he has received since posting his previous entry.

So far today, I have received 39 pieces of personal hate mail of varying degrees of literacy, all because I was rude to a cracker. Four of them have included death threats, a personal one day record. Thirty-four of them have demanded that I be fired.

He also has the Catholic League starting a witch-hunt against him.

The Catholic League are, of course, just showing themselves up as ridiculous fantasists. It's the twenty-first century. No-one is going to believe in transubstantiation unless it has been drummed into them from an obscenely early age. It's a nonsense. The communion wafer remains a communion wafer. The wine remains wine. You can believe whatever you like about what it represents. But it doesn't actually change.

The more that religious organisations like the Cathloic League over-react to situations like this, the more they will alienate themselves from the general public. This has to be a win for rationalism. This story needs to be seen by as many people as possible, so that as many people as possible have the chance to look at it and say, "What are they talking about? It's only a bloody cracker!"

I spent yesterday at Opentech. I had a great time there. Here are my thoughts on the talks that I saw.

Rembrandt, Pr0n and Robot Monkeys: Lessons From the Present About Flesh and Technology - Kim Plowright
This could have been interesting, but I think it was somewhat constrained by the short time allocated. It seemed to be a rather disjointed amble through a bit of history and a look at people see their physical bodies in cyberspace.

Living with Chaos: Why Nothing is Simple in IT - Simon Wardley
If you've been following Simon's blog, then you'll be familiar with his view of the commoditisation of software. Somehow, I've missed seeing him giving his talks on the subject over  the last couple of years, so it was nice to finally see one. Again, this would have benefited from a longer timeslot - but I think that'll be a common complaint as I go through the day.

What the Frog's Eye Tells the Future - Matt Webb
Exploring the early history of the science of cybernetics and pointing out some surprising coincidences and some interesting comparisons with today. Matt is always interesting and I'd love to read more about this subject.

Here's The UK EFF - Becky Hogge and Danny O'Brien
The Open Rights Group was formed out of a talk that took place at the last Opentech conference in 2005, when a pledge was set up for people to agree to pay £5 a month to support such an organisation. In this talk Becky Hogge and Danny O'Brien (who I didn't recognise in his full beard) talked about what had happened in the last three years.

Except they didn't really. Mainly they just asked for money. Apparenlly, of the 1,000 people who signed the original pledge, only about 750 kept their promise are making regular payments. So if you signed the pledge and haven't set up your standing order then why not do so now? Or, if you didn't sign the pledge but think that the UK needs a strong organisation campaigning for digital rights, then why not sign up? Or, if you are already making regular payments to them, why not increase the monthly amount? I just did.

Power to the people - one year on from the Power of Information Report
If you've see the Show Us A Better Way site, then you'll know that there's a growing movement within the UK government to free up public data and make it available in easy to use formats. In this session, various people behind this initiative spoke about how they've got to the current situation and where they hope to go next. It's great to see this amount of data coming from the civil service and it seems that the best way to encourage them is to use the dat ato create really cool things. You can find out more about the Power of Information team, by reading their blog.

3 Years of OpenStreetMap -Nick Black
It's been a while since I last looked at OpenStreetMap. And it looks like they've come a long way in a relatively short time. Many of their maps now look really impressive. I shall be watching them far more closely in the future. I may even edit the occasional map.

Opening Data - Rufus Pollack
The Open Knowledge Foundation exist to promote the sharing of knowledge and data. They have a repository called CKAN (modelled on the Perl repository CPAN) where you can share any useful data that you have. Looks very interesting.

Planning Alerts - Duncan Parkes
I already knew about the planning alerts project. It's one of the those ideas that seems simple and obvious - but no-one had thought of it until recently. You go to their site, give them your post code and email address and they send you regular messages about planning applications in your area. I signed up a few months ago, but I only get alerts from Lambeth (about 300 metres to the east of my house) as they don't yet have a parser to extract data from the Wandsworth Council site. They asked for help with missing councils. I should probaby do that.

Publishing with Microformats - Jeremy Keith
Microformats is one of those areas that I've read about and really want to start using. But I haven't really found a use for them yet. This talk helped a bit as it concentrated on using a couple of microformats (hCard and XFN) to mark up social relationships. I really need to investigate this further.

Information: Rewiring the London Gazette with RDFa - Jeni Tennison
Moving on from microformats, RDFa is a tremendously powerful way to add value to HTML pages. I'm not going to be using this any time soon, but it's interesting to know it's possible. And the data set (when it is released) is going to be incredible. Lodon Gazette (Jeni had a URL in her presentation, but I can't remember it now) is the government's official newspaper - it contains all of their announcements.

The Bastard Child of Baird and Berners Lee - Tom Loosemore
Tom gave an idea of some of the things he was thinking about just before he left the BBC a year ago. He's basically talking about creating a network of recording boxes that will record all TV ever broadcast in the UK. Sounds cool - if slightly hamstrung by copyright rules.

Finding Good TV on the Interwebs with RDF and REST - Chris Jackson
Chris introduced URIplay - a project to catalogue and simplify the metadata that is broadcast alongside TV and radio. The idea is to make it easier to track down programmes that you want to watch.

Intro to Hadoop - Tom White
I only went to this by mistake. I turned up early for the Guardian talk. I knew nothing about Hadoop before the talk and I know almost nothing more now.

Guardian.co.uk: building for the open web - Stephen Dunn and Mat Wall
Stephen and Mat talked about some of the design decisions that went into the recent (and ongoing) rebuild of the Guardian web site. It's great to see a national media site designed by people who really understand how the web works and who are making an effort to exist within that ecosystem. There were also some interesting hints about the forthcoming Guardian Developer Network


So that's what I saw. I think I pretty much made the right choices, but with three tracks it's impossible to see everything you want to see. I heard people saying interesting things about the talk on tracking arms dealers using Python. I also with I could have seen the sessions on MySociety and OpenID. Hopefully there will be slides and video available online soon. I also felt that the "hallway track" was better than ever. Everywhere I went I found myself having interesting conversations with people.

I dashed home at the end in order to watch Doctor Who as soon a possible. I shouldn't have bothered. What a waste of time that was.