Pezholio

08 Apr, 2010

The postcode – freed!

Posted by: Pez In: Open Data

Well, it’s finally happened, Ordnance Survey have gone from being the bad guy of the internet, a big, bumbling behemoth, squishing innovation wherever it goes, to being the darling of the internet, throwing out free data wherever it goes (well, sort of).

As well as lots of mapping data being released, it also release Codepoint Open, which matches every postcode up a geographical location, it’s what I, along with the oh-so-clever uber trolling project Ernest Marples have been banging on about for ages. Now, finally, developers can get a user’s location details based on one thing that almost everyone in the UK is guaranteed to know, their postcode – this can then be used to get all sorts of useful information, like their nearest school, fire station, police station, cafe, tanning salon (or whatever).

As well as matching postcodes to a physical location, CodePoint Open also matches postcodes to local authority information, something which, in the past, I’ve used the NESS Data Exchange for, but I’ll probably migrate to this, ‘cos it’s easier.

Anyway, although the data has been released and is out there, it’s in a whopping 250mb CSV file, so if anyone wants to use it, they need to download a local copy and import it into a database for their own use. I wanted to do this, but thought, instead of everyone doing this, why don’t I make it easier for everyone (myself included) and wrap it in a web service?

That is wot I dun – enter UK Postcodes. Initially I built this over the Easter break, in a few snatched moments between eating easter eggs and making sure my new puppy didn’t wee anywhere. On the surface it’s very simple, a request like this:

http://www.uk-postcodes.com/postcode/WS136YY.xml

will give you all the information about WS13 6YY in XML format, including latitude/longitude, easting/northing, a Geo Hash URI, as well as the county council (if applicable), district council and ward the postcode is in. You can also have JSON, CSV and RDF just by changing the extension (Thanks to Jeni Tennison for her help with the latter format!).

After posting the results of my fiddling on the UK government Data Developers mailing list – I got a lot of helpful people suggesting other ways I could interact with the data, so I know have methods for finding the nearest postcodes to a point, as well as reverse geocoding (going from a lat/lng point to a postcode) – there’s more information on the API page.

That’s basically it, I’m always happy to hear feedback, so if you’ve got any suggestions or feedback (good or bad!), please feel free to comment below!

(Oh, and a massive thanks goes out to Adrian Short (actually it was Adrian who created the torrent of the data – proving that file sharing isn’t always used for nefarious purposes :cough:#debill:cough:) Matthew Somerville, who translated all those Eastings and Northings in the original dataset to Latitude and Longitude – you can download his version from the MySociety mirror.)

28 Responses to "The postcode – freed!"

1 | Matthew

April 8th, 2010 at 10:56 am

Avatar

You called it UK-postcodes, but sadly it’s Great Britain, not the whole UK. And Adrian created the torrent, but I translated the E/N to lat/lon :)

Are you going to release the source? I’d be interested. How are you doing the nearest postcodes lookup – I’d have thought that’d be easier with easting/northings than lat/lon ;-)

My demo gazetteer using the three gazetteer-like products including CodePoint Open can be found at http://github.com/dracos/opendata-gazetteer

2 | Pez

April 8th, 2010 at 11:05 am

Avatar

Cool, sorry, assumed it was Adrian – will amend the blog post :)

I’ll put the source on Github in a mo – might clean the code up a bit first – agile development has its downsides ;)

For the nearest postcodes lookup, I used the Haversine formula, which works in mySQL like this:


SELECT id, ( 3959 * acos( cos( radians({lat}) ) * cos( radians( lat ) ) * cos( radians( lng ) - radians({lng}) ) + sin( radians({lat}) ) * sin( radians( lat ) ) ) ) AS distance
FROM {table} HAVING distance < 25 ORDER BY distance LIMIT 0 , 20;

3 | Andy Mabbett

April 8th, 2010 at 12:08 pm

Avatar

Neat. Here’s a sample HTML result, with added hCard microformat (including Geo and ADR microformats):

http://pastebin.com/H4EjPKts

All I’ve added are a few classes, and one DIV (the one with class=”adr”). It would be great if you can implement this, please.

Oh, and you really should let the puppy wee /somewhere/!

4 | Andy Mabbett

April 8th, 2010 at 12:17 pm

Avatar

Oh, and adding a link to http://openlylocal.com/areas/postcodes/ might be a good idea.

6 | Pez

April 8th, 2010 at 12:58 pm

Avatar

Both excellent ideas Mr Mabbett, consider them implemented! :)

7 | Andy Mabbett

April 8th, 2010 at 2:07 pm

Avatar

That was quick! Thank you. (Though you’re missing the class “geo” around the coordinates).

8 | Andy Mabbett

April 8th, 2010 at 2:11 pm

Avatar

And class “locality”.

Also, I’d add rel=”tag” to the OpenlyLocal link.

9 | Andy Mabbett

April 8th, 2010 at 3:40 pm

Avatar

All working now; thanks.

11 | Tom Morris

April 9th, 2010 at 9:03 pm

Avatar

Any chance you could do content negotiation? I had a look at the code, but it has been a long time since I did PHP, MySQL or Apache htaccess (I’m in Rails-land now).

Basically, what would be really nice is to have it so that as well having [postcode].json, you could send an Accept: header specifying the format, and it’ll return it.

Alternatively, any chance of putting a link element in the header pointing to the RDF version with the rel value set to ‘alternate’ and the type value set to ‘application/rdf+xml’.

Thanks.

12 | Pez

April 10th, 2010 at 12:24 am

Avatar

Good call – I’ve merged your changes into my original code on Github (which is no mean feat after the best part of a bottle of wine!) And yeah, Rails would be much easier, just been too lazy to learn it!

13 | Max Manders

April 10th, 2010 at 5:05 pm

Avatar

You beat me to it – well done :-) Nice implementation. It’s good to see some open data appearing for the UK!

14 | Anon

April 12th, 2010 at 3:24 pm

Avatar

I’d like to have the functionality to download the data for a ward/county/lsoa etc.

Great tool, thank you!

15 | Christoph Burgdorfer

April 14th, 2010 at 11:40 am

Avatar

That’s great! Thanks so much! It doesn’t contain Guernsey, Jersey etc. Post Codes though, right?

16 | Pez

April 14th, 2010 at 11:45 am

Avatar

That’s right sadly, but I don’t think they’re actually part of the United Kingdom per se, so (in this case anyway) it’s still correct to call it UK Postcodes. The lack of NI postcodes is another matter entirely though! :)

17 | Mark

April 14th, 2010 at 3:18 pm

Avatar

I’d also suggest switching gzip compressions on at least for distance.php. A 150Kb .kml file can be reduced to approx 9Kb, at the slight expense of server resources vs. bandwidth.

Mark

18 | Chris

April 26th, 2010 at 10:05 pm

Avatar

I’ve played around with Northern Ireland data from npemap.org.uk (mostly derived from the Postbox Locator), but it’s very patchy and of doubtful accuracy.

Then I came across http://www.nisra.gov.uk/geography/default.asp6.htm. It’s relatively simple to use the .dbf file in the ESRI data to get locations, in Irish grid format, and not too hard to convert those. But I can’t figure out what the licence is for these data.

19 | Pez

April 28th, 2010 at 11:14 am

Avatar

Interesting. Would like to add those, but, like you say, not sure of license restrictions. I’ve had some VERY angry people on data.gov.uk having a go because I called it UK Postcodes and it doesn’t contain NI data.

20 | Note to self » Blog Archive » Postcodes freed

April 29th, 2010 at 10:20 pm

Avatar

[...] I came across UK Postcodes, a system based on Code Point Open put together by Pezholio. UK Postcodes provides a simple API for fetching a range of data about a postcode in XML, JSON, CSV [...]

21 | Chris

May 2nd, 2010 at 10:20 pm

Avatar

I’m wondering if there is a reason for a slight inconsistency in the way data are returned? Looking at the XML, for most postcodes the postcode is echoed back with no space between outward and inward parts. But some, eg HR8 1AA, are echoed back with the space.

22 | Pez

May 3rd, 2010 at 11:13 am

Avatar

No reason, this is just how the raw data came originally from OS. All of the postcodes in the database have 7 characters, so if the are only 6 characters in the postcode, a space is inserted, strange that they chose to do it this way. No reason why I can’t change it though!

23 | Matt Wallis

May 13th, 2010 at 11:39 am

Avatar

Thanks for making this available as a web service. One picky point: When I first saw your results, I was surprised at the number of decimal points being presented for eastings and northings – usually 8, placing the postcode at an accuracy less than the width of a human hair! But when i compare your results with those in the original Code Point data, they are often not the same when rounded to the nearest metre.
e.g. at WS13 6YY, you give the easting as 411752.78123551, whereas the original CodePoint data has: 411754.
Why are you not presenting the eastings and northings (a) as integers (giving an accuracy of one metre), and (b) as the same numbers in the Code Point data?

24 | Suraj

July 6th, 2010 at 3:11 am

Avatar

How can the calculation for finding nearest postcodes within distance be modified to use the eastern/northings co-ords rather than lat long? Because the ordnance survey CSV file only contains eastern/northing co-ords :/

25 | bouton

July 13th, 2010 at 10:23 am

Avatar

Was looking for info on translating OS data eastings/northings to lat/lng and found your site. Downloaded your github src to see what it looks like and the postcode.sql only goes to GL116DH. I’ve tried a couple of times – is that correct?
anyway – thanks for the API. Looks interesting.

26 | Pez

July 13th, 2010 at 10:32 am

Avatar

I think Github truncated the upload, so not everything got uploaded – the full database dump can be found here:

http://www.uk-postcodes.com/postcodes.sql.gz

Alternatively, you can download the CSV versions in WGS84 format from here:

http://parlvid.mysociety.org:81/os/

Hope this helps!

27 | bouton

July 29th, 2010 at 9:38 am

Avatar

Thanks. the parlvid url always seems to be down when I try. Appreciate your work.

28 | Richard

August 12th, 2010 at 6:37 pm

Avatar

Hi There! This is a fantastic tool! One question about it. How did you calculate the lon/lat before you imported it into the MySQL database? I noticed the Code Point data only come with Northerly and Easterly headings, not lon/lat, so I assume you must have performed some form of calculation when you imported the csv into the database?

Thanks again.

Comment Form