Much of this data is still in very ‘flat’ formats (such as Excel docs and the like), but some data is being released in RDFa format, the preferred format of the semantic web.

The most interesting of these datasets has been the education one, giving me (and other developers) the ability to get information about schools in an open format. I’ve been playing with this data for a bit, and now (I think) I might have something useful.

The key way of interacting with it is using a language called SPARQL, which is very similar to SQL, but with the web of data in mind. An example query for the education dataset is as follows:


prefix sch-ont: <http://education.data.gov.uk/ontology/school#>
SELECT ?name WHERE {
?school a sch-ont:School;
sch-ont:establishmentName ?name;
sch-ont:districtAdministrative
<http://education.data.gov.uk/placeholder-id/administrativeDistrict/Lichfield> ;
}
ORDER BY ?name


This gets the names of all the educational establishments (schools, nurseries etc) in Lichfield District. You can give it a try yourself on the public facing SPARQL endpoint.

What I wanted to do though, was to get all the primary or secondary schools within a 2 mile radius of a postcode. This was a little harder, but perfectly doable.

As well as general details (such as address, name of school etc) and more detailed information (such as capacity, inspection details etc), the dataset also has the easting and northing of the school. Therefore, to get all Primary Schools within a 2 mile radius of a point with an easting of 411021 and a northing of 307291, I needed to do the following query:


prefix sch-ont: <http://education.data.gov.uk/ontology/school#>
SELECT ?name ?address1 ?address2 ?postcode ?town ?easting ?northing ?reference WHERE {
?school a sch-ont:School;
sch-ont:establishmentName ?name;
sch-ont:easting ?easting ;
sch-ont:northing ?northing ;
sch-ont:uniqueReferenceNumber ?reference ;
sch-ont:phaseOfEducation <http://education.data.gov.uk/ontology/school#PhaseOfEducation_Primary> .
FILTER (?easting < 414239 && ?easting > 411021 && ?northing < 310509
&& ?northing > 307291)
}


This gets the name, easting, northing and reference number of every school within a 2 mile (3201-ish metre) box, with the original easting and northing as the midpoint.

I’ve given it a test here (a bit of naughtiness is required using the Google Maps API to get the latitude and longitude of the postcode, but it’s the only way at the moment!), so please feel free to try it out.

Once you submit the form you’ll get a list of all the primary and secondary schools in that area, as well as a link to more info on the Edubase site.

I’d be interested to know your thoughts, so please feel free to comment!

There’s also a few more example queries on the Talis Blog, so if you’re that way inclined I heartily recommend having a play.

The evening was a laid back affair, with noms and drinks laid on by The Guardian (who are pretty much leading the way in open data for newspapers) and consisted of talks by Dave Harte from Digital Birmingham, Michael Brunton-Spal from the Guardian, Matthew Somerville (sometime of MySociety and sometimes just his own darn self) and Paul Bradshaw, Senior Lecturer in Online Journalism at Birmingham City University.

The focus of the evening was really for the presenters to showcase what they were doing with open data, whether that’s sourcing and generating it themselves, or using data from other sources to do cool stuff.

The first speaker was Dave Harte from Digital Birmingham, which is a partnership initiative, set up by Birmingham City Council to encourage people in Birmingham to make use of digital technologies. The main focus of Dave’s presentation was to discuss the work Digital Birmingham have been doing to get more council data into the public domain.

It’s been a softly softly approach so far, very different from my JFDI approach at Lichfield, basically consisting of a two pronged approach – finding out what people’s concerns are (partly though the social media surgeries being run across the city), and then getting the data based on that, as well as introducing people who don’t do the ‘ooh shiny!’ routine to the concept of open data, partly through bevocal.org.uk.

He also talked about the barriers to open data, such as ‘data hugging’ (which Tim Berners Lee talks about in his TED talk – a must watch if you haven’t seen it already), as well as licensing issues (such as Ordnance Survey) which prevent data being used freely.

Dave’s presentation can be seen in full here

Next up was Michael Brunton-Spal (@mibgames) from the lovely sponsors of the day – The Guardian, who have a fantastic store of open data, sourced by journalists and then shared with the general public to mash up and use as they wish.

As well as the Data Store, Michael talked about the work they did with MPs expenses earlier in the year. The Daily Telegraph had the full, unredacted expenses data before the Guardian, and had leigions of interns rifling through it looking for good stories, so the Guardian had to find some way to find stories first.

The solution here was crowdsourcing, they published the data online and got interested people to sift through it for them. So far 205,924 pages of documents have been reviewed, and they’re only half way through. Interest has dwindled a little since the story first broke, but people are still going through it, looking for interesting titbits.

Michael also went into a bit more detail about the data store, which I mentioned earlier, showing off some of the great stuff that people have done with Guardian data (there’s a gallery on Flickr here) – my favourite one is the mashup of media scare stories plotted against the actual deaths the scare caused. There’s also lots of cool datasets and examples of what people have done on the Data blog.

Another good mashup was the MP’s travel expenses map, which plotted the amount of money claimed by an MP to their constituency, Mps which lived closer to Westminster, but claimed more can be seen very clearly and their claims questioned in more detail.

The coolest thing though in my opinion was the Guardian Open Platform, which in in closed beta at the moment, this allows anyone to use Guardian stories and features on their own site. There aren’t too many examples in the wild at the moment, but it’s certainly something to watch, especially for hyperlocal sites who want to add extra content to their own sites, but don’t have the resources.

The next speaker was Matthew Somerville (@dracos), who talked about some of the work he’s been doing on his own, as well as the work he’s done with MySociety.

The first thing he talked about was Scenic or Not which is basically a Hot or Not for views around the UK – you’re confronted with a randomly selected photo, which you then rate for its prettiness or otherwise. Now, I’ve got to admit, I was pretty clueless as to what use this would have when I first heard about it on the MySociety mailing list, but the use of this data literally made my jaw drop.

Once there was enough data gathered, it was then fed in to Mapumental, a system set up by MySociety with Channel 4. At the moment the system allows you to work out your daily commute, but more datasets will be added as time goes on. The video below can show you better than I can explain, so check it out:

At the moment, Mapumental is in private beta (it takes a hell of a lot of processing power to generate those maps!), but you can be put on the waiting list for an invite on the Mapumental site.

Matthew then went on to talk about TheyWorkForYou, which scrapes Hansard (the parliamentary record) to record what Mps are talking about in the house of commons, as well as other useful stuff. This allows you to keep tabs on what your local MP is talking about, as well as their voting record, their attendance etc etc (you can see an example for my MP – Mike O-Brien here).

It also turns out that TheyWorkForYou is a better resource than Hansard, a few years back, the team found out that a dead MP (Jim Marshall) had voted in two divisions despite being dead! They ended up having to contact Hansard to notify them of this mistake.

One of Matthew’s other projects is the UK postbox locator – this arose from a Freedom of Information request made by someone for the location of every postbox in the UK. The Royal Mail provided this, but claimed not to have the exact location, Matthew has used this data to put together a crowdsourcing tool. You enter the first part of your postcode, click ‘Go’, and then place the postboxes in the correct location. Over 20,000 postboxes are done so far (although B46 looks a bit shabby – sorry!) – it’s a great example of crowdsourcing in action.

Update: Matthew’s presentation is now available here.

Finally came Paul Bradshaw (@paulbradshaw), who talked very briefly about Help me Investigate – an online tool for people to collaborate in getting information from organisations. It’s still in private beta, but already it has gathered a load of useful data, from the cost of the Birmingham City Council website (staggeringly high – although it ain’t the web team’s fault) to where parking tickets are issued the most in Birmingham. This is pretty inspiring stuff, and along with What Do They Know (which is responsible for 10% of all Freedom of Information requests made) is a great way of getting public data out there, using the stick, rather than the carrot.

Overall, a great evening was had by all (despite the stifling heat!), great pre and post presentation chatter, good food, and most of all free booze! In the spirit of open data, I’ve put together a Google Spreadsheet of all the tweets hashtagged #dmapi for you to use as you wish – go at it!

It’s certainly split the gov Twitterphere into three – some like it, some hate it and some are not convinced. I’m firmly in the ‘not convinced’ camp. I’ve been thinking about making our homepage just a simple search for a while now, but I don’t think our organisation was ready at the point of the last redesign, so we went for a compromise, with a huge search box at the top of the page to try and encourage searching, as opposed to the traditional navigational route.

With Lancashire however, (which seems to be heavily influenced by Bing – Microsoft’s attempt at a Google killer) I don’t think the search is emphasised enough. Sure, it’s outlined in red, but it’s very tiny and towards the bottom of the page – I can’t see less experienced web users picking up on it. Also, the ‘quick links’ are hidden at the top right and need to be activated by a click.

I much prefer Westminster’s approach, you can’t avoid the search box, it’s bold, clear and in your face. It’s also more service driven, with links to maps, forms and payments, as well as links to the most popular services. Also, if you scroll down, you can still see more traditional things like council news and events (as well as the lovely Google map, which utilises the premier Google Maps API, to get around the Ordnance Survey licensing restrictions). It’s also very Google like, without ripping them off completely, which makes users (experienced and non-experienced) feel at home.

It’s great that trailblazing councils are doing this work though. Hopefully now we’ll see more and more councils following this line, so when we get to our next redesign, the powers that be will be much more comfortable with a simple homepage design.

For years now, old media has had the monopoly on news and comment. Even since the days of the internet, people have still gone to the good ol’ solid reliable traditional media outlets for their news, even if it is just their websites. They could ignore (or even buy into) the worlds of MySpace and Facebook because they are walled gardens where friends talk amongst friends and generally don’t talk about much of importance.

Twitter, however, is different. The walls that were there in the old days have been removed (thanks to both Twitter’s amazing API and search) and people don’t just follow the people they know in real life (in fact, I can count on the fingers of one hand the people I follow on Twitter who I knew before I started using it). Twitter is a great sounding board – people can fire off opinions about events as they happen, and even in the case of the Mumbai terrorist attacks and the Denver plane crash, be on the scene before print and TV journalists have had time to put their coats on. And the audience is potentially global.

This amount of eyeballs is gold dust to old school journos, and this is why they fear Twitter so much. Instead of engaging with it, they revert to the language of the playground, name calling and smearing until they’re blue in the face. The desperation is palpable.

However, not all journalists are like this, some, like @paulbradshaw and @joannageary actively engage with Twitter and see its power. @charltonbrooker (A.K.A. Charlie Brooker, who, amongst other great things, writes a couple of columns for the Guardian) has been experimenting with Twitter, gathering opinions from Twitter users to use in his columns and posting ‘teaser’ content.

These are just a few examples of journalists who are going in the right direction, there are loads more bright, intelligent, switched on media types who are seeing Twitter and other new web tools as an opportunity, not a threat. If the nay sayers carry on deriding Twitter and claiming to see it as an irrelevance, they’re going to be the ones who effectively sign their own P45s.