Pezholio

02 Aug, 2010

A Local Spending Data Brain Splurge

Posted by: Pez In: Council Stuff|Open Data

Today I was at a ‘quick and dirty’ local spending data workshop at Birmingham City Council’s newly refurbished offices on Lancaster Circus. There’ll be more detailed info to come on the Local Open Data Community (login required), but I just wanted to blog a few of my thoughts post the meeting.

There was a lot of talk about what other councils have done (such as Windsor and Maidenhead, Barnet, Islington etc), and it was agreed that there was a lot of difference in how the data was presented, Paul Davidson also talked about some of the work that the Local eGovernment Standards Body had done regarding getting spending data out there in a linked data format. There did seem to be a bit of resistance to the linked data approach, mainly because agreeing standards seems to be a long, drawn out process, which is counter to the JFDI approach of publishing local data.

However, while I am a fan of the possibilities of linked data, I also recognise that there are difficulties in both publishing the data and also working with it. For example, I think it’s unrealistic to expect every local authority to maintain a triple store to publish their spending data. As we learned from the local elections project, often local authorities don’t even have people who are competent in HTML, let alone RDF, SPARQL etc.

Therefore, I think the way forward is a centralised approach, with authorities publishing CSVs in a standard format on their website and some kind of system picking up these CSVs (say, on a monthly basis) and converting this data to a linked data format (as well as publishing in vanilla XML, JSON and CSV format).

The great thing about the linked data approach is it will mean that each item of spending can have its own URI – e.g.

http://localspending.data.gov.uk/41UD/12345

(The first part of the URI would be the SNAC code for the authority, and then the second part of the URI would be the internal reference number)

As well as having a human-readable summary of the data (together with links to the actual data in RDF, XML, CSV and JSON), there would be a comments box (similar to Adrian Short’s fantastic Armchair Auditor), as well as the ability to ask any questions about an item of expenditure – the answers to these questions would then be automatically published next to the item of spend (hopefully helping to cut down on multiple FOI requests).

While this may be a bit of a pie in the sky idea, I do feel that there does need to be some kind of effort on the part of central government to help move this project along, as we’ve seen already (naming no names!) some authorities have got it drastically wrong, and while there is definitely mileage in the ‘just get it out there’ approach, I think if we’re going to end up with something *really* useful (for both members of the public and local authorities), we need to get the data in one place.

Thoughts?

16 Responses to "A Local Spending Data Brain Splurge"

1 | Chris Taggart

August 2nd, 2010 at 3:41 pm

Avatar

Stuart
See OpenlyLocal’s spending data section. Got pretty much most of what you’ve asked for:
- URL per transaction (e.g. http://openlylocal.com/financial_transactions/36396).
- Pulling in all files from CSV and exposing as data (XMLS and JSON, RDF when the vocab is sorted)
- human readable summary of the data (see the spending dashboards for each of the individual councils)

It’s also got the ability to link to actual companies and other councils (which a fair proportion of the data is), and will in about an hour’s time have the entire thing downloadable as a single zip archive. Phew!

The only thing it hasn’t got is comments, and that’s only because I’m not sure about distributed comments systems like disqus (which Adrian uses), which require javascript, and therefore have accessibility issues. However, happy to take suggestions/advice re this.

2 | Andrew Beeken

August 2nd, 2010 at 3:49 pm

Avatar

I definately like the idea; a standard approach is what’s needed. I’m currently in the process of pulling the data together for Lincoln. Chris, what layout are you expecting the CSV’s in?

3 | Pez

August 2nd, 2010 at 4:00 pm

Avatar

Yeah, Disqus is great for adding a quick and dirty comments system to a site, but I agree, the reliance on JS is a sticking point, and making a completely spam free comments system isn’t trivial! Would be nice if there was some way of adding user generated content though – especially if there’s a particularly tricky amount of spending which may generate a lot of FOIs

4 | Hadley Beeman

August 2nd, 2010 at 4:22 pm

Avatar

YES!!! I completely agree. I’m hoping that’s what this community data project I’m trying to organise will help accomplish. (And when I say “organise”, I really mean that– tell me what you think we should do, and I’ll see if we can bring people together to make it happen!)

A big chunk of today, for me, has been spent on working out how to create a “system” for CSV from council -> reformatted to RDF -> modelled (by crowdsourcing? by a few select who have time? still pending) into a government schema -> reformatted/enhanced data released to developers via APIs.

(I’m still at it, by the way. Learning a ton as I go.)

Since you’re obviously thinking about it… What do you think? How should we make this happen? :)

5 | Chris Taggart

August 2nd, 2010 at 4:37 pm

Avatar

Stuart:

I’m happy to add a comments system on there. Have down one before, so not too difficult, just wasn’t sure whether local comments or distributed ones was the right way to go.

Either way, it’s a doddle to add. Anything else you think we’re not doing that should be done, or do you think we’ve pretty much solved the prob.

Andrew:
Can cope with most things, but normal well-formed CSV (ideally UTF-8) is fine. The main thing is to give us as much detail as possible — date, amount, transaction id, supplier name, department/directorate, service, description, and ideally CIPFA BVACOP code and supplier details too.

6 | Ingrid Koehler

August 2nd, 2010 at 4:37 pm

Avatar

Please hold fire for TWO days (really that’s all I’m asking). I think you’ll be pleasantly surprised.

7 | Chris Taggart

August 2nd, 2010 at 6:33 pm

Avatar

Ingrid
Hope that doesn’t mean another centrally-funded/procurred scheme using up resources that would be better spent on the frontline.
C

8 | Ingrid Koehler

August 3rd, 2010 at 4:55 am

Avatar

No. Something completely optional, but with many of the features Stuart describes. No new big fancy procurement.

9 | A Local Spending Data wish… granted « countculture

August 3rd, 2010 at 8:45 am

Avatar

[...] The very wonderful Stuart Harrison (aka pezholio), webmaster at Lichfield District Council, blogged yesterday with some thoughts about the publication of spending data following a local spending data workshop in Birmingham. Sadly I wasn’t able to attend this, [...]

11 | Pez

August 3rd, 2010 at 2:14 pm

Avatar

Good stuff, ta! :)

12 | Ingrid Koehler

August 4th, 2010 at 11:14 am

Avatar

OK – well, all it is – is basically that the ESD toolkit is going to help with councils who might need help publishing the data – so will provide a conduit for that. This doesn’t yet fully grant your wish, of course – but it would be a platform for publication. Ideally, councils would publish to their own websites, but not every council is in that position.

I believe ESD toolkit are also offering help on some of the data formatting as well – which would help surfacing tools be that much easier to make. Some of the data formatting help would be about getting it clean and consistent, but some of the work would be about starting us down the path of linked data.

Although ESD toolkit is a subscription service mostly, it would be free to use and to view.

13 | Martin Stone

August 5th, 2010 at 12:23 pm

Avatar

May be we should be looking at a UK equivalent of http://www.Socrata.com

With this service you can upload CSV or excel and it creates the necessary machine readable formats.

It also allows viewers to comment on the data

14 | SPEED DATA: Ideas for local government spending transparency « The Dan Slee Blog

August 6th, 2010 at 11:59 am

Avatar

[...] Pezholio blog on the SOCITM Birmingham local data event. A useful summary and some very useful comments. [...]

15 | Theunis Viljoen

August 24th, 2010 at 5:40 pm

Avatar

Although some Councils have already started to publish details of payments to suppliers over £500, there is obviously a reticense to develop costly reporting solutions. Councils therefore appear to publish the information in downloadable format either as Excel spreadsheets or PDF documents. Although this may ‘tick’ the required disclosure requirements, we do not believe that this provides real value to the public.

Information such as this only has real value if it can be viewed in context (i.e. is a payment normal or abnormal). For a user to be able to make this interpretation, they may need to be able to look at spend in a month for a particular expense category against similar spend in previous periods or against other categories.

BIOLAP has developed an application, driven by arcplan technology, that we will provide to Councils free of charge to allow members of the public to analyse expenditure, slice and dice information and drill through to the underlying transactions.

Please feel free to try out our Council Expenses Dashboard at – http://www.biolap.co.uk/index.php/councilexpenses.html

16 | Pezholio » Blog Archive » Spending data – beyond the CSV

January 14th, 2011 at 10:06 am

Avatar

[...] been a while since my last post about local spending data, and, since then, our finance team have been busy beavering away behind the scenes to get our own [...]

Comment Form

Where I’m at