I am really proud of my government colleagues for getting the data together to enable us to publish the very first report on the costs, quality and usage of central government websites. You can find it, along with all the data, on www.coi.gov.uk/websitemetrics2009-10

 Local government has been comparing like for like to benchmark for many years.  It is more difficult in central government as the audiences and functions of websites vary considerably.  Some are for the public (Diectgov, NHS Choices), for businesses (businesslink.gov.uk) and for public sector workforces (civil servants, teachers, armed forces).  And others are for people to engage with a particular organisation in policy formation (corporate websites).  While yet others have a regulatory function (Ofgem).

It’s made harder by some public bodies doing lots of syndication, placing information onto other websites where people regularly go, and early adoption of re-usable information and data, so that people can present it in new ways.  Both these result in people using the information, but not being recorded as visiting the website to do so. So the cost per visit is not related in any way to the cost per use, nor indeed the value to the user.

The data makes for interesting reading if not immediate analysis.  The notes that reporting public bodies have attached should be read in conjunction with the data.  However, some things do jump out.

One that struck me was the variation in cost for hosting and infrastructure compared to usage.  One wouldn’t expect a bell-shaped curve, as some sites need to have much more resilience and security than others, but if a free competitive market was really in operation, then one would expect more clustering around a few price points.

The other areas of expenditure will vary more according to the stage of development.  This year you might spend a lot on design and build to bring it up to what people expect, next year perhaps very little. Per year websites tell you something, but are not the whole picture – but how much better than not having anything!  Having knowledge of the order of the cost is really helpful and, even taking out all those who didn’t get anything from their visit, the cost for each reach is very low compared to any alternative.

And we can begin to look at some other interesting aspects.  Does the spending made on testing and evaluation result in an increase in user satisfaction?  By releasing the data, we’re keen to see what use people make of it and how they can add insight into the variety of questions that arise. Ross Ferguson, for example,  has done a visualisation of the number of yearly unique visitors/browsers (a figure that not everyone could provide as it requires deduplicating across all the monthly data).

We know people are interested.  In the three and a bit days since publication, there have been 14,534 downloads of the PDF and 1,034 downloads of the CSV.  It’s been interesting too to follow the use of the official bit.ly tag that we tweeted and discover where people find where the data is.  We’re looking forward to seeing what people do with it.

In terms of numbers, 46 is a very small sample.  Next year all the open central government websites are due to report and it doesn’t seem sensible to issue a report simply listing the data as we have done this year.  (We will of course issue the data as a dataset.)  How people analyse the data will help shape the human-readable report.

Let us know what you’re doing, and what you’d like to see.

Read the full article (10 Comments)

How do you evaluate the cost-benefit of the government’s digital engagement?  As a result of the Improving government online review of measurement standards, several interesting discussions were started around extending the work on valuing and evaluating websites to all digital media.

Evaluation is a key priority for Government communicators. Matt Tee, Permanent Secretary and Head of Profession Government Communications, has prioritised evaluation as a key area of focus, along with skills and behaviour change. This is not surprising given the current economic climate. Government has to account for every pound spent and that means evaluating our communications activity to demonstrate cost-effectiveness.

Matt Tee has also requested that every government department develop a digital engagement strategy by March 2010, alongside the Public Accounts Committee recommendation that every department has a channel strategy, – a sign that digital engagement is being taken seriously.

So, how do we measure it?

As with any marketing communications activity, that depends on what the communications objectives are. However, there are commonalities across different campaigns and across different digital engagement tools and it’s those that I want to explore.

Recently, I’ve been working with colleagues in COI on this problem and we’ve come up with three common measures that appear to work across all digital engagement or social media tools:

1. Number of relationships
2. Number of user-generated content items
3. Number of referrals/recommendations

1. Number of relationships

The number of relationships or connections within a network is a measure of power or influence. For example, it could be the number of followers on Twitter, number of friends in Facebook or the number of subscribers to a blog. In social network analysis, this is the basic measure of centrality within a network, which is called degree centrality.

There are other interesting measures of power within a network. For example betweenness centrality measures the degree to which a member lies between other members of a network. In the Facebook analogy, a person may have 1000 friends but have less influence than a person with 50 friends, each of whom have 1000 friends.

Betweenness centrality from red (lowest) to blue (highest)

2. Number of user-generated content items

The number of user-generated content items measures participation within the network. For example, it could be the number of comments on a blog or the number of videos uploaded to a Youtube channel. It measures the level of engagement of an audience, suggestive of active participation not simply passive interest.

3. Number of referrals/recommendations

The number of recommendations is what many seek. This measures virality, advocacy, recommendability. For example, it could be the number of retweets, the number of  ‘share this’ actions or the number of pingbacks. It goes beyond mere participation; it means your content or message is valued enough to be recommended to others inside and outside the network.

We would be very interested to hear any thoughts on this. Many people are starting to think through return on investment in this area and it would be useful to have some level of consensus before applying to the government’s use of digital media for engagement.     Let us know what you think.

Read the full article (12 Comments)

This week has been one of conferences.  On Tuesday was the Public Sector Information annual conference .  It is amazing to think that it is just one year ago that, working with John Sheridan, I presented an overview of how data and structured information could be released using semantic web markup.  Since then London Gazette has been released in RDF/XML and people across government are busy implementing RDFa for consultations and in the public sector RDFa for jobs (for example Jobs Go Public’s local government jobs – LGJobs), the last two to be surfaced through Directgov.

More importantly, well underway is the Prime Minister’s drive for the release of data and creation of a single point of access (currently under development) through the appointment of Sir Tim Berners-Lee and Professor Nigel Shadbolt.   As the latter pointed out at the conference this is a single point of access not a single data base – the data will still sit in Departments, agencies and local government websites, but developers will be able to know what is available through some kind of searchable catalogue and get access to it.

He showed us a newspaper for a single postcode that had been demonstrated by some mash-up developers.  This included local data on crime, allotments, bus-stops and routes completely localised, along with lots of other useful information. We all liked it, because at present that data is something that each of us has to build up ourselves and we respond to it immediately as valuable.

The next day I spoke at the Public Sector Online annual conference organised by Kable.  The subject I was given was the cost of websites and I took the opportunity to remind all national and local government webbies that we need to be able to justify the expense of websites and demonstrate their value at this time of financial stringency.  In fact we have a great story to tell relative to many channels of communication, but I am guessing that the Finance Directors don’t yet always see that value and fund invest accordingly.

We’ve issued the standards on improving quality by measuring cost, usage and user satisfaction, following the Public Accounts Committee recommendations.  When these are reported people can start identifying lots of interesting aspects, answering the question, for example, of the value of the website channel to the nation.  Net value could be determined by the total online cost for satisfied minus unsatisfied users and subtracting the overall cost of provision. What we’d like to do is make the data available so that academics and economists can study this in more detail.

As a presenter it felt good to be getting email and comment from the audience floor which I was able to see shortly afterwards and makes a response.  The next conference on the Thursday was created to facilitate this – Government 2010.  Although I was invited to participate in person, I did so by logging in and watching the webstream. Lots of interesting thoughts, some of them inspired by Tom Steinberg’s contribution. 

It brought to mind a presentation from Martha Lane-Fox to COI on the Wednesday late afternoon, when she was asked the question about her experience in working in the public sector as the Champion for Digital Inclusion.  She responded that she had been really impressed by the calibre, intelligence and quality of the people she had met. 

It struck me that there are many talented expert e-communicators across government but hampered by the misperception that Web is IT.  There is an infrastructural element, but too often Web publishing is run as IT processes without the flexibility to change things day by day or initiate new trials or innovate.  We cannot even as I did in the old days, run two versions of a website in parallel and monitor what people do and continuously develop the more successful. 

It is like newspaper editors being unable to change the front page and only being able to stream new text into exactly the same shape of story, without being able to put in a major picture or give over the front page to a single story.  Newspapers and magazines would be very boring without that.  Likewise Web publishing should enable flexibility of all kinds of digital presentation and functionality.

Martha went on to say that public sector had many initiatives that needed joining up, not into a major programme, but under a banner that allowed lots to participate and encouraged a movement of activity adding up to more than the sum of the parts.  This would be a good description of what we want to do with the release of data and information for re-use.

A busy week of engagement that was encouraging as I and colleagues across government make the changes that support the directions sought.  What we do is often the road-building for the Ferraris (I wish!) and transport lorries that support trade to run upon, but without it they wouldn’t move.  So onto to improving quality of what we do in a measurable way, demonstrating its value to the nation, and structuring information for others to use and innovate. 

Read the full article (2 Comments)