How do you evaluate the cost-benefit of the government’s digital engagement?  As a result of the Improving government online review of measurement standards, several interesting discussions were started around extending the work on valuing and evaluating websites to all digital media.

Evaluation is a key priority for Government communicators. Matt Tee, Permanent Secretary and Head of Profession Government Communications, has prioritised evaluation as a key area of focus, along with skills and behaviour change. This is not surprising given the current economic climate. Government has to account for every pound spent and that means evaluating our communications activity to demonstrate cost-effectiveness.

Matt Tee has also requested that every government department develop a digital engagement strategy by March 2010, alongside the Public Accounts Committee recommendation that every department has a channel strategy, – a sign that digital engagement is being taken seriously.

So, how do we measure it?

As with any marketing communications activity, that depends on what the communications objectives are. However, there are commonalities across different campaigns and across different digital engagement tools and it’s those that I want to explore.

Recently, I’ve been working with colleagues in COI on this problem and we’ve come up with three common measures that appear to work across all digital engagement or social media tools:

1. Number of relationships
2. Number of user-generated content items
3. Number of referrals/recommendations

1. Number of relationships

The number of relationships or connections within a network is a measure of power or influence. For example, it could be the number of followers on Twitter, number of friends in Facebook or the number of subscribers to a blog. In social network analysis, this is the basic measure of centrality within a network, which is called degree centrality.

There are other interesting measures of power within a network. For example betweenness centrality measures the degree to which a member lies between other members of a network. In the Facebook analogy, a person may have 1000 friends but have less influence than a person with 50 friends, each of whom have 1000 friends.

Betweenness centrality from red (lowest) to blue (highest)

2. Number of user-generated content items

The number of user-generated content items measures participation within the network. For example, it could be the number of comments on a blog or the number of videos uploaded to a Youtube channel. It measures the level of engagement of an audience, suggestive of active participation not simply passive interest.

3. Number of referrals/recommendations

The number of recommendations is what many seek. This measures virality, advocacy, recommendability. For example, it could be the number of retweets, the number of  ‘share this’ actions or the number of pingbacks. It goes beyond mere participation; it means your content or message is valued enough to be recommended to others inside and outside the network.

We would be very interested to hear any thoughts on this. Many people are starting to think through return on investment in this area and it would be useful to have some level of consensus before applying to the government’s use of digital media for engagement.     Let us know what you think.

Read the full article (12 Comments)

This week has been one of conferences.  On Tuesday was the Public Sector Information annual conference .  It is amazing to think that it is just one year ago that, working with John Sheridan, I presented an overview of how data and structured information could be released using semantic web markup.  Since then London Gazette has been released in RDF/XML and people across government are busy implementing RDFa for consultations and in the public sector RDFa for jobs (for example Jobs Go Public’s local government jobs – LGJobs), the last two to be surfaced through Directgov.

More importantly, well underway is the Prime Minister’s drive for the release of data and creation of a single point of access (currently under development) through the appointment of Sir Tim Berners-Lee and Professor Nigel Shadbolt.   As the latter pointed out at the conference this is a single point of access not a single data base – the data will still sit in Departments, agencies and local government websites, but developers will be able to know what is available through some kind of searchable catalogue and get access to it.

He showed us a newspaper for a single postcode that had been demonstrated by some mash-up developers.  This included local data on crime, allotments, bus-stops and routes completely localised, along with lots of other useful information. We all liked it, because at present that data is something that each of us has to build up ourselves and we respond to it immediately as valuable.

The next day I spoke at the Public Sector Online annual conference organised by Kable.  The subject I was given was the cost of websites and I took the opportunity to remind all national and local government webbies that we need to be able to justify the expense of websites and demonstrate their value at this time of financial stringency.  In fact we have a great story to tell relative to many channels of communication, but I am guessing that the Finance Directors don’t yet always see that value and fund invest accordingly.

We’ve issued the standards on improving quality by measuring cost, usage and user satisfaction, following the Public Accounts Committee recommendations.  When these are reported people can start identifying lots of interesting aspects, answering the question, for example, of the value of the website channel to the nation.  Net value could be determined by the total online cost for satisfied minus unsatisfied users and subtracting the overall cost of provision. What we’d like to do is make the data available so that academics and economists can study this in more detail.

As a presenter it felt good to be getting email and comment from the audience floor which I was able to see shortly afterwards and makes a response.  The next conference on the Thursday was created to facilitate this – Government 2010.  Although I was invited to participate in person, I did so by logging in and watching the webstream. Lots of interesting thoughts, some of them inspired by Tom Steinberg’s contribution. 

It brought to mind a presentation from Martha Lane-Fox to COI on the Wednesday late afternoon, when she was asked the question about her experience in working in the public sector as the Champion for Digital Inclusion.  She responded that she had been really impressed by the calibre, intelligence and quality of the people she had met. 

It struck me that there are many talented expert e-communicators across government but hampered by the misperception that Web is IT.  There is an infrastructural element, but too often Web publishing is run as IT processes without the flexibility to change things day by day or initiate new trials or innovate.  We cannot even as I did in the old days, run two versions of a website in parallel and monitor what people do and continuously develop the more successful. 

It is like newspaper editors being unable to change the front page and only being able to stream new text into exactly the same shape of story, without being able to put in a major picture or give over the front page to a single story.  Newspapers and magazines would be very boring without that.  Likewise Web publishing should enable flexibility of all kinds of digital presentation and functionality.

Martha went on to say that public sector had many initiatives that needed joining up, not into a major programme, but under a banner that allowed lots to participate and encouraged a movement of activity adding up to more than the sum of the parts.  This would be a good description of what we want to do with the release of data and information for re-use.

A busy week of engagement that was encouraging as I and colleagues across government make the changes that support the directions sought.  What we do is often the road-building for the Ferraris (I wish!) and transport lorries that support trade to run upon, but without it they wouldn’t move.  So onto to improving quality of what we do in a measurable way, demonstrating its value to the nation, and structuring information for others to use and innovate. 

Read the full article (2 Comments)

In the current climate of open, transparent and accountable government, it is now mandatory for government websites to have stats audits. But how did this come about and why is it beneficial?

Policy background

Back in July ’06 the National Audit Office published the results of its survey of Government on the Internet. The results were pretty shocking:

Over a quarter of government organisations still do not know the costs of their websites, making it impossible to assess whether they are value for money

16% of government organisations have no data about how their websites are being used, inhibiting website improvements.

The quality of government websites has improved only slightly since 2002.

These findings were used as evidence before the UK Parliament’s Public Accounts Committee (PAC) hearing in November ’07. PAC recommended the development of a single set of measures for government website costs, quality and usage which were to be reported centrally. Government’s response to the PAC Sixteenth Report was laid before the House of Commons in September ‘08.

Consistent data

The single set of measures was developed and is now in place, but how can the data be collected reliably? Measuring website usage can be done in a number of ways with sites using different methods, tools, standards, filters and terminology. To get consistency is a real challenge.

The media industry has solved this problem. Advertising revenue is based on the number of Ad Impressions – like Page Impressions but for ads – and rates vary with volume of site usage. Advertisers need a reliable way to ensure return on investment. They need to know that the websites on which they are buying space and surfacing content measure usage accurately and consistently. The solution is to insist on a site audit certificate.

Government websites don’t tend to generate revenue from advertising – although the practice is not forbidden in principle – but they are accountable to the taxpayer. Surely taxpayers have the right to expect a decent return on their investment? If I visit a government website, how much does it cost me? Is it value for money? I want to know!

The ABCe audit

In May 2009, COI appointed ABCe to be the sole auditor of government websites. ABCe is the industry owned website auditor and is the standard for the media industry, both for media owners and media buyers. COI has negotiated cost savings for the taxpayer by centralising the spend. The average cost of an audit is approximately £2,500 compared to £4,000 if departments went to ABCe independently. By the end of the financial year, all websites run by central government departments will have had one month’s usage data audited by ABCe.

The bigger picture

Why go to all this trouble and is there any benefit to the government departments themselves? Aside from increased accountability to the taxpayer, departments do stand to benefit from the increased rigour in site measurement and evaluation. Website audits are the first step towards properly managed performance improvement. It is only with consistent and reliable data that performance metrics – or KPIs – can be developed. These are things like:

• Average number of Visits per Unique User which measures how often a user returns to a website (customer loyalty)
• Average number of Page Impressions per Visit which provides a measure of user engagement (sometimes referred to as stickiness)

When usage levels are considered alongside costs, we can also begin to consider value for money metrics such as Cost per Visit.

Central reporting of quality data also enables benchmarking of government websites against each other. For example, if I get an average Visit Satisfaction of 70% for my website, how do I know if that is good or bad compared to other websites in my sector? With a standard set of core survey questions, this is now possible. It is also worth mentioning that local government are ahead of central government in this respect. Because of initiatives like the SOCITM Website Take-up Service and Gov Metric, Local Authorities have integrated satisfaction benchmarking into their site performance management.

Monitoring KPIs over time is a key business tool for demonstrating performance improvement which is so important for getting the appropriate level of investment in government digital media.

Central reporting of Visit Duration is a contentious issue. While it is probably not useful to compare websites on this metric – a long time on site may indicate a high level of engagement or a site that is difficult to navigate – it does provide interesting census-level data. Measuring Visit Duration enables Government to calculate the total amount of time spent on its websites by citizens. We can begin to get a picture of the value delivered to citizens by government online. For example, if we compare the cost of delivery to the cost for the citizen then we can begin to address the cost-benefit of online services to the citizen. Now that would be interesting!

Read the full article (11 Comments)