How do you evaluate the cost-benefit of the government’s digital engagement? As a result of the Improving government online review of measurement standards, several interesting discussions were started around extending the work on valuing and evaluating websites to all digital media.
Evaluation is a key priority for Government communicators. Matt Tee, Permanent Secretary and Head of Profession Government Communications, has prioritised evaluation as a key area of focus, along with skills and behaviour change. This is not surprising given the current economic climate. Government has to account for every pound spent and that means evaluating our communications activity to demonstrate cost-effectiveness.
Matt Tee has also requested that every government department develop a digital engagement strategy by March 2010, alongside the Public Accounts Committee recommendation that every department has a channel strategy, – a sign that digital engagement is being taken seriously.
So, how do we measure it?
As with any marketing communications activity, that depends on what the communications objectives are. However, there are commonalities across different campaigns and across different digital engagement tools and it’s those that I want to explore.
Recently, I’ve been working with colleagues in COI on this problem and we’ve come up with three common measures that appear to work across all digital engagement or social media tools:
- Number of relationships
- Number of user-generated content items
- Number of referrals/recommendations
1. Number of relationships
The number of relationships or connections within a network is a measure of power or influence. For example, it could be the number of followers on Twitter, number of friends in Facebook or the number of subscribers to a blog. In social network analysis, this is the basic measure of centrality within a network, which is called degree centrality.
There are other interesting measures of power within a network. For example betweenness centrality measures the degree to which a member lies between other members of a network. In the Facebook analogy, a person may have 1000 friends but have less influence than a person with 50 friends, each of whom have 1000 friends.
Betweenness centrality from red (lowest) to blue (highest)
2. Number of user-generated content items
The number of user-generated content items measures participation within the network. For example, it could be the number of comments on a blog or the number of videos uploaded to a Youtube channel. It measures the level of engagement of an audience, suggestive of active participation not simply passive interest.
3. Number of referrals/recommendations
The number of recommendations is what many seek. This measures virality, advocacy, recommendability. For example, it could be the number of retweets, the number of ‘share this’ actions or the number of pingbacks. It goes beyond mere participation; it means your content or message is valued enough to be recommended to others inside and outside the network.
We would be very interested to hear any thoughts on this. Many people are starting to think through return on investment in this area and it would be useful to have some level of consensus before applying to the government’s use of digital media for engagement. Let us know what you think.
Posted in: Evaluation and measurement
Tags: centrality, cost, digital engagment, measurement, recommendations, social media, usage, user-generated, value
Hi Adam
We’ve done a lot of thinking in this area (and already shared some of it with your colleague Joolz as part of the PR evaluation consultation). It’s also worth looking at Forrester’s work in this area:
http://www.forrester.com/Research/Document/Excerpt/0,7211,42124,00.html
http://www.forrester.com/Research/Document/Excerpt/0,7211,44421,00.html
I’d be happy to come in and chat about this if you felt it useful…
Robin Grant, We Are Social says:
December 8, 2009 at 12:17 amRobin, thanks for the links. I’ll will definitely check those out. I did look at a different Forrester report on Social Media Metrics but these look interesting too.
Actually, Joolz was one of the colleagues I referred to!
Happy to chat any time.
Adam Bailin says:
December 9, 2009 at 11:51 amHi,
We have also done extensive research and come up with a set of eGov measurement KPIs over here: http://www.imue.org
Some Spanish ministries and local governments are already implementing the IMUE scorecard.
Only problem, it is all in Spanish thus far
But happy to translate if you give us some time!
Regards,
Sergio Maldonado says:
December 9, 2009 at 12:11 pmAs measures, that’s a reasonable starting point. But there’s a real danger in picking a handful of measurable elements and defining – perhaps inadvertently – that that’s what counts in a publicly-funded digital engagement project in the eyes of agencies and auditors.
What about:
- how much use the policy team clients found the project to be
- what role digital played in relation to other channels
- what the outcome was in terms of the take up of the call to action
- how the participants felt about the process, and what they did next (did they stay involved? join the group but never re-visit?) etc
- what the return on investment was in terms of effort/cost to useful outcomes?
Will McInnes is doing interesting things on these questions at Measurementcamp, I understand: http://measurementcamp.wikidot.com/
We’ve also struggled rather to come up with an evaluation model we think works, but the ingredients for me would have to be:
- quantitative assessments of effort/cost input
- qualitative assessments of success in delivering the project
- quantitative measures of reach
- quantitative and qualitative assessments of engagement
- qualitative assessments of the value to the policy/comms clients
- an element of follow-up/review to see what the longer term impact was, including in unintended areas such as greater engagement by policy officials
Steph Gray says:
December 9, 2009 at 11:55 pmMeasurement should be against the objectives of the whole communications exercise. I think anything else is a distraction…
Steph’s right when he raises the need for qual and quant metrics to fully understand the depth of engagement. I saw one presentation at Forrester’s London conference last month which really crystallised this approach. The trick is to home in on the communications objective, and then tease out what “influence” “reach” or “quality of interaction” mean as a measurable target through each channel chosen in the digital layer of the project.
The most easily measurable parts of an exercise could well be the most distracting and unuseful in the long-term. It’s the same reason we keep being asked for “hits” rather than anything more meaningful.
Neil Franklin says:
December 11, 2009 at 3:36 pmThanks for the comments guys. I agree there is a need for further qual and quant measures in addition to the ones suggested. I guess what I was trying to do was to isolate those specifically characterised by the social nature of channel, those arising out of peer-to-peer realtionships. All of the additional measures you suggest make sense.
The risk of creating a set of standard metrics is that they become the focus and could lead to other important measures being overlooked. Conversely, the risk of not having a core set is that it’s harder to compare with other campaigns/engagements and therefore judge success. I agree that the most important thing is delivery of objectives but how do the comparable bits rate and did they contribute to the overall success?
Perhaps it is a futile exercise but it’s an interesting debate all the same!
Adam Bailin says:
December 11, 2009 at 5:39 pmP.S. Measurementcamp looks interesting, I might have to go to the next one.
Adam Bailin says:
December 11, 2009 at 5:40 pm