A new paper on evaluating communication can be found on this site and we are looking for your feedback. Called Payback and Return on Marketing Investment (ROMI) in the public sector (PDF 1.2Mb), it examines the challenges of calculating the financial effectiveness and efficiency of public sector marketing and communication. It also proposes a series of universal definitions; outlines a 10-step process to help determine sensible and robust estimates of Payback and ROMI; and provides six key principles that should underpin this.
The paper has been developed by COI on behalf of the GCN, in collaboration with two leading practitioners in the field of marketing evaluation, DDB’s Les Binet and Sarah Carter. It is thought to be the first study of its kind and is meant to stimulate discussion and debate within government, the industry and academia. So, circulate it widely and let us know what you think.
The deadline for feedback is 6 January 2010. After this, COI plans plan to hold a COI Big Thinkers session led by Les Binet and a technical workshop for departmental evaluation leads. The intention is to then produce both a ‘quick guide’ and ‘technical guide’ by April 2010.
Join the debate online by leaving your comments below or email your thoughts directly to evaluation@coi.gsi.gov.uk
Posted in: Uncategorized
Tags: feedback, investment, measurement, paper, return, ROMI
We welcome the publication of this paper and believe it should, as intended, stimulate discussion and debate about government marketing communication. There is definitely a need for more consistent evaluation of public sector communication and campaigns and this should provide a good basis for a more professional approach.
A few comments might be helpful
– The paper seems to focus on the evaluation of individual campaigns, almost as a marketing effectiveness exercise. This is worthwhile and helpful but, in our opinion, should also be extended to cover the setting of budgets to achieve maximum ROMI, and how to allocate budgets between different media and channels to achieve best effectiveness.
– It would be useful for all concerned if this was the start of building a database of case studies about government communication, so that those evaluating can benchmark against norms in behavioural change, recruitment etc. A wealth of IPA case studies obviously exist but these are self selective, in that they represent those campaigns entered for awards and are not therefore necessarily representative.
– The target audience would, we believe, benefit from more advice in two areas – firstly, how should they go about setting targets, so that they are not just ‘finger in the air’, and secondly, how do they go about isolating the impact of marketing from other factors? These are part of the ten steps but would benefit from more guidance.
– Are there any thoughts about directing public sector marketers/communication managers towards organisations that can help with evaluation (such as COI’s own econometrics framework)? We imagine that most government communication evaluation is conducted by the agencies that develop the campaigns. No doubt they conduct evaluations in a robust and professional way but issues could be raised about their independence and objectivity.
– Finally, and to be transparent about an element of our own interest,it seems odd that the main case study for the TDA relates to a period ending in 2005. Much more recently, we (Quadrant Consultants and Data2Decisions)carried out a VfM study for the TDA which updated the IPA award winning evaluation. If the TDA is going to be used as a role model for government marketing communication, it would seem to be more credible to bring the results of teacher recruitment activity up to date (2008)
We hope our comments are of benefit to the debate and are happy to be involved in further discussions.
William Annandale and Karl Weaver says:
January 4, 2010 at 7:10 pmThe discussion of this topic is to be applauded. However, I agree with Karl Weaver’s post. Accountabile marketing requires much more than measurement of individual campaigns; indeed it’s hard to see what purpose can be served by individual studies.
Science employs many laboratory assistants but we would not learn much about science by just asking the lab assistants about their experiments; what’s important in science is the accumulation of knowledge.
Marketing too is employing many statisticians to carry out solus studies, but often they provide no more insight than lab-assistants. Cass Business School has reviewed the marketing accountability practices of over 100 organisations, and published the Return on Ideas report (free copies can be obtained from CIM, CIMA and DMA). What we found is that the best organisations accumulate knowledge and learning in a systematic process, not by one-off statistical studies.
For example, an entertainment company has been modelling every campaign for 18 months and views the rolling results every week, when it reviews its media plans. The accumulated learning from many dozens of weekly reports across hundreds of campaigns has a much greater impact on media-buying decisions than the traditional annual stats PowerPoint show (which often has very little impact on decision making).
I hope these comments stimulate debate and would be happy to keep involved in the discussions.
Professor Robert Shaw says:
February 5, 2010 at 1:20 pmMany thanks for your comments on our paper – which we will take on board in moving forward over the next few months. Briefly, however, we have been working on a number of more recent case stuides and these will be reflected in the next stage – which is to publish a quick guide and a technical guide to the calculation of Payback & ROMI.
We take on board your point about strengthening target setting. We are making this point strongly in the cross-government training sessions that we are starting to run – and will do so in our guides.
We take on board your point about pointing people in the direction of the right organisations to help. This will be addressed in our guides.
The TDA case study has been updated by COI with TDA and DDB and the TDA is following this up with a further up-to-date case study.
Again, many thanks for the interest.
Kevin
Kevin Traverse-Healy says:
February 11, 2010 at 12:24 pm