Here at Big Thinkers we would really like to hear your views – philosophical or technical – so please post below or email us at evaluation@coi.gsi.gov.uk.

Les, winner of more IPA Effectiveness Awards than anyone in the history of the competition, and co-editor of the COI’s Payback and Return on Marketing Investment(ROMI)in the public sector (PDF1.2Mb), explained the ten steps outlined in the document.

He opened on the importance of setting clear objectives; and understanding how a campaign will work and whom it will affect. With this in mind he gave some top tips on measurement:

• Think about evaluation before the campaign.
• Allocate time and money for measurement – and analysis of data.
• Metrics should reflect objectives and strategy.
• Tracking for diagnosis, behaviour for evaluation.
• Direct response is not the whole picture – by a long way.
• Measure actual (not claimed) exposure to campaign
• Plan activity with evaluation in mind. Build in tests and controls if possible.
• Measure and take account of other factors.
• Track all measures pre and post, and look beyond the short term.

This led to the difficult bits: how do you isolate the effect of a campaign on behaviour change from the effects of other factors – and how do you put a financial value on success?

Les presented a number of techniques to deal with the former and an interesting discussion arose about the difficulties of isolating the effects of smaller-scale or non-campaign-based activities where there is too little data or budget. It was suggested that one way around this is to look at the overall payback from observed behaviour change. Given the cost of the campaign, what proportion of this would it have to claim in order to have paid for itself? How much to have paid for itself twice over?

On financial valuation, it was clear that although there are sometimes straightforward proxies (like the money saved on supply teaching by recruitment of permanent staff), policy and economics teams in departments should be consulted for the latest on the ‘value’ of behaviour change to the exchequer.

Finally, but most importantly, came the debate about what to do with the results. If campaign A pays for itself several times over but campaign B barely breaks even, should campaign B be axed? The answer was no – at least not on this evidence alone. It might be that more should be invested in campaign B so it can reach a ‘tipping point’. It might be that B is past the point of greatest financial return but is rightly seeking to influence a hardened minority as part of government policy.

In all cases, Net Payback (the absolute value of benefit delivered, less the cost of the campaign) and ROMI (the number of pounds of Net Payback for every pound spent) should be seen as just one part of the evaluation of government marketing communication.

So why try to calculate Return on Investment at all? Because there are real benefits from undertaking the analysis:

• It requires a dialogue with policy colleagues about the contribution of marketing communication alongside other policy levers – such as legislation and intervention;
• It leads to a greater understanding of how marketing communication delivers value and the need to quantify and isolate campaign impact on behaviour change drives better evaluation;
• By speaking the same language and working to the same rules as accountants and economists, it gives more credibility to marketing communication and allows it to be considered as an ‘investment’; and
• In an era of economic uncertainty, it is one way to demonstrate the value of marketing communication to the taxpayer and ensure that spending decisions are made ‘intelligently’.

The COI team are working to turn their beta version report into two guides; one simple and one technical; by mid-April – and they would love to hear your views – either on this blog or direct.