Today is World Usability Day and to mark the occasion, we here at Digigov have developed an introduction to low cost usability methods that might be useful to you in these austere times.

The Government announced its intention to cut spending on websites by up to 50%. But we need to deliver a consistently excellent user experience on the government web estate. So the question is how can we manage usability budgets most effectively and efficiently?

Clearly the most necessary aspect of usability is to test with users from the target audience.  Where the product or service is live, then feedback will be given through their use and feedback.  What can we do to get things right much earlier on?  To get it as close to effective for the user as possible? What can we do once the product or service is live and we want to check improvements?

People often associate usability only with user testing in laboratories, which can be expensive if there are many iterations.  In this article I will describe some alternative lower cost usability evaluation methods and look at the pros and cons of each method.  These don’t replace testing with users.  They are tools that offer getting to a usable product earlier with more efficiency.

Task analysis

Task analysis involves taking the tasks a user would do on your website and mapping out the individual steps required to complete each task. This type of analysis often yields a number of sub-tasks required to complete each individual step and each sub-task may be broken down further and so on. This approach is known as hierarchical task analysis. The purpose is to identify any steps that are unnecessary or that may slow the user down in achieving their desired goal. The process of stripping out unnecessary steps and working out the best way to design a given task can lead to a highly optimised and efficient user experience.

Pros

• Relatively cheap
• Focussed on user actions
• It can be used both on existing websites but especially when designing new services to identify any obvious extra steps that may be removed.

Cons

• Can be complex depending on the task in question
• Can be lengthy to complete a thorough task analysis, e.g. a complex website with many different functions
• Doesn’t identify any new tasks that may be appropriate for the site to deliver.

Task analysis assumes you are offering the right services and helps to optimise them but doesn’t provide any insight into whether you are designing the right thing.  That comes from user feedback.

Online surveys

Online surveys are an inexpensive method of eliciting user feedback. Crucially they allow people to say what they think about their online experience immediately after the event and therefore provide a good indication of people’s true feelings. Surveys are used to establish site ratings (e.g. satisfaction, ease of use etc) and to assess why people are coming to a site (i.e. the user goal) and the extent to which they achieve the intended purpose of their visit (i.e. goal completion).

Pros

• Low cost
• Close to the experience
• Provides insight into who your users are and why they are coming to your site

Cons

• Subjective and therefore not as reliable as observing users
• Can be annoying to users
• Takes time to analyse results and make actionable recommendations for improvement

There’s an art to questionnaire design, so we advise getting a professional to do it testing with a few people before going live. Don’t ask unneccessary questions, every question should have a reason for being asked. A useful starting point is the set of core questions defined in the COI guidance on survey design.

Heuristic evaluation

Heuristic evaluation is a kind of expert review where a website (or any user interface) is evaluated against a set of established design heuristics (or rules of thumb). The most commonly used set is Nielsen’s Ten Usability Heuristics. However there are others, for example Schneiderman’s Eight Golden Rules and  Don Norman’s principles of design. An experienced usability practitioner will evaluate against a the most appropriate set for the type of interface being evaluated and, dependent on experience, may have a bespoke set that works best for them.

Heuristic evaluation is most effective when carried out on selected user tasks. As the evaluator works through the individual steps to complete a task, usability problems are recorded along with the heuristic or design principle that it breaches. Positive points may also be noted but on the whole the purpose of usability evaluation is to find problems and make recommendations for improvement.

Pros

• Quickly identifies potential usability problems
• Useful to spot obvious issues early on in the design process
• Can save money spent fixing problems further down the line

Cons

• Effectiveness depends on the experience of the expert
• Less reliable than user testing because different experts interpret heuristics differently

Heuristic evaluation can be as effective as user testing in identifying usability problems and cost a fraction of the price. If using this method try to get the most experienced testers you can afford and agree up front how each heuristic will be interpreted to avoid any misunderstandings later on.

Cognitive and pluralistic walkthroughs

Cognitive walkthrough is where an expert walks through a task associated with your website, putting themselves in the shoes of a typical user. At each step, the following questions are posed:

• Will the user know what to do?
• Is the correct option available?
• Is the option is available, will the user expect that it will yield the intended result?
• Will the user know that the intended result has been yielded?

Examining each step in this way, the expert notes successes and failures along the way. As with heursitic evaluation, cognitive walkthroughs are used early in the design process.

Pros

• Low cost
• Task focussed
• Quickly identifies usability problems early in the design process

Cons

• Doesn’t test with real users
• Relies on expert’s ability to put themselves in user’s shoes

Cognitive walkthroughs assume a well defined set of tasks and steps are available to carry out the evaluation.

Walkthroughs can also be performed along with users,  domain experts and developers in a group scenario. This is know as a pluralistic wakthrough and can help to identify further usability problems due to the different types of participant involved.

Standards and guidelines

Standards and guidelines such as the W3C Web Content Accessibility Guidelines have been designed to take the hard work out of web design. Following established good practice can be an easy way to avoid many of the potential pitfalls that could lead to a poor user experience. There’s no point spending money on user testing or heuristic evaluation when the basic minimum standards haven’t been applied. In a government context the COI web standards and guidelines and the Usability Toolkit are a useful starting point.

Pros

• Low cost
• Someone has done the hard work for you

Cons

• Can lead to a tickbox mentality where standards compliance is valued above user needs
• Not necessarily user-centred

Compliance with standards and guidelines can be checked using automated tools but it usually requires an expert to do a full conformance inspection.

Summary

The methods introduced in this article are not intended as a replacement for user testing. User testing is still the most reliable way to get real user insight and generate improvements. Use these methods where appropriate and in support of other analysis, design and evaluation activities.

Read the full article (2 Comments)

I am really proud of my government colleagues for getting the data together to enable us to publish the very first report on the costs, quality and usage of central government websites. You can find it, along with all the data, on www.coi.gov.uk/websitemetrics2009-10

 Local government has been comparing like for like to benchmark for many years.  It is more difficult in central government as the audiences and functions of websites vary considerably.  Some are for the public (Diectgov, NHS Choices), for businesses (businesslink.gov.uk) and for public sector workforces (civil servants, teachers, armed forces).  And others are for people to engage with a particular organisation in policy formation (corporate websites).  While yet others have a regulatory function (Ofgem).

It’s made harder by some public bodies doing lots of syndication, placing information onto other websites where people regularly go, and early adoption of re-usable information and data, so that people can present it in new ways.  Both these result in people using the information, but not being recorded as visiting the website to do so. So the cost per visit is not related in any way to the cost per use, nor indeed the value to the user.

The data makes for interesting reading if not immediate analysis.  The notes that reporting public bodies have attached should be read in conjunction with the data.  However, some things do jump out.

One that struck me was the variation in cost for hosting and infrastructure compared to usage.  One wouldn’t expect a bell-shaped curve, as some sites need to have much more resilience and security than others, but if a free competitive market was really in operation, then one would expect more clustering around a few price points.

The other areas of expenditure will vary more according to the stage of development.  This year you might spend a lot on design and build to bring it up to what people expect, next year perhaps very little. Per year websites tell you something, but are not the whole picture – but how much better than not having anything!  Having knowledge of the order of the cost is really helpful and, even taking out all those who didn’t get anything from their visit, the cost for each reach is very low compared to any alternative.

And we can begin to look at some other interesting aspects.  Does the spending made on testing and evaluation result in an increase in user satisfaction?  By releasing the data, we’re keen to see what use people make of it and how they can add insight into the variety of questions that arise. Ross Ferguson, for example,  has done a visualisation of the number of yearly unique visitors/browsers (a figure that not everyone could provide as it requires deduplicating across all the monthly data).

We know people are interested.  In the three and a bit days since publication, there have been 14,534 downloads of the PDF and 1,034 downloads of the CSV.  It’s been interesting too to follow the use of the official bit.ly tag that we tweeted and discover where people find where the data is.  We’re looking forward to seeing what people do with it.

In terms of numbers, 46 is a very small sample.  Next year all the open central government websites are due to report and it doesn’t seem sensible to issue a report simply listing the data as we have done this year.  (We will of course issue the data as a dataset.)  How people analyse the data will help shape the human-readable report.

Let us know what you’re doing, and what you’d like to see.

Read the full article (10 Comments)

A big thank you and well done to everyone who contributed to a great event last Thursday.

30 people from Government executive agencies and non-departmental public bodies (NDPBs) came to find out about website auditing.

Attendees were given an introduction by Alex Butler, COI’s Transformational Strategy Director. Alex told how this work came about as the result of the National Audit Office (NAO) report Government on the Internet (July 2007). The NAO report found that:

• a quarter of government organisations did not know the costs of their websites
• 16% had no data about how much their websites were being used
• quality standards had only improved slightly since 2002

Alex explained how the audits form part of a package of improvements offered by COI including standard measures for website quality, value for money and usability. These measures form part of the requirement by the Public Accounts Committee for greater accuracy and transparency in how the Government manages its investment in digital media.

Those present then watched a short clip of COI Chief Executive Mark Lund’s presentation at the Internet Advertising Bureau event IAB Engage 2009. Mark describes the work COI is doing with the Cabinet Office and other parts of Government in setting standards and evaluating the cost-effectiveness government websites.

Richard Foan, ABCe’s Managing Director, then talked through the details of website auditing. Richard emphasised the need for standards in an industry where there are so many different methods, metrics and tools to choose from. He explained that ABCe are an independent non-profit organisation, auditing to industry standards. ABCe certify that government websites comply with the standards and measure website usage in a consistent way.

The question and answer session that followed was lively and yielded some useful actions:

• We are planning an FAQ page on the COI website based on the questions asked
• Technical guidance will be available from the planned FAQ page but also from this page: TG116a ABCe Audit (PDF, 541 kB) and Guide to Google Analytics Mirror Method (PDF, 157 kB)
• Follow up meetings will be arranged with those who requested them
• Slides from the event are available from this page (PDF, 1.5 MB)
• Access to the Government Website Database is available by contacting graham.phillips@coi.gsi.gov.uk

Thanks to all those who attended, I hope you found it useful. And thanks particularly to Linda who worked the floor extremely well and made a star entrance!

For those who missed the event, we are planning to do similar events in the future. But in the meantime hopefully this blog article gives a flavour of what took place.

Read the full article (6 Comments)

It’s a new year and here at COI the digital policy team have loads of exciting events and projects coming up.  I’ll tell you all about that in a minute.  My first priority is the government website audit project.  I’ll remind you  just in case you have forgotten what the project is all about. The Public Accounts Committee (PAC) in their Sixteenth Report recommended that Government develop a single set of reporting metrics for website usage. The Government agreed and made it mandatory for all government websites to be audited. Hence the Government Website Audit project.

We have an event coming up in London on the 21^st of January aptly named “Auditing Government Websites”.  Speaking at the event is our very own Alex Butler, COI Transformational Strategy Director, and ABCe’s Managing Director Richard Foan.  The event is aimed at government agencies (EAs) and non-departmental public bodies (NDPBs) and will cover the following topics:

• importance of auditing
• industry standards for measuring websites
• how to get ready for the audit
• the audit process (pre audit and post audit)

The event provides an opportunity for government agencies and NDPBs to audit their websites for approximately half the amount if procured independently. What better way to start the year? The event is free and registration for the event and the audits are pouring in, so register now and make your websites 2010 compliant.

Please email me at linda.morakinyo@coi.gsi.gov.uk if you would like to attend.

Hope to see you all at there.

Invitation to auditing government websites event 21 Jan 2010

Read the full article (1 Comment)

How do you evaluate the cost-benefit of the government’s digital engagement?  As a result of the Improving government online review of measurement standards, several interesting discussions were started around extending the work on valuing and evaluating websites to all digital media.

Evaluation is a key priority for Government communicators. Matt Tee, Permanent Secretary and Head of Profession Government Communications, has prioritised evaluation as a key area of focus, along with skills and behaviour change. This is not surprising given the current economic climate. Government has to account for every pound spent and that means evaluating our communications activity to demonstrate cost-effectiveness.

Matt Tee has also requested that every government department develop a digital engagement strategy by March 2010, alongside the Public Accounts Committee recommendation that every department has a channel strategy, – a sign that digital engagement is being taken seriously.

So, how do we measure it?

As with any marketing communications activity, that depends on what the communications objectives are. However, there are commonalities across different campaigns and across different digital engagement tools and it’s those that I want to explore.

Recently, I’ve been working with colleagues in COI on this problem and we’ve come up with three common measures that appear to work across all digital engagement or social media tools:

1. Number of relationships
2. Number of user-generated content items
3. Number of referrals/recommendations

1. Number of relationships

The number of relationships or connections within a network is a measure of power or influence. For example, it could be the number of followers on Twitter, number of friends in Facebook or the number of subscribers to a blog. In social network analysis, this is the basic measure of centrality within a network, which is called degree centrality.

There are other interesting measures of power within a network. For example betweenness centrality measures the degree to which a member lies between other members of a network. In the Facebook analogy, a person may have 1000 friends but have less influence than a person with 50 friends, each of whom have 1000 friends.

Betweenness centrality from red (lowest) to blue (highest)

2. Number of user-generated content items

The number of user-generated content items measures participation within the network. For example, it could be the number of comments on a blog or the number of videos uploaded to a Youtube channel. It measures the level of engagement of an audience, suggestive of active participation not simply passive interest.

3. Number of referrals/recommendations

The number of recommendations is what many seek. This measures virality, advocacy, recommendability. For example, it could be the number of retweets, the number of  ‘share this’ actions or the number of pingbacks. It goes beyond mere participation; it means your content or message is valued enough to be recommended to others inside and outside the network.

We would be very interested to hear any thoughts on this. Many people are starting to think through return on investment in this area and it would be useful to have some level of consensus before applying to the government’s use of digital media for engagement.     Let us know what you think.

Read the full article (12 Comments)

This week has been one of conferences.  On Tuesday was the Public Sector Information annual conference .  It is amazing to think that it is just one year ago that, working with John Sheridan, I presented an overview of how data and structured information could be released using semantic web markup.  Since then London Gazette has been released in RDF/XML and people across government are busy implementing RDFa for consultations and in the public sector RDFa for jobs (for example Jobs Go Public’s local government jobs – LGJobs), the last two to be surfaced through Directgov.

More importantly, well underway is the Prime Minister’s drive for the release of data and creation of a single point of access (currently under development) through the appointment of Sir Tim Berners-Lee and Professor Nigel Shadbolt.   As the latter pointed out at the conference this is a single point of access not a single data base – the data will still sit in Departments, agencies and local government websites, but developers will be able to know what is available through some kind of searchable catalogue and get access to it.

He showed us a newspaper for a single postcode that had been demonstrated by some mash-up developers.  This included local data on crime, allotments, bus-stops and routes completely localised, along with lots of other useful information. We all liked it, because at present that data is something that each of us has to build up ourselves and we respond to it immediately as valuable.

The next day I spoke at the Public Sector Online annual conference organised by Kable.  The subject I was given was the cost of websites and I took the opportunity to remind all national and local government webbies that we need to be able to justify the expense of websites and demonstrate their value at this time of financial stringency.  In fact we have a great story to tell relative to many channels of communication, but I am guessing that the Finance Directors don’t yet always see that value and fund invest accordingly.

We’ve issued the standards on improving quality by measuring cost, usage and user satisfaction, following the Public Accounts Committee recommendations.  When these are reported people can start identifying lots of interesting aspects, answering the question, for example, of the value of the website channel to the nation.  Net value could be determined by the total online cost for satisfied minus unsatisfied users and subtracting the overall cost of provision. What we’d like to do is make the data available so that academics and economists can study this in more detail.

As a presenter it felt good to be getting email and comment from the audience floor which I was able to see shortly afterwards and makes a response.  The next conference on the Thursday was created to facilitate this – Government 2010.  Although I was invited to participate in person, I did so by logging in and watching the webstream. Lots of interesting thoughts, some of them inspired by Tom Steinberg’s contribution. 

It brought to mind a presentation from Martha Lane-Fox to COI on the Wednesday late afternoon, when she was asked the question about her experience in working in the public sector as the Champion for Digital Inclusion.  She responded that she had been really impressed by the calibre, intelligence and quality of the people she had met. 

It struck me that there are many talented expert e-communicators across government but hampered by the misperception that Web is IT.  There is an infrastructural element, but too often Web publishing is run as IT processes without the flexibility to change things day by day or initiate new trials or innovate.  We cannot even as I did in the old days, run two versions of a website in parallel and monitor what people do and continuously develop the more successful. 

It is like newspaper editors being unable to change the front page and only being able to stream new text into exactly the same shape of story, without being able to put in a major picture or give over the front page to a single story.  Newspapers and magazines would be very boring without that.  Likewise Web publishing should enable flexibility of all kinds of digital presentation and functionality.

Martha went on to say that public sector had many initiatives that needed joining up, not into a major programme, but under a banner that allowed lots to participate and encouraged a movement of activity adding up to more than the sum of the parts.  This would be a good description of what we want to do with the release of data and information for re-use.

A busy week of engagement that was encouraging as I and colleagues across government make the changes that support the directions sought.  What we do is often the road-building for the Ferraris (I wish!) and transport lorries that support trade to run upon, but without it they wouldn’t move.  So onto to improving quality of what we do in a measurable way, demonstrating its value to the nation, and structuring information for others to use and innovate. 

Read the full article (2 Comments)

In the current climate of open, transparent and accountable government, it is now mandatory for government websites to have stats audits. But how did this come about and why is it beneficial?

Policy background

Back in July ’06 the National Audit Office published the results of its survey of Government on the Internet. The results were pretty shocking:

Over a quarter of government organisations still do not know the costs of their websites, making it impossible to assess whether they are value for money

16% of government organisations have no data about how their websites are being used, inhibiting website improvements.

The quality of government websites has improved only slightly since 2002.

These findings were used as evidence before the UK Parliament’s Public Accounts Committee (PAC) hearing in November ’07. PAC recommended the development of a single set of measures for government website costs, quality and usage which were to be reported centrally. Government’s response to the PAC Sixteenth Report was laid before the House of Commons in September ‘08.

Consistent data

The single set of measures was developed and is now in place, but how can the data be collected reliably? Measuring website usage can be done in a number of ways with sites using different methods, tools, standards, filters and terminology. To get consistency is a real challenge.

The media industry has solved this problem. Advertising revenue is based on the number of Ad Impressions – like Page Impressions but for ads – and rates vary with volume of site usage. Advertisers need a reliable way to ensure return on investment. They need to know that the websites on which they are buying space and surfacing content measure usage accurately and consistently. The solution is to insist on a site audit certificate.

Government websites don’t tend to generate revenue from advertising – although the practice is not forbidden in principle – but they are accountable to the taxpayer. Surely taxpayers have the right to expect a decent return on their investment? If I visit a government website, how much does it cost me? Is it value for money? I want to know!

The ABCe audit

In May 2009, COI appointed ABCe to be the sole auditor of government websites. ABCe is the industry owned website auditor and is the standard for the media industry, both for media owners and media buyers. COI has negotiated cost savings for the taxpayer by centralising the spend. The average cost of an audit is approximately £2,500 compared to £4,000 if departments went to ABCe independently. By the end of the financial year, all websites run by central government departments will have had one month’s usage data audited by ABCe.

The bigger picture

Why go to all this trouble and is there any benefit to the government departments themselves? Aside from increased accountability to the taxpayer, departments do stand to benefit from the increased rigour in site measurement and evaluation. Website audits are the first step towards properly managed performance improvement. It is only with consistent and reliable data that performance metrics – or KPIs – can be developed. These are things like:

• Average number of Visits per Unique User which measures how often a user returns to a website (customer loyalty)
• Average number of Page Impressions per Visit which provides a measure of user engagement (sometimes referred to as stickiness)

When usage levels are considered alongside costs, we can also begin to consider value for money metrics such as Cost per Visit.

Central reporting of quality data also enables benchmarking of government websites against each other. For example, if I get an average Visit Satisfaction of 70% for my website, how do I know if that is good or bad compared to other websites in my sector? With a standard set of core survey questions, this is now possible. It is also worth mentioning that local government are ahead of central government in this respect. Because of initiatives like the SOCITM Website Take-up Service and Gov Metric, Local Authorities have integrated satisfaction benchmarking into their site performance management.

Monitoring KPIs over time is a key business tool for demonstrating performance improvement which is so important for getting the appropriate level of investment in government digital media.

Central reporting of Visit Duration is a contentious issue. While it is probably not useful to compare websites on this metric – a long time on site may indicate a high level of engagement or a site that is difficult to navigate – it does provide interesting census-level data. Measuring Visit Duration enables Government to calculate the total amount of time spent on its websites by citizens. We can begin to get a picture of the value delivered to citizens by government online. For example, if we compare the cost of delivery to the cost for the citizen then we can begin to address the cost-benefit of online services to the citizen. Now that would be interesting!

Read the full article (11 Comments)