Today is World Usability Day and to mark the occasion, we here at Digigov have developed an introduction to low cost usability methods that might be useful to you in these austere times.

The Government announced its intention to cut spending on websites by up to 50%. But we need to deliver a consistently excellent user experience on the government web estate. So the question is how can we manage usability budgets most effectively and efficiently?

Clearly the most necessary aspect of usability is to test with users from the target audience.  Where the product or service is live, then feedback will be given through their use and feedback.  What can we do to get things right much earlier on?  To get it as close to effective for the user as possible? What can we do once the product or service is live and we want to check improvements?

People often associate usability only with user testing in laboratories, which can be expensive if there are many iterations.  In this article I will describe some alternative lower cost usability evaluation methods and look at the pros and cons of each method.  These don’t replace testing with users.  They are tools that offer getting to a usable product earlier with more efficiency.

Task analysis

Task analysis involves taking the tasks a user would do on your website and mapping out the individual steps required to complete each task. This type of analysis often yields a number of sub-tasks required to complete each individual step and each sub-task may be broken down further and so on. This approach is known as hierarchical task analysis. The purpose is to identify any steps that are unnecessary or that may slow the user down in achieving their desired goal. The process of stripping out unnecessary steps and working out the best way to design a given task can lead to a highly optimised and efficient user experience.

Pros

• Relatively cheap
• Focussed on user actions
• It can be used both on existing websites but especially when designing new services to identify any obvious extra steps that may be removed.

Cons

• Can be complex depending on the task in question
• Can be lengthy to complete a thorough task analysis, e.g. a complex website with many different functions
• Doesn’t identify any new tasks that may be appropriate for the site to deliver.

Task analysis assumes you are offering the right services and helps to optimise them but doesn’t provide any insight into whether you are designing the right thing.  That comes from user feedback.

Online surveys

Online surveys are an inexpensive method of eliciting user feedback. Crucially they allow people to say what they think about their online experience immediately after the event and therefore provide a good indication of people’s true feelings. Surveys are used to establish site ratings (e.g. satisfaction, ease of use etc) and to assess why people are coming to a site (i.e. the user goal) and the extent to which they achieve the intended purpose of their visit (i.e. goal completion).

Pros

• Low cost
• Close to the experience
• Provides insight into who your users are and why they are coming to your site

Cons

• Subjective and therefore not as reliable as observing users
• Can be annoying to users
• Takes time to analyse results and make actionable recommendations for improvement

There’s an art to questionnaire design, so we advise getting a professional to do it testing with a few people before going live. Don’t ask unneccessary questions, every question should have a reason for being asked. A useful starting point is the set of core questions defined in the COI guidance on survey design.

Heuristic evaluation

Heuristic evaluation is a kind of expert review where a website (or any user interface) is evaluated against a set of established design heuristics (or rules of thumb). The most commonly used set is Nielsen’s Ten Usability Heuristics. However there are others, for example Schneiderman’s Eight Golden Rules and  Don Norman’s principles of design. An experienced usability practitioner will evaluate against a the most appropriate set for the type of interface being evaluated and, dependent on experience, may have a bespoke set that works best for them.

Heuristic evaluation is most effective when carried out on selected user tasks. As the evaluator works through the individual steps to complete a task, usability problems are recorded along with the heuristic or design principle that it breaches. Positive points may also be noted but on the whole the purpose of usability evaluation is to find problems and make recommendations for improvement.

Pros

• Quickly identifies potential usability problems
• Useful to spot obvious issues early on in the design process
• Can save money spent fixing problems further down the line

Cons

• Effectiveness depends on the experience of the expert
• Less reliable than user testing because different experts interpret heuristics differently

Heuristic evaluation can be as effective as user testing in identifying usability problems and cost a fraction of the price. If using this method try to get the most experienced testers you can afford and agree up front how each heuristic will be interpreted to avoid any misunderstandings later on.

Cognitive and pluralistic walkthroughs

Cognitive walkthrough is where an expert walks through a task associated with your website, putting themselves in the shoes of a typical user. At each step, the following questions are posed:

• Will the user know what to do?
• Is the correct option available?
• Is the option is available, will the user expect that it will yield the intended result?
• Will the user know that the intended result has been yielded?

Examining each step in this way, the expert notes successes and failures along the way. As with heursitic evaluation, cognitive walkthroughs are used early in the design process.

Pros

• Low cost
• Task focussed
• Quickly identifies usability problems early in the design process

Cons

• Doesn’t test with real users
• Relies on expert’s ability to put themselves in user’s shoes

Cognitive walkthroughs assume a well defined set of tasks and steps are available to carry out the evaluation.

Walkthroughs can also be performed along with users,  domain experts and developers in a group scenario. This is know as a pluralistic wakthrough and can help to identify further usability problems due to the different types of participant involved.

Standards and guidelines

Standards and guidelines such as the W3C Web Content Accessibility Guidelines have been designed to take the hard work out of web design. Following established good practice can be an easy way to avoid many of the potential pitfalls that could lead to a poor user experience. There’s no point spending money on user testing or heuristic evaluation when the basic minimum standards haven’t been applied. In a government context the COI web standards and guidelines and the Usability Toolkit are a useful starting point.

Pros

• Low cost
• Someone has done the hard work for you

Cons

• Can lead to a tickbox mentality where standards compliance is valued above user needs
• Not necessarily user-centred

Compliance with standards and guidelines can be checked using automated tools but it usually requires an expert to do a full conformance inspection.

Summary

The methods introduced in this article are not intended as a replacement for user testing. User testing is still the most reliable way to get real user insight and generate improvements. Use these methods where appropriate and in support of other analysis, design and evaluation activities.

Read the full article (2 Comments)

I am really proud of my government colleagues for getting the data together to enable us to publish the very first report on the costs, quality and usage of central government websites. You can find it, along with all the data, on www.coi.gov.uk/websitemetrics2009-10

 Local government has been comparing like for like to benchmark for many years.  It is more difficult in central government as the audiences and functions of websites vary considerably.  Some are for the public (Diectgov, NHS Choices), for businesses (businesslink.gov.uk) and for public sector workforces (civil servants, teachers, armed forces).  And others are for people to engage with a particular organisation in policy formation (corporate websites).  While yet others have a regulatory function (Ofgem).

It’s made harder by some public bodies doing lots of syndication, placing information onto other websites where people regularly go, and early adoption of re-usable information and data, so that people can present it in new ways.  Both these result in people using the information, but not being recorded as visiting the website to do so. So the cost per visit is not related in any way to the cost per use, nor indeed the value to the user.

The data makes for interesting reading if not immediate analysis.  The notes that reporting public bodies have attached should be read in conjunction with the data.  However, some things do jump out.

One that struck me was the variation in cost for hosting and infrastructure compared to usage.  One wouldn’t expect a bell-shaped curve, as some sites need to have much more resilience and security than others, but if a free competitive market was really in operation, then one would expect more clustering around a few price points.

The other areas of expenditure will vary more according to the stage of development.  This year you might spend a lot on design and build to bring it up to what people expect, next year perhaps very little. Per year websites tell you something, but are not the whole picture – but how much better than not having anything!  Having knowledge of the order of the cost is really helpful and, even taking out all those who didn’t get anything from their visit, the cost for each reach is very low compared to any alternative.

And we can begin to look at some other interesting aspects.  Does the spending made on testing and evaluation result in an increase in user satisfaction?  By releasing the data, we’re keen to see what use people make of it and how they can add insight into the variety of questions that arise. Ross Ferguson, for example,  has done a visualisation of the number of yearly unique visitors/browsers (a figure that not everyone could provide as it requires deduplicating across all the monthly data).

We know people are interested.  In the three and a bit days since publication, there have been 14,534 downloads of the PDF and 1,034 downloads of the CSV.  It’s been interesting too to follow the use of the official bit.ly tag that we tweeted and discover where people find where the data is.  We’re looking forward to seeing what people do with it.

In terms of numbers, 46 is a very small sample.  Next year all the open central government websites are due to report and it doesn’t seem sensible to issue a report simply listing the data as we have done this year.  (We will of course issue the data as a dataset.)  How people analyse the data will help shape the human-readable report.

Let us know what you’re doing, and what you’d like to see.

Read the full article (10 Comments)

The British Standard for Web Accessibility BS8878 has been published as a second draft and is available for comments until 30 June 2010.

As part of its ongoing involvement in the development of digital policies and standards, COI has worked with the likes of BBC, RNIB, IBM, Lloyds-TSB and Nomensa to develop this standard.

The Government has already adopted PAS 78, through TG102 Delivering inclusive websites, requiring websites to have and accessibility policy detailing the process for maintaining website accessibility over time. BS8878 builds on PAS 78 and provides a comprehensive approach to delivering web accessibility.

It’s not a technical standard with a strict set of criteria for compliance. It’s a process standard dealing with organisational responsibilities for ensuring web accessibility and looking at the entire production process including:

• Procurement
• Design and evaluation
• Maintenance
• Managing feedback

It also covers topics such as the UK Equality Act 2010 and the use of personalisation to deliver inclusive websites.

The standard is not a replacement of existing web standards, namely the W3C Web Content Accessibility Guidelines (WCAG). Rather, it is meant to work alongside WCAG 2.0 and link to it where appropriate.

There is still time to get your comments in on the BSI website, so please tell us what you think. We know there’s still room for improvement.

Read the full article (3 Comments)

A big thank you and well done to everyone who contributed to a great event last Thursday.

30 people from Government executive agencies and non-departmental public bodies (NDPBs) came to find out about website auditing.

Attendees were given an introduction by Alex Butler, COI’s Transformational Strategy Director. Alex told how this work came about as the result of the National Audit Office (NAO) report Government on the Internet (July 2007). The NAO report found that:

• a quarter of government organisations did not know the costs of their websites
• 16% had no data about how much their websites were being used
• quality standards had only improved slightly since 2002

Alex explained how the audits form part of a package of improvements offered by COI including standard measures for website quality, value for money and usability. These measures form part of the requirement by the Public Accounts Committee for greater accuracy and transparency in how the Government manages its investment in digital media.

Those present then watched a short clip of COI Chief Executive Mark Lund’s presentation at the Internet Advertising Bureau event IAB Engage 2009. Mark describes the work COI is doing with the Cabinet Office and other parts of Government in setting standards and evaluating the cost-effectiveness government websites.

Richard Foan, ABCe’s Managing Director, then talked through the details of website auditing. Richard emphasised the need for standards in an industry where there are so many different methods, metrics and tools to choose from. He explained that ABCe are an independent non-profit organisation, auditing to industry standards. ABCe certify that government websites comply with the standards and measure website usage in a consistent way.

The question and answer session that followed was lively and yielded some useful actions:

• We are planning an FAQ page on the COI website based on the questions asked
• Technical guidance will be available from the planned FAQ page but also from this page: TG116a ABCe Audit (PDF, 541 kB) and Guide to Google Analytics Mirror Method (PDF, 157 kB)
• Follow up meetings will be arranged with those who requested them
• Slides from the event are available from this page (PDF, 1.5 MB)
• Access to the Government Website Database is available by contacting graham.phillips@coi.gsi.gov.uk

Thanks to all those who attended, I hope you found it useful. And thanks particularly to Linda who worked the floor extremely well and made a star entrance!

For those who missed the event, we are planning to do similar events in the future. But in the meantime hopefully this blog article gives a flavour of what took place.

Read the full article (6 Comments)

This week has been one of conferences.  On Tuesday was the Public Sector Information annual conference .  It is amazing to think that it is just one year ago that, working with John Sheridan, I presented an overview of how data and structured information could be released using semantic web markup.  Since then London Gazette has been released in RDF/XML and people across government are busy implementing RDFa for consultations and in the public sector RDFa for jobs (for example Jobs Go Public’s local government jobs – LGJobs), the last two to be surfaced through Directgov.

More importantly, well underway is the Prime Minister’s drive for the release of data and creation of a single point of access (currently under development) through the appointment of Sir Tim Berners-Lee and Professor Nigel Shadbolt.   As the latter pointed out at the conference this is a single point of access not a single data base – the data will still sit in Departments, agencies and local government websites, but developers will be able to know what is available through some kind of searchable catalogue and get access to it.

He showed us a newspaper for a single postcode that had been demonstrated by some mash-up developers.  This included local data on crime, allotments, bus-stops and routes completely localised, along with lots of other useful information. We all liked it, because at present that data is something that each of us has to build up ourselves and we respond to it immediately as valuable.

The next day I spoke at the Public Sector Online annual conference organised by Kable.  The subject I was given was the cost of websites and I took the opportunity to remind all national and local government webbies that we need to be able to justify the expense of websites and demonstrate their value at this time of financial stringency.  In fact we have a great story to tell relative to many channels of communication, but I am guessing that the Finance Directors don’t yet always see that value and fund invest accordingly.

We’ve issued the standards on improving quality by measuring cost, usage and user satisfaction, following the Public Accounts Committee recommendations.  When these are reported people can start identifying lots of interesting aspects, answering the question, for example, of the value of the website channel to the nation.  Net value could be determined by the total online cost for satisfied minus unsatisfied users and subtracting the overall cost of provision. What we’d like to do is make the data available so that academics and economists can study this in more detail.

As a presenter it felt good to be getting email and comment from the audience floor which I was able to see shortly afterwards and makes a response.  The next conference on the Thursday was created to facilitate this – Government 2010.  Although I was invited to participate in person, I did so by logging in and watching the webstream. Lots of interesting thoughts, some of them inspired by Tom Steinberg’s contribution. 

It brought to mind a presentation from Martha Lane-Fox to COI on the Wednesday late afternoon, when she was asked the question about her experience in working in the public sector as the Champion for Digital Inclusion.  She responded that she had been really impressed by the calibre, intelligence and quality of the people she had met. 

It struck me that there are many talented expert e-communicators across government but hampered by the misperception that Web is IT.  There is an infrastructural element, but too often Web publishing is run as IT processes without the flexibility to change things day by day or initiate new trials or innovate.  We cannot even as I did in the old days, run two versions of a website in parallel and monitor what people do and continuously develop the more successful. 

It is like newspaper editors being unable to change the front page and only being able to stream new text into exactly the same shape of story, without being able to put in a major picture or give over the front page to a single story.  Newspapers and magazines would be very boring without that.  Likewise Web publishing should enable flexibility of all kinds of digital presentation and functionality.

Martha went on to say that public sector had many initiatives that needed joining up, not into a major programme, but under a banner that allowed lots to participate and encouraged a movement of activity adding up to more than the sum of the parts.  This would be a good description of what we want to do with the release of data and information for re-use.

A busy week of engagement that was encouraging as I and colleagues across government make the changes that support the directions sought.  What we do is often the road-building for the Ferraris (I wish!) and transport lorries that support trade to run upon, but without it they wouldn’t move.  So onto to improving quality of what we do in a measurable way, demonstrating its value to the nation, and structuring information for others to use and innovate. 

Read the full article (2 Comments)

Today the official accessibility guidance was updated to allow a choice of either WCAG 1.0 or WCAG 2.0 as the minimum standard for public sector websites:

Compliance with the Web Content Accessibility Guidelines is acceptable at Level Double-A of version 1.0 or the equivalent level in version 2.0. Future policy and timetables for implementation will align with European Commission recommendations on the adoption of WCAG 2.0. Planned future updates to this guidance will include details of the specific conformance requirements for version 2.0.

- Paragraph 5, Delivering inclusive websites (TG102)

My last post on accessibility suggested it would be a good first step. The suggestion was generally supported, not least by Jack Pickard, who since his initial response has written a more detailed article. In this, he sets out what he thinks the appropriate level of WCAG 2.0 should be for public sector websites. It’s an excellent article and will certainly help COI to develop the accessibility guidance further and smooth the transition to version 2.0. If you have a view on this, I urge you to read Jack’s article and comment.

Read the full article (6 Comments)