The Government announced its intention to cut spending on websites by up to 50%. But we need to deliver a consistently excellent user experience on the government web estate. So the question is how can we manage usability budgets most effectively and efficiently?

Clearly the most necessary aspect of usability is to test with users from the target audience.  Where the product or service is live, then feedback will be given through their use and feedback.  What can we do to get things right much earlier on?  To get it as close to effective for the user as possible? What can we do once the product or service is live and we want to check improvements?

People often associate usability only with user testing in laboratories, which can be expensive if there are many iterations.  In this article I will describe some alternative lower cost usability evaluation methods and look at the pros and cons of each method.  These don’t replace testing with users.  They are tools that offer getting to a usable product earlier with more efficiency.

Task analysis

Task analysis involves taking the tasks a user would do on your website and mapping out the individual steps required to complete each task. This type of analysis often yields a number of sub-tasks required to complete each individual step and each sub-task may be broken down further and so on. This approach is known as hierarchical task analysis. The purpose is to identify any steps that are unnecessary or that may slow the user down in achieving their desired goal. The process of stripping out unnecessary steps and working out the best way to design a given task can lead to a highly optimised and efficient user experience.

Pros

• Relatively cheap
• Focussed on user actions
• It can be used both on existing websites but especially when designing new services to identify any obvious extra steps that may be removed.

Cons

• Can be complex depending on the task in question
• Can be lengthy to complete a thorough task analysis, e.g. a complex website with many different functions
• Doesn’t identify any new tasks that may be appropriate for the site to deliver.

Task analysis assumes you are offering the right services and helps to optimise them but doesn’t provide any insight into whether you are designing the right thing.  That comes from user feedback.

Online surveys

Online surveys are an inexpensive method of eliciting user feedback. Crucially they allow people to say what they think about their online experience immediately after the event and therefore provide a good indication of people’s true feelings. Surveys are used to establish site ratings (e.g. satisfaction, ease of use etc) and to assess why people are coming to a site (i.e. the user goal) and the extent to which they achieve the intended purpose of their visit (i.e. goal completion).

Pros

• Low cost
• Close to the experience
• Provides insight into who your users are and why they are coming to your site

Cons

• Subjective and therefore not as reliable as observing users
• Can be annoying to users
• Takes time to analyse results and make actionable recommendations for improvement

There’s an art to questionnaire design, so we advise getting a professional to do it testing with a few people before going live. Don’t ask unneccessary questions, every question should have a reason for being asked. A useful starting point is the set of core questions defined in the COI guidance on survey design.

Heuristic evaluation

Heuristic evaluation is a kind of expert review where a website (or any user interface) is evaluated against a set of established design heuristics (or rules of thumb). The most commonly used set is Nielsen’s Ten Usability Heuristics. However there are others, for example Schneiderman’s Eight Golden Rules and  Don Norman’s principles of design. An experienced usability practitioner will evaluate against a the most appropriate set for the type of interface being evaluated and, dependent on experience, may have a bespoke set that works best for them.

Heuristic evaluation is most effective when carried out on selected user tasks. As the evaluator works through the individual steps to complete a task, usability problems are recorded along with the heuristic or design principle that it breaches. Positive points may also be noted but on the whole the purpose of usability evaluation is to find problems and make recommendations for improvement.

Pros

• Quickly identifies potential usability problems
• Useful to spot obvious issues early on in the design process
• Can save money spent fixing problems further down the line

Cons

• Effectiveness depends on the experience of the expert
• Less reliable than user testing because different experts interpret heuristics differently

Heuristic evaluation can be as effective as user testing in identifying usability problems and cost a fraction of the price. If using this method try to get the most experienced testers you can afford and agree up front how each heuristic will be interpreted to avoid any misunderstandings later on.

Cognitive and pluralistic walkthroughs

Cognitive walkthrough is where an expert walks through a task associated with your website, putting themselves in the shoes of a typical user. At each step, the following questions are posed:

• Will the user know what to do?
• Is the correct option available?
• Is the option is available, will the user expect that it will yield the intended result?
• Will the user know that the intended result has been yielded?

Examining each step in this way, the expert notes successes and failures along the way. As with heursitic evaluation, cognitive walkthroughs are used early in the design process.

Pros

• Low cost
• Task focussed
• Quickly identifies usability problems early in the design process

Cons

• Doesn’t test with real users
• Relies on expert’s ability to put themselves in user’s shoes

Cognitive walkthroughs assume a well defined set of tasks and steps are available to carry out the evaluation.

Walkthroughs can also be performed along with users,  domain experts and developers in a group scenario. This is know as a pluralistic wakthrough and can help to identify further usability problems due to the different types of participant involved.

Standards and guidelines

Standards and guidelines such as the W3C Web Content Accessibility Guidelines have been designed to take the hard work out of web design. Following established good practice can be an easy way to avoid many of the potential pitfalls that could lead to a poor user experience. There’s no point spending money on user testing or heuristic evaluation when the basic minimum standards haven’t been applied. In a government context the COI web standards and guidelines and the Usability Toolkit are a useful starting point.

Pros

• Low cost
• Someone has done the hard work for you

Cons

• Can lead to a tickbox mentality where standards compliance is valued above user needs
• Not necessarily user-centred

Compliance with standards and guidelines can be checked using automated tools but it usually requires an expert to do a full conformance inspection.

Summary

The methods introduced in this article are not intended as a replacement for user testing. User testing is still the most reliable way to get real user insight and generate improvements. Use these methods where appropriate and in support of other analysis, design and evaluation activities.

 Local government has been comparing like for like to benchmark for many years.  It is more difficult in central government as the audiences and functions of websites vary considerably.  Some are for the public (Diectgov, NHS Choices), for businesses (businesslink.gov.uk) and for public sector workforces (civil servants, teachers, armed forces).  And others are for people to engage with a particular organisation in policy formation (corporate websites).  While yet others have a regulatory function (Ofgem).

It’s made harder by some public bodies doing lots of syndication, placing information onto other websites where people regularly go, and early adoption of re-usable information and data, so that people can present it in new ways.  Both these result in people using the information, but not being recorded as visiting the website to do so. So the cost per visit is not related in any way to the cost per use, nor indeed the value to the user.

The data makes for interesting reading if not immediate analysis.  The notes that reporting public bodies have attached should be read in conjunction with the data.  However, some things do jump out.

One that struck me was the variation in cost for hosting and infrastructure compared to usage.  One wouldn’t expect a bell-shaped curve, as some sites need to have much more resilience and security than others, but if a free competitive market was really in operation, then one would expect more clustering around a few price points.

The other areas of expenditure will vary more according to the stage of development.  This year you might spend a lot on design and build to bring it up to what people expect, next year perhaps very little. Per year websites tell you something, but are not the whole picture – but how much better than not having anything!  Having knowledge of the order of the cost is really helpful and, even taking out all those who didn’t get anything from their visit, the cost for each reach is very low compared to any alternative.

And we can begin to look at some other interesting aspects.  Does the spending made on testing and evaluation result in an increase in user satisfaction?  By releasing the data, we’re keen to see what use people make of it and how they can add insight into the variety of questions that arise. Ross Ferguson, for example,  has done a visualisation of the number of yearly unique visitors/browsers (a figure that not everyone could provide as it requires deduplicating across all the monthly data).

We know people are interested.  In the three and a bit days since publication, there have been 14,534 downloads of the PDF and 1,034 downloads of the CSV.  It’s been interesting too to follow the use of the official bit.ly tag that we tweeted and discover where people find where the data is.  We’re looking forward to seeing what people do with it.

In terms of numbers, 46 is a very small sample.  Next year all the open central government websites are due to report and it doesn’t seem sensible to issue a report simply listing the data as we have done this year.  (We will of course issue the data as a dataset.)  How people analyse the data will help shape the human-readable report.

Let us know what you’re doing, and what you’d like to see.

As part of its ongoing involvement in the development of digital policies and standards, COI has worked with the likes of BBC, RNIB, IBM, Lloyds-TSB and Nomensa to develop this standard.

The Government has already adopted PAS 78, through TG102 Delivering inclusive websites, requiring websites to have and accessibility policy detailing the process for maintaining website accessibility over time. BS8878 builds on PAS 78 and provides a comprehensive approach to delivering web accessibility.

It’s not a technical standard with a strict set of criteria for compliance. It’s a process standard dealing with organisational responsibilities for ensuring web accessibility and looking at the entire production process including:

• Procurement
• Design and evaluation
• Maintenance
• Managing feedback

It also covers topics such as the UK Equality Act 2010 and the use of personalisation to deliver inclusive websites.

The standard is not a replacement of existing web standards, namely the W3C Web Content Accessibility Guidelines (WCAG). Rather, it is meant to work alongside WCAG 2.0 and link to it where appropriate.

There is still time to get your comments in on the BSI website, so please tell us what you think. We know there’s still room for improvement.

The one that bugs me most is ‘content management system’ (CMS).  Two different noun qualifiers, but which qualifies which of the three?  From my experience they are content systems and management systems but rarely a system for managing the aggregated collection of content.  Let me explain.

Many of the larger CMS come from an enterprise background where the task was to get in control of the many documents wandering around.  They have a lot of functionality including nifty conversions from one format to another.  And good tracking of where these documents are.   They also often contain many tools for managing the workflow in their production, authoring, approving and publishing.  The last is usually done by moving to a pre-publication area with a publisher role making the final act of releasing to the intended audience.  Then Web came along and a functionality was added to send to a website.

There are also many content systems that store words, pictures and other media, in a structured way in a database and then publish to the Web by adding in (X)HTML coding and style sheets and, if the developer is smart, rules for displaying one or another type of content depending on the situation in the system or type of viewer.  These come from a background of Web publishing and although they do that well, they usually don’t offer tools that work across the database.

What we need in government though is the third – content management:

• ‘Give me all the content that does not contain an entry in a particular metadata field and list with the contact emails of the authors.’
• ‘List all the pages that have been viewed only by internal staff.’
• ‘Find all the content containing expression X and change to Y and add to a new metadata field’

And, most of all,

• ‘Extract out all the content according to these rules, structure them in this particular way and send to go off to another CMS’.

It’s all do-able but so much hard work at present.  We should have easy interfaces to manage content.  I care, because we have many talented digital media staff in government (maybe not enough, but they are certainly there), who waste much of their time and effort in struggling with getting content in and out and between CMS’s. Usually because there is so little functionality to manage content across the whole collection and because exchange between systems is so difficult.  There must be a much better way of doing this and so releasing government and public sector expertise to contribute to the public agenda.  I’m carrying a banner to all those providers of CMS and web publishing services to say ‘Free our digital media experts!’

By doing more information structuring and using common structures, as we’ve been introducing for jobs and consultations, and have been done for a long while in press releases and descriptions of documents, we should be able to build systems that easily exchange information when there are machinery of government changes and that make it easy to manage the content. (And, of course, add in semantic web coding in a supported way!)  Then we might be able to start describing systems as those that enable and facilitate content management.

30 people from Government executive agencies and non-departmental public bodies (NDPBs) came to find out about website auditing.

Attendees were given an introduction by Alex Butler, COI’s Transformational Strategy Director. Alex told how this work came about as the result of the National Audit Office (NAO) report Government on the Internet (July 2007). The NAO report found that:

• a quarter of government organisations did not know the costs of their websites
• 16% had no data about how much their websites were being used
• quality standards had only improved slightly since 2002

Alex explained how the audits form part of a package of improvements offered by COI including standard measures for website quality, value for money and usability. These measures form part of the requirement by the Public Accounts Committee for greater accuracy and transparency in how the Government manages its investment in digital media.

Those present then watched a short clip of COI Chief Executive Mark Lund’s presentation at the Internet Advertising Bureau event IAB Engage 2009. Mark describes the work COI is doing with the Cabinet Office and other parts of Government in setting standards and evaluating the cost-effectiveness government websites.

Richard Foan, ABCe’s Managing Director, then talked through the details of website auditing. Richard emphasised the need for standards in an industry where there are so many different methods, metrics and tools to choose from. He explained that ABCe are an independent non-profit organisation, auditing to industry standards. ABCe certify that government websites comply with the standards and measure website usage in a consistent way.

The question and answer session that followed was lively and yielded some useful actions:

• We are planning an FAQ page on the COI website based on the questions asked
• Technical guidance will be available from the planned FAQ page but also from this page: TG116a ABCe Audit (PDF, 541 kB) and Guide to Google Analytics Mirror Method (PDF, 157 kB)
• Follow up meetings will be arranged with those who requested them
• Slides from the event are available from this page (PDF, 1.5 MB)
• Access to the Government Website Database is available by contacting graham.phillips@coi.gsi.gov.uk

Thanks to all those who attended, I hope you found it useful. And thanks particularly to Linda who worked the floor extremely well and made a star entrance!

For those who missed the event, we are planning to do similar events in the future. But in the meantime hopefully this blog article gives a flavour of what took place.

But there was deviation into another meaning.  The same task was used by sailors to scan the horizon – systematically looking around it for shipping or land.  This gradually shifted to the question being asked – is there anything interesting to see?  And that’s the meaning we generally attribute to the verb nowadays.  A quick glance or skim over something to see if there is anything that grabs our attention.

Well why is this relevant to the usual subject covered by this blog?  Because we sometimes mistake when people are scanning or scanning.  And it is important in getting people to the information or service quickly.

People quickly scan (in the second sense) and recognise something familiar in search results lists.  So seeing something in a search results list that says it is on Directgov, for example, should signal to the reader that they can trust it as authoritative, that it will be easy to read (because written for them) and that it will point people to other relevant content to their need.

Building brand is important to enable people to quickly skim over something and determine what they choose.  It is also important in memorability.  That’s why the guidance insists on avoiding sub-domains and using directory structure.  In plain words, avoiding URLs of the form YAA.website.gov.uk and instead using website.gov.uk/YAA.  The human brain does something different in the two cases.

In the former, one registers the YAA (standing for Yet Another Acronym) and then tends to forget the rest.  If that is what we want people to do, then using instructions such as ‘Search for Change4Life’ is a more effective means.  We’re increasingly using that to focus people on major campaigns.

However, if we want people to find it easily on the Web, then saying you’ll find it on Directgov, or wherever, is more effective – and the form direct.gov.uk/YAA does exactly this.  It both uses and builds the brand.  And if it is really current, it will be on the home page.

Sub-domains may be the technical means to delivering services and packaging content into useful and manageable chunks as part of internal processes, but for marketing the directory structure is much more helpful to get people to what they’re looking for.

The first correction is in paragraph 31.

Old version

<div about="#this"
  typeof="foaf:Document"
  rel="dc:type" resource="[argot:Consultation]"
  >
 ...
</div>

Correct code

<div about="#this" typeof="foaf:Document">
  <span rel="dc:type" resource="[argot:Consultation]"></span>
...
</div>

Notice how the @rel attribute is in a separate span element.

There is a similar correction in paragraph 34.

Old version

<div about="#this"
  typeof="foaf:Document"
  rel="dc:type" resource="[argot:Consultation]"
 >
  <span property="dc:publisher" content="Ministry of Justice"></span>
 ...
</div>

or even merged onto the first element:

<div
 about="#this"
 typeof="foaf:Document"
 rel="dc:type" resource="[argot:Consultation]"
 property="dc:publisher" content="Ministry of Justice"
>
...
</div>

Correct code

<div about="#this" typeof="foaf:Document">
 <span rel="dc:type" resource="[argot:Consultation]"></span>
 <span property="dc:publisher" content="Ministry of Justice"></span>
...
</div>

or even merged onto the first element:

<div about="#this" typeof="foaf:Document"
 property="dc:publisher" content="Ministry of Justice">
 <span rel="dc:type" resource="[argot:Consultation]"></span>
...
</div>

Apologies for the changes. If you have any questions, please contact me at adam.bailin@coi.gsi.gov.uk.

We have an event coming up in London on the 21^st of January aptly named “Auditing Government Websites”.  Speaking at the event is our very own Alex Butler, COI Transformational Strategy Director, and ABCe’s Managing Director Richard Foan.  The event is aimed at government agencies (EAs) and non-departmental public bodies (NDPBs) and will cover the following topics:

• importance of auditing
• industry standards for measuring websites
• how to get ready for the audit
• the audit process (pre audit and post audit)

The event provides an opportunity for government agencies and NDPBs to audit their websites for approximately half the amount if procured independently. What better way to start the year? The event is free and registration for the event and the audits are pouring in, so register now and make your websites 2010 compliant.

Please email me at linda.morakinyo@coi.gsi.gov.uk if you would like to attend.

Hope to see you all at there.

Invitation to auditing government websites event 21 Jan 2010

The Web is basically a document standard – a description of what constitutes a Web page, together with a process for describing it’s location (the URL) and so of linking from one to another.  When you do a Web search, for example using Google or Bing, then you get a list of documents in which the information you seek might be in. 

A URI enables a unique way to identify a particular bit of data inside the Web page, and so link one bit to another.  Thus it might be useful to distinguish London-the-place from the other London-the-places and from the several authors with surname London.  We can get some way towards this by intelligent contextual analysis, the approach that Microsoft, for example, told me they are taking.  This involves heavyweight data crunching using search technologies.  The URI approach is to identify something as distinctive,  for example, London the place in this particular geospatial location, and then give it a URI that others can use to refer to it to disambiguate it from all other occurrences of the concept or word.

This is the core idea of a URI, that a place, event, person, concept, document, or whatever can be given a unique identifier that others can use.  Of course you need to do something more than that, as Sir Tim Berners-Lee describes in his four steps:

1.  Use URIs as names for things
2. Use http URIs so that people can look up those names
3. When someone looks up a URI, provide useful information using the W3C standards (RDF, SPARQL)
4. Include links to other URIs so that they can discover more things

 I usually add one more:

5.  the provider of a set of URIs provides a Lookup service to take the object being named and provide a URI for it (i.e. the converse of 2.)

 So what would be useful for government to do?  One fruitful area to explore are those things that come and go, or move around, or change.  For example MPs get appointed to serve in HM Government and then move around.  Giving each MP a URI so that every time a press release reports their activities would be helpful, particularly as they are often described in different ways.  Clicking on a URI link could take you to a page of information about them – for example their biography, committees they serve on etc, or, with a little macro on the side, a set of relevant links about them.  And then there might be URIs for Departments.  They come and go – when were they in existence?  What were their responsibilities?  Is there archived content about them?  What is the current list of Departments? That kind of information we know would be useful to provide, as we get asked for them

Those are two examples of sets of URIs that government could usefully run:  the MP names, and the list of Departments.  Another might be the roles that comprise HMGovernment, i.e. the Ministers.  Clearly at local government level the set of Local Authorities would be one that would be useful, so that one person referring to a public body would know it was the same one that another called by a different name or abbreviation. 

The government has developed a draft standard for designing sets of URIs and we are now exploring what core sets of URIs it would  be useful to provide.  Let us know and we’ll see if we can do so.

Evaluation is a key priority for Government communicators. Matt Tee, Permanent Secretary and Head of Profession Government Communications, has prioritised evaluation as a key area of focus, along with skills and behaviour change. This is not surprising given the current economic climate. Government has to account for every pound spent and that means evaluating our communications activity to demonstrate cost-effectiveness.

Matt Tee has also requested that every government department develop a digital engagement strategy by March 2010, alongside the Public Accounts Committee recommendation that every department has a channel strategy, – a sign that digital engagement is being taken seriously.

So, how do we measure it?

As with any marketing communications activity, that depends on what the communications objectives are. However, there are commonalities across different campaigns and across different digital engagement tools and it’s those that I want to explore.

Recently, I’ve been working with colleagues in COI on this problem and we’ve come up with three common measures that appear to work across all digital engagement or social media tools:

1. Number of relationships
2. Number of user-generated content items
3. Number of referrals/recommendations

1. Number of relationships

The number of relationships or connections within a network is a measure of power or influence. For example, it could be the number of followers on Twitter, number of friends in Facebook or the number of subscribers to a blog. In social network analysis, this is the basic measure of centrality within a network, which is called degree centrality.

There are other interesting measures of power within a network. For example betweenness centrality measures the degree to which a member lies between other members of a network. In the Facebook analogy, a person may have 1000 friends but have less influence than a person with 50 friends, each of whom have 1000 friends.

Betweenness centrality from red (lowest) to blue (highest)

2. Number of user-generated content items

The number of user-generated content items measures participation within the network. For example, it could be the number of comments on a blog or the number of videos uploaded to a Youtube channel. It measures the level of engagement of an audience, suggestive of active participation not simply passive interest.

3. Number of referrals/recommendations

The number of recommendations is what many seek. This measures virality, advocacy, recommendability. For example, it could be the number of retweets, the number of  ‘share this’ actions or the number of pingbacks. It goes beyond mere participation; it means your content or message is valued enough to be recommended to others inside and outside the network.

We would be very interested to hear any thoughts on this. Many people are starting to think through return on investment in this area and it would be useful to have some level of consensus before applying to the government’s use of digital media for engagement.     Let us know what you think.