Last week, Roo Reynolds wrote about how we’ve been improving site search on GOV.UK. Throughout the past couple of months we’ve been analysing our search data to find out more about how people are searching the site. Here are some of the most interesting things we’ve learned.
Getting people to what they need
First of all, we looked at everything we already knew from the first 6 months since GOV.UK was released back in October 2012. About 4% of visitors use the site search, making more than 2 million searches a month.
We regularly review the most popular search terms and check how they’re performing - particularly when people refine their search or leave the site. ‘Refinements’ show how many users are changing their search term and trying again, usually to make their results more specific. Overall, the average refinement rate was 27% - we’d like to get that down so that more people are finding what they want first time.
‘Exits’ show how many users are leaving the site straight from a search results page, rather than clicking on a link within the site. The average exit rate was 15%, but this includes clicks on external links to other useful websites for information that isn’t covered by GOV.UK, such as NHS Choices and the National Careers Service. However, if searches without external links have high exits, it suggests that those results aren’t relevant enough so users are going elsewhere.
Inside Government searches
Around 20% of site searches now start from an Inside Government page (the corporate part of the site with government policy, announcements and publications). The rest come from the services and information for citizens and businesses or from the GOV.UK homepage.
A lot of these Inside Government searches are made from the main department or organisation pages. This may be partly because department websites have recently moved to GOV.UK so regular users are finding their way around the new content. But we’ve also seen in user testing that some members of the public still think in terms of the department responsible for their needs - for example, they’ll go to the Home Office or HM Passport Office to renew their passport, though these pages are aimed mainly at 'professional' users (those whose work involves research, the detail of policies or the workings of government).
Many of the top Inside Government search terms are general needs that are also in the overall most common searches, for example: ‘jobs’, ‘visa’, ‘passport’, ‘careers’, ‘ehic’, ‘travel advice’.
This crossover presents a challenge for site search: how do we meet the advanced needs of professional users without confusing members of the public who just want a simple answer? We can’t rely on the page they searched from to define which type of user they are; some people expect to search only within that department, but others have landed in the wrong place and need to find the general results. One of our priorities for this project was to start making search better for advanced users, without getting in the way of less experienced users.
The long tail of search
One of the first questions from the search team was: ‘how big is the long tail?’ Out of the 2 million site searches a month, about 800,000 different search terms are used.
We sliced up a month’s data by filtering on the number of searches: first the top 100, 200 and 1,000 search terms (that is, 101-200 and 201-1,000), which were used about 100 times or more, and then everything searched for more than 50 times, 20 times, and so on.
At one extreme, the word ‘jobs’ alone makes up 1.5% of all our searches, with almost 30,000 a month. The 100 most commonly used search terms (including ‘jobs’) account for 13% of all searches; the top 1,000 search terms combined make up 28% of searches.
At the other end of the scale, more than half a million different search queries were only made once in the whole month - that’s 34% of searches, but 84% of search terms.
We also found that refinement rates increase considerably as you go further down the tail. In the top 100 search terms, most of which are quite specific and well optimised, only 14% of users searched again. In the middle groups, the percentages rose through the 20s, and in the large set of more complex search queries that were only entered once, 34% of users changed their search terms and tried again.
Word count of search terms
We also analysed the length of site search terms, and found that most people use short queries of up to 5 words - half of all searches are 2 or 3 words.
These tend to be simply the name of the thing they need to find, for example: ‘passport’, ‘student finance’, ‘working tax credit’, ‘employment and support allowance’. Sometimes they include the action people want to take, for example: ‘find a job’, ‘renew child passport’, ‘book practical driving test’.
As queries get longer, they start to get more specific, for example: ‘driving licence out of date’, ‘returning to work after maternity leave’, ‘change the date of my practical test’, ‘how long does it take to get a passport’.
Some users enter very detailed queries describing their personal situation, for example: ‘I have a British passport and I had a British driving licence but I have lived in Spain for the last 10 years and now have a Spanish licence, I am returning to UK and need to get a British licence’.
We found that search exits (users leaving the site) increased with longer queries, rising steadily from 14% for 1 word up to 20% for 9 words or more. Interestingly, refinements (searching again) didn’t follow the same pattern; they rose slightly from 28% to a peak of 31% for 5-word queries, but then dropped back down to 27% for the longest queries.
Why does all this matter?
We’ve been improving the search algorithm, to make it better at matching exact phrases or words that appear close together, and to reduce irrelevant results that only match some of the words. Knowing the most common types of searches helped our developers decide where to focus their efforts to make the most effective improvements.
Now that we’ve released these changes, we’ll continue to monitor and analyse search performance to see how much difference they’ve made and to prioritise further development work.
7 comments
Comment by The GOV.UK homepage – our latest version | Government Digital Service posted on
[…] GOV.UK homepage use the search box to navigate the site, and we’ve recently been spending time improving the performance of site search, so we’ve increased the prominence of the search box on the […]
Comment by Tony Russell-Rose posted on
Nice post, Tara. Although I don't understand your reasoning in the 3rd para: "Overall, the average refinement rate was 27% – we’d like to get that down so that more people are finding what they want first time".
In my experience (and many published studies - e.g. see the references at http://isquared.wordpress.com/2011/10/11/findability-is-just-so-last-year/), a large proportion of users enter sites like gov.uk either not knowing precisely what they want, or not knowing how to articulate it (or both). In this context, query refinement is a simple but vital strategy for gaining a better understanding of the information space, and as such is something we should embrace and support rather than eliminate. So I don't understand why you interpret it as dysfunctional - could you elaborate a little pls?
Comment by David Dinsdale posted on
Interesting blog, thanks. Does the search algorithm you have developed use meaning or words? For example, would it recognise both 'find a job' and 'search for work' as the same query before then looking for content that was applicable?
Comment by Jan Ford posted on
Fascinating and as a regular visitor I appreciate the intent to continuously improve. Just wondered.....we tweet as departments such as the Home Office-does that nudge users to search that way? Should we consider tweeting as citizen services such as 'driving licence' too?
Comment by peterbjordan posted on
Interesting question. The chief organisations that people search for (navigational searches) are DVLA and HMRC. R social media, I think you can think of it as two facets - the organisation @ukhomeoffice and the topic or subject - that you could hashtag - as i see you ar doing re #glastonbury
Comment by John McC posted on
Good work, and keep it going. I hope you are analysing the full spectrum of information discovery and retrieval (e.g. search, browsing, asking, subscribing, etc). This excellent presentation by Peter Morville, who wrote might be useful - http://ilg.so/19liSRk
Comment by John McC posted on
Oops, forgot the reference. Peter Morville's book is Search Patterns: Design for Discovery (ISBN-10: 0596802277). The Information Architecture presentation follows on from work done on the book.