Task performance measurement is a way to benchmark how easily and how quickly our audiences can complete what they came to do (their tasks) on the Commission's websites. The tests result in an indicator known as the task performance indicator (TPI).

Using task performance measurement as a continuous-improvement management model offers:

Methodology

Task performance measurement involves recording and analysing 15-20 one-on-one remote sessions (about 60 minutes) in which representative users try to complete a prioritised set of tasks.

Preparation: task instructions for testing

TPI testing involves asking participants to complete specific tasks that have been identified as most important. The instructions are real task scenarios and framing them correctly is essential to the success of the measurement process. The following detailed criteria are used to develop instructions: 

  1. Repeatable
  2. Representative and typical for the task
  3. Universal—everyone can do it
  4. One unique answer
  5. Does not contain hidden clues
  6. Not search-engine optimised
  7. Emotionally neutral
  8. Not confidential
  9. Clearly different from other instructions
  10. Independent of other instructions (i.e. not linked to another instruction in the test)
  11. Can be performed directly on website (does't require prior log-in)
  12. Short – 30 words or less
  13. Stable – no change within the testing period

Examples:

Task referenceTest instruction

Top task 1 (EU law, rules, treaties, judgments)

Find the current EU law governing investigations by the European Anti-Fraud Office.
Top task 2 (Research and innovation)Find the page listing potential research and innovation project partners in the field of robotics in order to share knowledge.
Top task 3 (Funding, grants, subsidies)

Find the opening date for the current opportunity for small businesses (SMEs) to submit their proposals in the field of healthcare biotechnology in Horizon 2020.

Top task 5 (EU strategy, political priorities)How many infrastructure and innovation projects were approved for Estonia as of [choose recent date] under the European Fund for Strategic Investments?

In the Commission, task instructions are developed with the relevant DGs. Once a task instruction is agreed, each step in the navigation path that leads to the correct answer is documented in the form of URLs and screen shots.

Although the instructions from the baseline should be used for subsequent iterations, they are nevertheless reviewed with stakeholders before each iteration.


Remote tests

Remote testing is both representative and real because participants are in their own environment and on their own computers. Screen interactions and speech are recorded (with the participant's consent). The time it takes for the test participant to complete each task is measured from the moment they understand it and start working on it until they come up with a definitive answer or give up. 

Each participant starts the test from the same starting page. To ensure that results remain comparable over successive iterations, the starting page needs to be repeatable. In the Commission, therefore, the agreed starting page is the Commission homepage (ec.europa.eu/index_en.htm). Test participants are explicitly told that, although the test starts from this page, they are free to use the internet as they would usually do (Google, Bing, etc.).

Analysis and reporting

Analysis involves identifying:

Reporting involves compiling actionable findings on navigation, search, design, content for the class and per instruction, including annotated screen shots.

Video recordings are available per participant and per task instruction.

TPI test reports
Baseline reports are available for the following content classes: