Discover

Build a greater understanding of your problem and the people it impacts.

Cognitive walkthrough

What

An evaluation method in which people work through a set of representative tasks and ask questions about the task as they go.

Why

To get quick and early feedback on whether a design solution is easy for a new or infrequent user to learn, and why it is or isn’t easy. This method is useful for catching big issues at any stage in the design process when you don’t have access to real users, but it is not a substitute for user evaluation.

Time required

30 minutes to one hour per person

How to do it

  1. Identify specific traits for new or infrequent users of a design solution.
  2. Develop a set of representative tasks that emphasize new use or infrequent use.
  3. Designate a member of the design team to play the role of a user. That person will use the traits you’ve identified to participate in a moderated usability testing session. (The traits can overlap.)
  4. Ask the user to accomplish their goal using a printed or interactive design. As they go, ask what they would attempt to do next or how they would learn.
  5. Don’t lead the user through the task, but encourage them to stay focused on what they’re trying to accomplish.
  6. Pay attention to expected outcomes and how quickly/easily participants are able to pick up a task.
  7. Analyze the walkthrough results to highlight where the user struggled and what needs improvement.

Considerations for use in government

No PRA implications. The PRA explicitly exempts direct observation and non-standardized conversation (e.g., not a survey) that a cognitive walkthrough entails, 5 CFR 1320.3(h)3.

If you are not working with government employees, you will need to observe standard precautions for archiving personally identifiable information.

18F

Contextual inquiry

What

The product team unobtrusively observes participants at work, with their permission, then asks questions.

Why

To learn how and why users do what they do; to discover needs and attitudes that might not emerge in an interview to map how tools, digital and otherwise, interact during complex activities.

Time required

1-2 hours per user

How to do it

  1. With permission from a supervisor and from the participant, schedule a time to watch a typical work activity and record data.
  2. While observing, ask the participant to act normally. Pretend you’re a student learning how to do the job. Ask questions to help you understand what the person is doing and why.
  3. At the end of the session, explain what you have learned and check for errors.
  4. Immediately after, write up your notes.

Example

A pair of 18F team members visited two Department of Labor/Wage Hour Division investigators as they interviewed home health care workers who were subject to unpaid overtime and other infractions. Since it was a sensitive subject, the 18F team did not question the health care workers directly, but instead asked the investigators clarifying questions in private. 18F staff also made sure that photos did not include faces.

Additional resources

Considerations for use in government

No PRA implications, if done properly. Contextual interviews should be non-standardized, conversational, and based on observation. The PRA explicitly exempts direct observation and non-standardized conversation, 5 CFR 1320.3(h)3. See the methods for Recruiting and Privacy for more tips on taking input from the public.

For internal folks, get permission from the right level of management. If participants could be under union agreements, contact the agency’s labor relations team.

18F

Design studio

What

An illustration-based way to facilitate communication (and brainstorming) between a project team and stakeholders.

Why

To create a shared understanding and appreciation of design problems confronting the project team.

Time required

3–4 hours

How to do it

  1. Invite between six and 12 participants: stakeholders, users, and team members who need to build a shared understanding. Before the meeting, share applicable research, user personas (unless users will be present), and the design prompt for the exercise.
  2. Bring drawing materials. At the start of the meeting, review the design prompt and research you shared.
  3. Distribute drawing materials. Ask participants to individually sketch concepts that address the prompt. Remind them that anyone can draw and artistic accuracy is not the goal of the exercise. 15–20 minutes.
  4. Have participants present their ideas to one another in groups of three and solicit critiques.
  5. Ask the groups to create a design that combines the best aspects of members’ individual contributions.
  6. Regroup as a whole. Have each group of three present their ideas to everyone. Discuss.
  7. After the meeting, note areas of consistent agreement or disagreement. Incorporate areas of consensus into design recommendations and areas of contention into a research plan.

Example from 18F

Considerations for use in government

No PRA implications. If conducted with nine or fewer members of the public, the PRA does not apply, 5 CFR 1320.5(c)4. If participants are employees, the PRA does not apply.

18F

Dot voting

What

A simple voting exercise to identify a group’s collective priorities.

Why

To reach a consensus on priorities of subjective, qualitative data with a group of people. This is especially helpful with larger groups of stakeholders and groups with high risk of disagreement.

Time required

15 minutes

How to do it

  1. Bring plenty of sticky notes and colored stickers to the meeting.
  2. Gather everyone on the product team and anyone with a stake in the product.
  3. Quickly review the project’s goals and the conclusions of any prior user research.
  4. Ask team members to take five minutes to write important features or user needs on sticky notes. (One feature per sticky note.)
  5. After five minutes, ask participants to put their stickies on a board. If there are many sticky notes, ask participants to put their features next to similar ones. Remove exact duplicates.
  6. Give participants three to five colored stickers and instruct them to place their stickers on features they feel are most important to meeting the project’s goals and user needs. If the group is recently formed, or for other reasons has a low level of established trust and equity, consider a set-up where people can vote privately, e.g. on paper. You’ll need a few extra minutes to compile and display.
  7. Identify the features with the largest number of stickers (votes). Have the team review the voting for whether familiarity bias is resulting in enshrining the status quo.

Considerations for use in government

No PRA implications: dot voting falls under “direct observation”, which is explicitly exempt from the PRA, 5 CFR 1320(h)3. See the methods for Recruiting and Privacy for more tips on taking input from the public.

18F

Five whys

What

An iterative process for identifying the root cause of a problem by posing the question “Why?” at least five times to help separate symptoms from causes.

Why

To identify the root cause(s) of an issue or problem.

Time required

Less than 1 hour

How to do it

Select a particular issue or problem from your user research to investigate further. This could be the most commonly occurring problem or a problem that has been prioritized by the team. Ask why the problem occurred and write down an answer. Repeat this process another four times, building off of the previous response each time to drill down to a root cause. As you probe, make sure you remain sensistive to the emotional response of the interviewee. Sometimes asking why multiple times can cause the interviewee to feel frustrated or defensive if they don’t feel as if they are being heard. See example below:

Starting problem: “We didn’t meet our goal for public feedback during the open comment period.”

  1. Why?
    “Not enough people submitted comments.”
  2. Why?
    “Not enough people made it to the comment submission form.”
  3. Why?
    “The comment submission form was hard to find.”
  4. Why?
    “The link to the comment submission form was buried on the page.”
  5. Why?
    “We didn’t formulate and publish a call to action to submit comments.”

After getting to a root cause, frame or reframe your problem solving approach to address it (e.g., “how might we create a call to action for comment submission?”).

Note: You may ask “why” more or less than five times during this process. The purpose of this exercise is to help identify what is the root cause. Ask “why” as many times as needed to get to what you think the root cause is, while keeping the mental cost of the interviewee in mind.

Considerations for use in government

No PRA implications. No information is collected from members of the public.

18F

Heuristic evaluation

What

A quick way to find common, large usability problems on a website.

Why

To quickly identify common design problems that make websites hard to use without conducting more involved user research.

Time required

1–2 hours

How to do it

  1. Recruit a group of three to five people familiar with evaluation methods. These people are not necessarily designers, but are familiar with common usability best practices. They are usually not users.
  2. Ask each person to individually create a list of “heuristics” or general usability best practices. Examples of heuristics from Nielsen’s “10 Usability Heuristics for User Interface Design” include:
    1. The website should keep users informed about what is going on, through appropriate feedback within reasonable time.
    2. The system should speak the user’s language, with words, phrases and concepts familiar to the user, rather than system-oriented terms.
  3. Consider what hazards the service or product might hold for its users, and nclude heuristics to evaluate whether the site protects against possible harms.
  4. Ask each person to evaluate the website against their list and write down possible problems.
  5. After individual evaluations, gather people to discuss what they found and prioritize potential problems.

Additional resources

Considerations for use in government

No PRA Implications, as heuristic evaluations usually include a small number of evaluators. If conducted with nine or fewer members of the public, the PRA does not apply, 5 CFR 1320.5(c)4. If participants are employees, the PRA does not apply. See the methods for Recruiting and Privacy for more tips on taking input from the public.

18F

Hopes and fears

What

An exercise that quickly surfaces a group’s hopes and fears for the future

Why

To establish a baseline understanding of a group’s expectations and concerns about a project and to give each person an opportunity to voice their perspective

Time required

30–60 mins

How to do it

  1. Ahead of the session, establish what you want to elicit hopes and fears about. For example, you could ask participants to focus on the whole project or that day’s workshop.
  2. At the beginning of the session, create two columns labeled “Hopes” and “Fears” on a white board or large sticky pad. (In a remote setting, you can do this online using collaboration software such as Mural or Google Docs)
  3. Ask participants to take 1-2 mins to write down their hopes on sticky notes (one hope per sticky note).
  4. Invite participants to come up one at a time and add their “hopes” sticky notes to the board and say more about what they wrote. Have participants group their sticky notes as they add them to the board to illustrate emerging themes.
  5. Repeat steps 3 and 4 with fears.

This format can be adapted to include other categories. For example, asking participants to write down skills and experiences can help contextualize each person’s place in the group.

Example

Considerations for use in government

No PRA implications. No information is collected from members of the public.

18F

KJ method

What

A facilitated exercise in which participants list their individual priorities onto cards, collect them as a group, organize them by relationship, and establish group priorities through individual voting.

Why

To reach a consensus on priorities of subjective, qualitative data with a group of people. This is especially helpful with larger groups of stakeholders and groups with high risk of disagreement.

Time required

1–2 hours

How to do it

  1. Gather four or more participants for 90 minutes. Provide sticky notes and markers.
  2. Create a focused question about the project’s needs and select a facilitator to run the exercise.
  3. Give participants five minutes to write at least three responses to the question, each on its own note.
  4. Give participants 15 minutes to put their answers on the wall, read everyone else’s, and make additions. Have participants cluster similar answers without discussion.
  5. Ask participants to write names for each cluster on their own - this is mandatory. They may also split clusters.
  6. Put each name on the wall by its cluster. Exclude word-for-word duplicates.
  7. Reiterate the question and have each person rank their three most important clusters. Visually tally points.
  8. Combine duplicates and their points if the entire group agrees they’re identical. Three or four groups usually rank higher than the rest - these are the priorities for the question.

Example from 18F

18F conducted this exercise with 20 Federal Election Commission staff members to define priorities around conflicting requests. We used this method to get data from staff (not the decision makers) about what they saw as the most pressing needs. We synthesized and presented the data back to the decision makers.

Considerations for use in government

At 18F, KJ participants are almost always federal employees. If there is any chance your KJ workshop could include participants who are not federal employees, consult OMB guidance on the Paperwork Reduction Act and the Privacy Act. Your agency’s Office of General Counsel, and perhaps OIRA desk officers, also can ensure you are following the laws and regulations applicable to federal agencies.

18F

Lean coffee

What

A format for running a meeting without a predefined agenda

Why

To give everyone equal opportunity to surface ideas and vote on agenda topics, allowing meeting attendees to be co-owners in the meeting agenda.

Time required

Flexible

How to do it

  1. Give meeting participants two minutes to write what they would like to talk about on sticky notes (one idea per sticky note)
  2. Have meeting participants review the topics generated and dot vote on the topics they are most interested in
  3. Decide how much time will be spent talking about each topic
  4. Start with the topic that got the most votes
  5. At the end of the allotted time, have meeting participants vote:
    • Thumbs up: Continue talking about the topic for a shorter set amount of time
    • Thumbs down: Move to the next topic

Example from 18F

At 18F, Lean coffee is often used to facilitate community of practice meetings and team meetings when the objective is to provide a forum for the meeting attendees to raise issues that are of interest to them. This method provides structure to these meetings and ensures topics are democratically selected for conversation.

Considerations for use in government

No PRA implications. No information is collected from members of the public.

18F

Stakeholder and user interviews

What

A wide-spanning set of semi-structured interviews with living experts who have an interest in a project’s success, including stakeholders and users.

Why

To build consensus about the problem statement and research objectives.

Time required

1–2 hours per interviewee

How to do it

  1. Create a guide for yourself of some topics you’d like to ask about, and some specific questions as a back up. Questions will often concern the individual’s role, the organization, the individuals’ needs, and metrics for success of the project. Consider how the interview could harm the participant, and adjust your questions to avoid those hazards. For example, might your questions trigger thoughts of painful experiences?
  2. Sit down one-on-one with the participant, or two-on-one with a note-taker or joint interviewer, in a focused environment. Introduce yourself. Explain the premise for the interview as far as you can without biasing their responses.
  3. Follow the conversation where the participant takes it. They will focus on their priorities and interests. Be comfortable with silences, which allow the participant to elaborate. To keep from getting entirely off course, use your interview guide to make sure you cover what you need to. Ask lots of “why is that” and “how do you do that” questions. Consider asking if there are ways the service or product could cause harm to its users if not done carefully, or what assumptions are being made (leaving it ambiguous as to whether we’re referring to our questions or in the service/product).
  4. If there are other products they use or your product doesn’t have constraints imposed by prior work, observe the stakeholders using a competing product and consider a comparative analysis.

Additional resources

Considerations for use in government

No PRA implications. The PRA explicitly exempts direct observation and non-standardized conversation, 5 CFR 1320.3(h)3. See the methods for Recruiting and Privacy for more tips on taking input from the public.

18F

Stakeholder influence mapping

What

A visual representation of stakeholders — the people who are involved — and their potential influence and impact on a project or service system, in comparison to one another.

Why

To uncover and describe power dynamics — the often-unspoken balances of influence and control — that could impact project outcomes; prioritize which stakeholders to engage with and how, and inform a communication and engagement approach.

Time required

~1 hour

How to do it

  1. Gather the team and, ideally, at least one crucial stakeholder familiar with their organization and how it works from both a technical and an interpersonal point of view.
  2. If meeting in person, you’ll need sticky notes and a whiteboard; if meeting remotely, use a virtual whiteboard or tools that support online document collaboration, ideally simultaneously.
  3. Divide the whiteboard into four areas, as a grid. Label the x axis influence and the y axis interest.
  4. List out stakeholders together. Write down names of people, groups, communities, or organizations that your work may impact, and organize them into the four quadrants based on your understanding of their relative influence and interest.
  5. Look at each quadrant to sort who to engage with and how:
    • interested and influential - collaborate with them
    • either influential or interested - keep them informed
    • neither influential nor interested - allow them to drive their own involvement
  6. If your map reveals power dynamics that route around policy, consider whether the information poses personal or professional risk to any stakeholders. Avoid possible harm by sharing this map with only the people who need to understand it, and consider the consequences of those you do share with — or share an edited version.
  7. Review and update the map as you understand the situation better.

Considerations for use in government

No PRA implications. No information is collected from members of the public.

18F