Designate balanced datasets

The final step in setting up your measurement tool is to designate the datasets that describe the indicators you’ve just established. To illustrate, we’ll continue to use the World Happiness Index and the Enterprise Digital Experience (EDX) Index from the previous section as examples.

Since indicators are the facets or dimensions of your big why, they can be measured in a variety of ways. Some indicators are easily measured by quantitative metrics, but there might be other datasets featuring different types of data to describe that indicator, as well. So think hard about how you could measure more precisely. One way to do this is with the opposites attract method.

Activity: Opposites attract

This brainstorm activity will help you quickly generate ideas for datasets with appropriate counterbalances. The idea is to work quickly and not get bogged down in fine-tuning your dataset designations.

Set the stage

Write out each of your indicators in a list, either on paper or a whiteboard, or in your digital whiteboard.

Gather materials

In addition to your paper or whiteboards, you’ll need a timer to complete this exercise.

Take action

1. Rapid response round 1: Set a timer for 2 minutes. In that time, write down one dataset that could describe each of your indicators.

2. Rapid response round 2: Set a timer for 2 minutes. Write down another dataset that could describe each of your indicators.

3. Tag your data: Review your list of proposed datasets and tag each one with its datatype: qualitative, quantitative, or historic.

4. Opposites attract: Set a timer for 5 minutes. In that time, again review your list of proposed datasets. Next to each, write down an idea for a dataset that is totally opposite to the first datasets you wrote down, in terms of data type or attributes. For example:

   • Is your original dataset brand new? Consider what a dataset that’s 10 years old might be named or might comprise.
   • Is your dataset qualitative? Consider one that’s quantitative.
   • Is your original dataset a survey of public users? Consider a survey of federal users.
   • Is your original dataset a count of visitor numbers? Consider a dataset on the cost of supporting what those visitors are coming to see.

After you’ve done this exercise, step back and look at your indicators and proposed datasets. You should see how each indicator can best be described by using mutually balancing datasets, and how datasets almost always have an opposite that you should consider.

Dataset examples

In the World Happiness Index, all the indicators are quantitative, but some are subjective. These subjective answers occur when the format of the research is a survey and the respondents are asked to quantify emotions or perceptions. Because the respondents have quantified their emotions, data and evaluation scientists consider these datasets quantitative data, even though they are neither objective, nor can one respondent’s answers be accurately compared to another. Since that is the professional practice, we will tag these as “quantitative data, but subjective” in our list below.

The Index uses the following datasets to describe those indicators:

• GDP per capita: Purchasing power parity (PPP) at constant 2017 international dollar prices are from World Development Indicators (for the 2023 report)

  • Datatype: Quantitative

• Social support: The national average of the binary responses (either 0 or 1) to the [Gallup Poll] question “If you were in trouble, do you have relatives or friends you can count on to help you whenever you need them, or not?”

  • Datatype: Quantitative, but subjective

• Healthy life expectancy: Data extracted from the World Health Organization’s (WHO) Global Health Observatory data repository¹

  • Datatype: Both qualitative and quantitative

• Freedom: The national average of responses to the Gallup World Poll (GWP) question “Are you satisfied or dissatisfied with your freedom to choose what you do with your life?”

  • Datatype: Quantitative, but subjective

• Generosity: The national average response to the GWP question “Have you donated money to a charity in the past month?”

  • Datatype: Quantitative

• Corruption: The national average of the survey responses to two questions in the GWP: “Is corruption widespread throughout the government or not” and “Is corruption widespread within businesses or not?” The overall perception is just the average of the two 0-or-1 responses

  • Datatype: Quantitative, but subjective

All the datasets used to support the indicators are defensible, replicable, and verifiable, but these datasets aren’t the only datasets that could support the indicators. For example, the Social Support indicator could be supported by quantitative datasets such as the number of governmental (such as food assistance programs) and non-governmental (such as soup kitchens) social support programs that were accessed in the past year. A later iteration of the Index might add social support program access to the Social Support indicator, if the Index team has bandwidth to do so. One of the strengths of the compiled indicator approach is that it provides defensible, replicable, and verifiable results that are also extendable. That is, the composite has room to grow.

In the EDX Index, the team uses the indicators of accessibility, customer-centricity, performance/SEO, use of required links, user behavior, and USWDS implementation to determine if GSA is improving the overall digital experience for customers. Like the World Happiness Index, they use a variety of datasets to describe these indicators. They are:

1. Accessibility: Scored by agency-standard accessibility-scanning software. The EDX team scans the homepage and top ten most-visited pages of a website. The GSA Accessibility team scores the homepage and the first 1000 linked pages on a website.
   • Datatype: Quantitative
2. Customer-centricity: Scored by human-centered design interview
   • Datatype: Qualitative
3. Performance/SEO: Scored by agency-standard SEO scorecard
   • Datatype: Quantitative
4. Required links: Scored by the EDX Command Line Interface (EDXcli) and Site Scanner
   • Datatype: Quantitative
5. User behavior: Scored by user behavior records
   • Datatype: Qualitative
6. USWDS implementation: Scored by the EDXcli and Site Scanner
   • Datatype: Quantitative

As in the World Happiness Index, there are other ways to describe these indicators, using different datasets than the ones chosen, but they were chosen because they are defensible, replicable, and verifiable.

For example, in the EDX Index, accessibility is described by a single software solution, but that’s not the only way to measure accessibility. Indeed, we could add to this indicator a dataset gathered by manual accessibility evaluations, which would be both qualitative and quantitative. Automated scanning is a good way to get a baseline for certain elements of accessibility because it’s an industry standard, but we have room for improvement in the datasets we use to describe our indicators, just like the World Happiness Index has room to grow.

That’s the bottom line in gathering the datasets to support your indicators:

• You must curate the datasets you use
• You must limit them to a small number of sets that your team can reasonably support.

The datasets you choose must be defensible, replicable, and verifiable. As you have bandwidth to grow, or gain greater fidelity in your indicators, you can add more datasets to create an even richer view of your indicators. At their core, data should describe your indicators, so that you can describe the impact of your work on the big why your team, department, or agency takes on.

At this stage, your composite is not the only possible, or even the best composite in the world, but you’ve learned enough to start using your measurement tool.

Up Next