Produce an indicator score

How to complete your compiled indicator

After normalizing all your data, it’s time to combine sets to get each indicator score. After you have each indicator’s score, you can consider your work done: you’ve successfully built your compiled indicator and will be able to track the impact of your intervention.

If you want to distill your composite into a single number, however, you can repeat this step and combine each indicator score into a single number, producing an overall compiled indicator score. Whether or not you decide to go with the dashboard or the single score route, let’s focus on completing the compiled indicator by determining whether to weight or not to weight your datasets and/or indicators.

Weighting

Weighting is an important aspect of this scoring process. Weighting means assigning more or less importance, or weight, to one or more dataset. This same consideration applies to the optional last step in compiled indicator production, producing a compiled indicator score.

To assign weights, assess whether any dataset(s) is more important than others to the composition of the overall big why. If any of them are, you’ll want to use weighting.

For example, if you’re building a customer experience compiled indicator score, you might consider weighting the sentiment scores from your surveys more heavily than an indicator that is less directly aligned with the customer, such as website performance. Document this decision and your logic in your project notes. If none of your indicators are more important, you’ll use equal weighting.

Weighting can substantially change how the compiled indicator brings your big why into focus, and how the impact of your intervention shows up in the big why. For example, if you were creating a customer experience compiled indicator and you heavily weighted the sentiment score, changes in sentiment score will show up with more impact in the overall compiled indicator score.

In this way, you can essentially focus your compiled indicator to certain parts of your big why. In the example, the compiled indicator score is tuned towards customer sentiment over other elements of customer experience. If, on the other hand, website performance were weighted more heavily in the score than sentiment score, your compiled indicator score would more readily show the impact of changes in website performance than customer sentiment.

Equal weighting

If each indicator is of the same importance in building out the big why, we recommend equally weighting the datasets.

In the EDX Index, each of the six indicators has multiple datasets feeding into them, but they use equal weighting for all of them. As an example, let’s look at the datasets supporting the customer-centricity score. Those datasets are:

1. Stated audience
2. Stated purpose
3. Repeatable customer feedback mechanism
4. Ability to take action based on feedback
5. Ability to measure impact

The team gathers this data through semi-structured Discovery phase interviews with each website team. They then interpret the qualitative data, and code the teams’ answers to yes or no for each question. We then convert it to quantitative, subjective data by scoring 2 points for each “yes” answer and 1 point for each “no” answer.¹ For example, can a team clearly state their website’s audience?² If they can, the scored answer is yes and equals 2 points in their overall score; if they can’t, the scored answer is no and they receive 1 point. There are no 0 scores unless a team does not show up for the interview after three attempts to meet with them.

Since each indicator in the customer-centricity compilation is equally important to providing excellent customer experience, each gets equal weighting. The team does the same when rolling up to the overall indicators of GSA digital experience. The overall indicators of digital experience at GSA are:

1. Accessibility
2. Performance/search engine optimization (SEO)
3. User behavior/non-duplication
4. U.S. Web Design System (USWDS) use
5. Required site links
6. Customer-centricity

All of these have multiple datasets underpinning them. User behavior/non-duplication and USWDS use have mixed quantitative and qualitative datasets as part of their collection. None of these elements can be determined as more or less important than the other, so they are given equal weight.

When to use different weights

A common use of compiled indicators in the federal government is in human resources performance scores, where employee performance measures have a variety of categories, each with different weights unique to the employee. The Enterprise Digital Experience team in the General Services Administration used:

1. Customer experience principles and practices
2. Critical and strategic thinking
3. Teamwork and leadership
4. Project management
5. Communication

Everyone on the team can be scored according to these performance indicators, but not everyone’s indicators should be weighted equally. For example, a person whose job title is “Project Manager” will have a heavier weight assigned to the project management section than someone whose title is “Strategist.” A strategist will have a higher weight assigned to the critical and strategic thinking indicator.

While each person has the same five indicators in their performance plan, each element is assigned a different percentage (weight). One person may have 10% for leadership, and another may have 40%, but the total of all five areas together always equals 100%.

As explained above, the different weights in a performance plan allow the same indicators of performance to be used in application to many different team members with different roles. This flexibility is one of the main advantages of compiled indicators in measuring complex topics, such as human performance.

Using compiled indicators in your work

As you use your compiled indicator to track impact on your big why, remember that others will likely have assembled their own composites to do the same or similar work. This is okay: complex problems don’t have single solutions or right answers in the traditional sense, so we encourage teams to take a non-competitive approach.

When to use multiple indicators

Multiple compiled indicators can be useful to teams who are tackling the same big why. This is especially useful when the teams work in similar, but not identical, parts of the big why. When you learn about other teams using compiled indicators to measure their impact on the same or similar big why, consider looking at their methodology to learn about:

• Ideas that could improve your own compiled indicator.
• Indicators or data that might further inform or influence your indicators or data.
• Places to crosswalk data and validate each other’s indicators.

Also consider how, collectively, this work could influence other teams’ work. Consider how you might collaborate, and open discussions so you’re working efficiently, and in concert.

When to try to combine indicators

If teams are using different compiled indicators to measure an identical big why, there might be a need to combine indicators.

We recommend each group not try to create a composite that encompasses the others, but instead to take on a good faith effort to design an indicator based on the best data and indicators from each individual composite. This new compiled indicator should work for both groups to take on the big why. This will be a design process, so you might consider consulting the design phase guide for information on how to best create a version 1 compiled indicator and then to iterate upon it.