An introduction to compiled indicators

How to create a compiled indicator

The setup for your compiled indicator has three parts; you’ll work from the big why, then zoom into the smaller details.

1. Name the multidimensional situation: First you’ll document the situation that your intervention, office, department, and/or agency works in. We’ll call this the “big why” and discuss it in the next section.

2. Establish indicators: Next, you’ll break the big why into different dimensions that indicate how the problem arises.

3. Designate individual datasets for each indicator: Finally, for each indicator, you’ll identify a collection of datasets. These datasets should represent a diverse mix of quantitative, qualitative, and/or historical data whenever possible.

Using your compiled indicator

Once you’ve assembled your compiled indicator, you’ll begin using it by inverting the set-up process. You’ll start with filling in the individual datasets, then seeing how they affect the indicators, and finally assembling the numerical compiled indicator itself. We’ll go into greater detail later in this guide, but here’s an overview of the process:

1. Gather the designated datasets: Inventory the datasets you have on-hand and identify any gaps in what you need to build a complete inventory of relevant datasets. There may be gaps where the data exists, but you don’t have access to it (yet). This could take some digging and asking around. The datasets you need could be in the data repositories of your or other organizations. Sometimes, you’ll find that the gap is there because the data you need hasn’t been gathered yet. In this situation, decide whether you can or should substitute in a thematically similar dataset to which you have access. Beware of the pitfalls of confirmation and searchability biases as you make this decision, and balance those against the time and resources you have for discovery work. If you don’t substitute in a similar dataset, go back to the discovery phase to research and produce the data you need.

2. Normalize the data: Since the datasets you have for each indicator likely rely on differing units of measure, you’ll need to find a way to put all the data on the same scale so we can combine and look across datasets. This process is called normalization. This is the most math you’ll need to use in this guide. But don’t worry – we’ll walk you through the process step-by-step later in this guide. It may sound complicated, but normalization is well within your grasp. If you don’t have time, you can move forward anyway. Remember, measuring something is better than measuring nothing!

3. Produce an indicator score: Once you’ve normalized the data, you’ll combine the normalized datasets, sometimes with weighting and averaging, to identify a score for each indicator.

4. Produce the compiled indicator score (Optional): By creating a normalized score for each indicator, you have successfully created a compilation from which you can understand the effectiveness of your intervention and the big why of your work. If you would like, you can take this one step further and create a fully compiled indicator score. This final score has advantages, like ease of communication, and disadvantages, like obscuring the underlying indicators. Making the decision about whether to show your composite parts or combine into a single score is up to you and the context of your work.

Advantages of compiled indicators

1. Ease of communication: Since a compiled indicator is a single number, it’s easy to communicate to both leadership and to the public.
2. big why view: Through compositing, we can see our daily work as part of a much bigger picture.
3. Lateral understanding: Because compiled indicators include not just our work but others’ work as well, we can start to understand and track how that other work affects our work, as well as the larger problem space.

Disadvantages of compiled indicators

1. Obscurity: Compiled indicators can obscure data and details, so carefully footnote, link, and provide transparency into your methodology.
2. Sprawl: Including too many indicators will result in a composite that’s so vast you can’t responsibly track movements in it back through the indicators, and into datasets. To properly scope your work, we suggest the Rule of Eight.
3. Squabble: Compiled indicators owe “…more to the craftsmanship of the modeller (sic) than to universally accepted scientific rules for encoding…”¹ This means that you could find yourself in squabbles with colleagues or stakeholders about the selection of datasets or indicators. To avoid this, be careful to craft a compiled indicator that is defensible, replicable, and verifiable. A well-crafted, sturdy indicator will stand up to questioning.
4. Crowding: This happens when several groups produce competing compiled indicators describing the same problem. In these situations, we encourage each team to work to understand the work of the other teams.

You may find that each compiled indicator describes different facets of the same massive problem, in which case all the measurement tools can be useful. You might find that some parts overlap, so you should work with the other team to decide how you might swap datasets in and out of the different compiled indicators to make each one stronger. You may also find that there is meaningful competition between equally strong compiled indicators, so you could consider use cases for employing both simultaneously, to develop the best solution for your big why.

For more details on using and communicating compiled indicators, see chapter 7 of the UNECE’s Guidelines on Producing Leading, Composite, and Sentiment Indicators.²

Up Next