Measuring mutlidimensional problems
Understand how to measure the impact of your HCD work
Compiled Indicators
We call the measurement tool that we’ll build in this guide a compiled indicator. It’s a generalist’s version of a measurement tool widely used by evaluation, data science, and statistics professionals called a composite indicator.
The difference between the generalist’s compiled indicator and the measurement professional’s composite indicator is the amount of mathematical computation required for each.
- In our compiled indicators, we will stop computation at simple averages and basic weighting.
- In composite indicators, advanced computation such as a regression and uncertainty analysis are regularly used.
If you first create a compiled indicator and then start working with measurement professionals, all the logic, if not all the exact data, from your compiled indicator will translate to their work, and they will be able to help you evolve your measurement from a simple compile to a more sophisticated composite.
Compiled indicators are used to measure situations that are multidimensional in nature, which is a match for human-centered projects. The U.S. Agency for International Development (USAID) describes a composite indicator as a measurement tool that “combines two or more data sources into a single measure. They are often used for measuring results that are multidimensional in nature.”1
We’ve chosen to use compiled indicators because measuring multiple dimensions and establishing the value of designed interventions cannot be measured directly. The compiled indicator will help us leverage multiple data sources to give us a more nuanced understanding of what our impact might be. That leads us to a more holistic understanding of our impact, which can more easily be compared across contexts, as well as more easily communicated to others, including leadership and the public.
We will not abandon direct measurements altogether. Rather, we’ll use them as our base, and combine them to make sense of the bigger picture. A compiled indicator stacks together (compiles) several dimensions of a complex situation that signal (indicate) how that situation is going. The individual dimensions are made up of direct measurements. Those direct measurements are the datasets—qualitative, quantitative, and historical—that are discussed in the measurement concepts guide. In the next sections, we’ll go through a two-step process: setting up your compiled indicator, and using it. First, we’ll explain the processes at a high level, then go into detail on each step in later chapters.
The most important thing to remember in this process is that you always need to know why you’ve chosen the big why that you have and its specific indicators and datasets. Compiled measures can be built in many ways, so you need to be able to share why you’ve made the choices you’ve made. Your decisions should be defensible, replicable, and verifiable, and always drive toward answering why this particular measurement construct is important.
Pros and cons of compiled indicators
Combined measures like compiled indicators and their parent, composite indicators, may be a new concept to you. The OECD Handbook on Constructing Composite Indicators provides a useful pros and cons list for using composite indicators, which also works as a pros and cons list for compiled indicators.
Specifically, the OECD states that composite indicators provide value by aggregating the data in complex measurement spaces to “…point out the direction of change across different units and through time.”1 The Handbook also helps “…in setting policy priorities and in benchmarking or monitoring performance.” In simpler terms, a compiled indicator is a bunch of information that can be regularly tracked and revisited so that you can see and evaluate the impact of designed things over time. The Handbook also provides a pros and cons table for using compiled indicators, which we’ve adapted here:2
| Pros | Cons |
|---|---|
| Can summarize complex or multi-dimensional issues in view of supporting decision-makers. | May send misleading policy messages if they are poorly constructed or misinterpreted. |
| Easier to interpret than trying to find a trend in many separate indicators. | May invite simplistic policy conclusions. |
| Facilitate the task of ranking complex issues in a benchmarking exercise. | May be misused, i.e., to support a desired policy, if the construction process is not transparent or lacks sound statistical or conceptual principles. |
| Can assess progress…over time on complex issues. | The selection of indicators and weights could be the target of political challenge. |
| Reduce the size of a set of indicators or include more information within the existing size limit. | May disguise serious failings in some dimensions and increase the difficulty of identifying proper remedial action. |
| Place issues of…performance and progress at the center of the policy arena. | May lead to inappropriate policies if dimensions of performance that are difficult to measure are ignored. |
| Facilitate communication with the general public (citizens, media, etc.) and promote accountability. |
These considerations should inform your use of combined measures and the data you include in them. These pros and cons are not reasons for or against using combined measures; they are simply things to be aware of as you identify and compile your data.
Footnotes
-
OECD/European Union/EC-JRC (2008), Handbook on Constructing Composite Indicators: Methodology and User Guide, OECD Publishing, Paris, 22 Aug 2008. 15. ↩
-
This table was adapted from Saisana, M and Tarantola, S State-of-the-art report on current methodologies and practices for composite indicator development Institute for the Protection and Security of the Citizen (Joint Research Centre) ISSN 1018-5593. Catalog number: EU-NA-20408-EN-C. 2002. ↩