The mixed methods approach

The mixed methods approach

In the previous section, we talked about the three different data types you’ll need to use when measuring impact: qualitative, quantitative, and historical. In this section, we’ll go into how to put those data types together in a “mixed methods” approach that will produce a strong and resilient measurement strategy.

Producting comprehensive insights

A mixed methods approach allows for a deeper exploration of your research questions, to help you draw more meaningful conclusions. This isn’t to suggest that quantitative or qualitative research alone are always insufficient, but in measuring human-centered interventions, using both types will produce a more high fidelity understanding, than limiting your use to a single data type.

One important thing to keep in mind is that a mixed methods approach is not just about having different types of data; it’s about using them in concert. If you don’t consider your datasets as part of an orchestrated whole, you’re simply gathering data in silos, instead of understanding the multi-dimensional space. But if you craft a collection of datasets that all play different parts in creating a whole, you’ll not only increase your ability to understand that whole, but also create a basis for informed decision-making.

Principles for your measurement strategy

A strong measurement strategy has three main qualities: it is defensible, replicable, and verifiable.

Defensible

Defensible data is data that can stand up to the scrutiny of questioning. In building a defensible measurement strategy, you must be able to uphold and even advocate for the data you used and the interpretations you drew from it when they are rigorously questioned. Document your work,

only use a dataset when you can confirm its origins, and you’ll be well on your way to building a defensible measurement strategy.

Replicable

Ideally, other people should be able to replicate your impact measurements by setting up the same situation you were faced with, and taking the same recordings. As time passes, however, perfect replicability is not possible; those years of impact cannot be recreated perfectly. But your work should be “arguably replicable insofar as others can be ‘walked through’ the analyst’s thought processes, data collection, and interpretations.”¹

In the absence of the possibility of a perfect recreation, any questioner of your measurement strategy approach should be able to follow your work closely, and reasonably draw the same conclusions you did.

Verifiable

Using multiple datasets in concert will produce the most complete picture possible of the situation that’s being measured. This approach - called triangulation¹ - is most often mentioned as the main advantage of the mixed method approach. Triangulation not only allows you to check and balance data sources against each other, it also allows you to use partial or imperfect data sources, if parts of them are useful and verifiable. According to the National Science Foundation’s User-Friendly Handbook on Mixed Methods Evaluation, “The validity of results can be strengthened by using more than one method to study the same phenomenon.” ¹

Addressing some logical doubts

Using a mixture of qualitative, quantitative, and historical data to produce a defensible, replicable, and verifiable measurement strategy means orchestrating data that does not share a unit of measure, or a format, or even a common logic model. Because of this, the datasets in your measurement strategy will look substantially different from one another. That’s okay. Referencing the National Science Foundation once again, we learn that each data type has:

“…advantages and drawbacks when it comes to [measurement strategy] design, implementation, findings, conclusions, and utilization. The challenge is to find a judicious balance in any particular situation…” ²

Even accepting mixed methods, you might think that building a measurement strategy based on subjective terms like defensible, replicable, and verifiable seems dubious. You might ask yourself whether this approach is objective enough to be called measurement. These are good doubts to have. They mean that you’re concerned about creating an accurate and meaningful impact measure, one that can help you and your organization truly understand what is going on, and intelligently inform the evolution of your designed thing.

To address those doubts, we offer three references, two from the National Science Foundation (NSF)’s User-Friendly Handbook on Designing and Conducting Mixed Method Evaluations (1997) and one from the United Nations Economic Commission for Europe’s (UNECE) Guidelines on production leading, composite, and sentiment indicators (2019).

In the User-Friendly Handbook, the authors stated that:

“according to Cronbach (1982), There is no single best plan for an evaluation, not even for an inquiry into a particular program at a particular time, with a particular budget.”²

Further, the Handbook points out that any situation in the real world operates in a:

“…complex social environment with features that affect the success of

the project. To ignore the complexity of the background is to impoverish the evaluation…when investigating human behavior and attitudes, it is most fruitful to use a variety of data collection methods (Patton, 1990). By using different sources and methods at various points in the evaluation process, the evaluation team can build on the strength of each type of data collection and minimize the weaknesses of any single approach. [Therefore], A multimethod approach to evaluation can increase both the validity and reliability of evaluation data.” ²

The more recent Guidelines from the UNECE builds on this. In their handbook, they point out that, in the past, things like perceptions, attitudes, or expectations:

“…that involve subjective assessment on the part of the respondent were widely considered outside the purview of official

statistics…but increasingly…Subjective measures…are considered as reliable measures backed by international studies and guidelines. Subjective measures have also turned out to be relatively consistent with objective indicators which function as external validators.”³

As a core tenet of human-centered design, the inclusion of human perception in datasets built and vetted by statistical experts is validating. While measurement can seem messy or subjective upon a cursory glance, by using multiple types of data, you can use the inevitable gaps in data to triangulate around an accurate measurement of impact.

The mixed methods approach helps you make sense of complex situations, measure them, and track changes over time. This allows you to first understand impact, and then, ultimately, to understand the value of your interventions, all based on a diverse collection of data orchestrated in a defensible, replicable, and verifiable manner.

Up Next