Dealing with a data-filled world

Dealing with a data-filled world

Sometimes it’s OK to use imperfect data

“Measure what matters!” is a common refrain in customer experience documentation, but while it’s well-intentioned, it lacks direction. How do we know what matters, and by extension, what to measure, especially when talking about impact? Frequently, we start the measurement phase by using data we have access to, not the best data to answer our research questions. To compound this issue, we frequently don’t know what we don’t know; we simply know that we don’t know everything.¹ As explained in the data types section, we must accept that data will always be imperfect, and we must triangulate data to produce the most accurate understanding possible.

The coastline paradox: perfect data doesn’t exist

Using “imperfect data” might seem like strange advice. Shouldn’t we seek to use data that is as unflawed as possible? Of course. But data without flaws is rare. Even in situations that seem like they should be straightforward and empirical, like how to measure a coastline², researchers make underlying decisions that shift the data in different directions.

The need for transparency in measurement design only increases when we seek to measure the impact of human-made interventions. Like coastlines, human-made things are affected by a huge number of variables. Unlike coastlines, human-made products, programs, services, and systems are made intentionally, with goals in mind, and almost always affect change in both foreseeable and unforeseeable ways. This is why the coastline paradox is particularly acute when dealing with human-generated data about human-designed things; the data is never, ever free of the human-imposed judgements and decisions imposed upon it. All datasets are reflections of the designers of the set, of the data-gathering tool(s) used, and of the thing they measure.

Conclusion

It’s impossible to measure human-centered projects directly and with perfect clarity, so the question we must ask ourselves cannot be “How do I measure everything about this intervention?”

Instead, we must ask “How might I collect as much relevant data as possible, in as many dimensions as possible, so that I can understand how the intervention is working in the wider world?” To do that, you must design your measurement instrument with care, defining and documenting at each step, and be aware and able to communicate the limitations of your measurement instrument to others. And for the rest of these measurement guides, we’ll try to set you up to accomplish just that task.

Up Next