Evaluating Impact: How to Measure a 'Good Life'?

Did it Work…?

I got a report in my inbox a few weeks ago: Has Devolution Worked?

My first thought was, “wow, that doesn’t sound like a question that anyone can answer definitively!”

My second thought (I generally ignore first thoughts) went in the direction of understanding the question ‘did it work?’ What is the measure of success? What is the counter-factual? And why are we asking?

That last point is important because, sure it’s an interesting exploration but knowing whether we’re exploring something for intellectual curiosity, or whether we’re reviewing the structures of government, or whether it’s to support a political point is kinda relevant to how we go about answering.

[I have no real interest in the answer to that question about devolution btw, but rather than leave you hanging, the report was, in fact, pretty interesting. Rather than try to establish and evaluate a single definitive answer, several authors contributed different perspectives , one might say ‘opinions backed by selective evidence’, on whether devolution had contributed to a) improved public trust and legitimacy of government, (mixed responses), b) enhanced growth or redistributed economic effects (a clear no although how easy is it to be definitive without a counter-factual I wondered), and c) policy innovation and testing, the policy-lab effect (a cautious yes).]

What I really ‘heard’ through the report wasn’t the content, which I have little interest in, but a question, and a question that seems so relevant right now with the corona virus.

What if most of what we do is more complex than we imagine?

And, if that’s true, how do we go about measuring complexity?

I even felt myself wondering, ‘does it really matter?’

Let me explain.

It’s More Complex Than We Think…

A year or so ago I decided to prioritise buying pre-loved clothes. Slow fashion is the buzzy insta-pop term—one small step to reduce the apparently significant negative environmental impact of the clothing industry.

It felt like a fun experiment, a creative challenge to seek out the cool amongst the drivel, and a way of ‘doing good’ that fed my social conscience. Honestly, as a way to save the planet, there might be other decisions I could have / could take as well as or instead of.

This is how we do most things—something that occurs to us to do, and we do it, layering on our justification afterwards, and tweaking if it turns out not to go as expected.

A few weeks in, I read an article about the failing clothing sector in Bangladesh and the many reforms it has been through and I had a wobble,

Oh no, is my decision not to buy any new clothes having a negative impact on employment for women in Bangladesh?!

Of course I don’t assume I’m single-handedly keeping the Bangladeshi clothing industry in business, when I say “I” I mean “people like me” / “the movement for pre-loved rather than new”.

What I’m really saying is,

Gosh, this is more complex than I first realised.

With, no doubt, a small, implied personal judgement about whether I had made a ‘bad’ decision with more negative consequences than positive ones.

For a moment I weighed these two ideas in my mind (the former-economist in me coming out to play), and I wondered whether it would be possible to resolve them (probably, I concluded but I doubted the effort was proportionate to any answers I might find, and would it change my behaviour anyway?) and decided to continue on the narrow path of aiming to buy only pre-loved fashion for the year.

But it reminded me, as did the devolution report, that very little is as simple as we see at first glance.

It’s exactly the same right now with corona virus and weighing health and economic costs against lives saved and (unexpected) environmental benefits. I’m not going to play it out because temperatures are high, but let me say that the answer is probably not as straightforward as you might think and it’s worth taking a moment to recognise what it is we’re evaluating, and why.

Sure, when there’s a tsunami, there’s no question in our minds that we wouldn’t rush in and pull people away from the water, rather than calculate lives lost against the cost of flood defences. But that’s no reason, as the waters subside, not to do both.

…Or is It?

My mild jitter over my decision to buy second-hand clothes reminded me of the Taoist fable, Sai Weng loses his horse,

Once upon a time there was a Chinese farmer whose horse ran away. That evening, all of his neighbours came around to commiserate. They said, “We are so sorry to hear your horse has run away. This is most unfortunate.”

The farmer replied, “Maybe.”

The next day the horse came back bringing seven wild horses with it, and in the evening everybody came back and said, “Oh, isn’t that lucky. What a great turn of events. You now have eight horses!”

The farmer again replied, “Maybe.”

The following day his son tried to break one of the horses, and while riding it, he was thrown and broke his leg. The neighbours then said, “Oh dear, that’s too bad,” and the farmer responded, “Maybe.”

The next day the conscription officers came around to conscript people into the army, and they rejected his son because he had a broken leg. Again all the neighbours came around and said, “Isn’t that great!”

Again, he said, “Maybe.”

Planning for Random

The fable highlights the impact of unanticipated events.

Accidentally breaking a leg means a son is saved from conscription.

Coronavirus might change our social and economic structures in ways we can’t imagine. Or it might not.

Brexit will for sure impact the question ‘has devolution worked?’; who knows maybe the question could even be asked in reverse—would there be a Brexit without devolution, How could we possibly know what one small decision will create in futures as yet unfolded.

If I value the principle of devolution, then I’m less likely to be interested in an economic cost or benefit, that becomes a side-note, rather than a yardstick.

I want to make a small dint in my habit of consumerism, then I might shop less, swap more, and revamp what I have. And while I might take account of the potential impact on t-shirt factories in Bangladesh, I can choose to vote for a government with a compassionate and pro-active international policy that will support responsive economies in countries with less of a social safety net.

What we see to be true, and therefore what we value, determines how we act. In this context, it determines both the questions we ask and the answers we hear.

Seeing that alone gives us the potential to take a higher perspective, and to therefore define our outcome differently.

There’s a wisdom, and a humility, to accepting that there will never a definitive answer to the question, ‘has it worked?’, only an answer to the question as we construct it in our assumptions and biases.

The more we can see that, the more leverage, and freedom, we have over our actions and the powerful we can be as social actors and individuals living a ‘good’ life.

Measuring a Good Life…

If I was measuring ‘a good life’ in terms of how many horses I had, I might look at the farmer and judge the outcome as ‘good’; if I was interested in preserving native species in the wild I might see equine domestication as ‘bad’.

Life, however, is not about horses, and it’s not about whether or not we join the army; in fact it’s not about anything that is fixed in time and space.

Asking ‘has it worked?’, or the related question, ‘is this a good thing?’, depends entirely on what we value and, therefore, what we measure, as well as where we start and stop the clock.

It doesn’t mean we shouldn’t ask those questions—I think we can learn a lot by looking back—but we should be very wary about the meaning we give to the answers we come up with.

With love,

Cathy

<3