Skip to main content

The Journal Gazette

  • Illustration by Gregg Bender | The Journal Gazette

  • Stone

Sunday, May 06, 2018 1:00 am

Impact analysis

Setting useful standards for judgment of complex systems

Michael E. Stone

A recent exchange in The Journal Gazette between Christopher Guerin and Councilman John Crawford about Fort Wayne United's initiatives on behalf of young black men included a few comments that caught my attention. Crawford stated that the use of public funds for the initiative will depend on the ability of Fort Wayne United to demonstrate success based on evidentiary analysis. A previous statement from Guerin, that measuring the efforts of Fort Wayne United is an appropriate surrogate for outcomes, drew a sharp rebuke from Crawford. This approach, according to Crawford, typifies thinking of the “unscientific mind.”

While I agree that evidence is paramount in funding and policy decisions, I know also how important it is to calibrate expectations for results based on the nature of the work we undertake.

At its core, what is occurring between Crawford and Guerin is not a battle of wits between the scientific and the unscientific mind. It is more nuanced than that. The source of the conflict is the differentiated thinking between the scientific mind and the sociological imagination.

Launching a rocket into space is a complicated undertaking. Our ability to achieve this amazing feat is due to the fact that we have deep understanding of the laws of physics, the nature of gravity and the calculus of trajectory. In other words, rocket launches occur because the variables we encounter are understood, predictable and, in part, controllable.

Similarly, clinical trials for cancer treatments are governed by airtight protocols which are possible because of the domain within which medical researchers work. Cancer cells are contained within a human body; treatment dosages are direct and precise; and we can measure critical indicators of progress, such as white blood cell counts. Difficult work, indeed, but this type of science has the benefit of occurring within firm boundaries.

Measuring the success of Fort Wayne United is akin to neither rocket science nor a drug trial. It is, in some fundamental ways, more challenging than both. That is because social systems are not complicated. They are complex.

The difference between complicated and complex systems is not the number of variables involved, but the nature of those variables. Social systems are populated with variables that rely on the power of human construction and interpretation to understand.

While demographers can show us the distribution of such things as poverty and unemployment, what we understand least is the phenomenology of these variables. How do the effects of poverty play out in the everyday lives of people? Why is the culture of despair so difficult to flip? Why does one youth react to a threat with violence while another chooses to walk away?

Further, social variables have permeable boundaries. Where, for example, do the negative effects of poverty end and the positive presence of a male role model begin in shaping one's aspirations?

It would seem having this knowledge would take the guesswork out of social intervention. Yet, we simply have no reliable means of pursuing answers to these questions.

This does not mean that effort alone should be used to judge the benefits of a program. But it is equally misguided to think that behaving more “scientifically” will give us the clear answers we seek. To provide useful information on which to base funding decisions, we need to maintain the rigor of the scientist while negotiating the ambiguity of social relations.

Analytics expert Jeff Moher reminds us that every system is perfectly designed to get the results that it gets. For Fort Wayne United to succeed, it must disrupt the existing social system while having little or no control over the actors, structures, and experiences that comprise it.

Therein lies the challenge. How do you instill a sense of hope and purpose in individuals whose life circumstances might suggest otherwise? How do you melt away hardened perceptions that dictate how people treat people they regard as the other?

In some important ways, this is much more daunting than the task of calculating the amount of thrust required to propel a rocket beyond the earth's atmosphere. Because of this, our expectations for results from Fort Wayne United – what type, for whom, within what time frame – need to be adjusted accordingly.

I offer the following advice to those funders, decision-makers and policy-makers who are following the progress of Fort Wayne United.

• Begin with realistic expectations for social change.

It is difficult to measure prevention, and even more difficult to attribute specific changes to any particular activity. These limitations notwithstanding, it is a good bet that a lot of good things will happen in the lives of individuals before the needle moves on communitywide metrics related to homicide or crime.

Recognize and value incremental successes while learning from the mistakes that inevitably will occur.

• Value consistency over precision.

The logic of the effort does matter, especially when the results we seek are long term and are likely to come from an ongoing pattern of trial and error.

A starting point for evaluating Fort Wayne United is to examine the underlying logic for what it is doing, who benefits from each activity and how each set of outcomes contributes to the larger aims of the initiative. Too often we value an airtight plan over consistent application of a thoughtful concept.

• Value inference over proof.

No individual or organization should bear the burden of proving causality between actions and specific results. Instead, we need to adopt what Michael Quinn Patton, evaluation guru, refers to as the “reasonableness standard.”

When assessing performance of a program, we should ask ourselves whether a reasonable person would conclude that the actions of the organizations contributed in a meaningful way to the positive results we see.

The truth is that in the complex mix of everyday life, we never know which action – large or small – will flip the switch inside a particular individual within a set of social circumstances.

Will late-night basketball reduce the crime rate? Maybe. But that is the wrong question.

The more appropriate questions are these:

1. What factors that contribute to the crime rate are being addressed through late-night basketball? and

2. What evidence can Fort Wayne United gather to convince a reasonable person that late-night basketball is doing its part to ameliorate those factors?

It is trite to say that social change is difficult. But it is. Despite our wishes to the contrary, there is no magic combination of ingredients that will solve the social problems we face.

Consequently, we must substitute our longing for quick solutions with the humility and patience that allow us to figure things out as we continue to move forward.

Fort Wayne resident Michael E. Stone, founder of Mike Stone Consulting, advises nonprofit organizations on strategy, evaluation and other topics. He is the author of “From the Inside Out: A Nonprofit's Guide to Meaningful Strategy.”