System Evaluation Theory
A Blueprint for Practitioners Evaluating Complex Interventions Operating and Functioning as Systems
SYSTEM EVALUATION THEORY: PRESS
SYSTEM EVALUATION THEORY: BLOG POSTS
Rules vs principles and context
February 27th, 2023
I was recently at a conference with Michael Quinn Patton in Ottawa Canada learning Developmental Evaluation from the master himself. Why anyone would plan a conference in Ottawa in February is beyond me, it was -20c plus windchill.
As part of our training, Michael was discussing how in the absence of a set evaluation strategy, evaluators might choose to rely on principles to guide their work. He, of course, has written a book on that topic too.
Michael went on to explain the difference between a principle and a rule. Little did I know that the night after the conference, a fire alarm would go off in my hotel, an event which would perfectly illustrate the difference between a principle and rule, but also how important context is in making evaluative judgements as to which standard, the rule or principle, might be at play.
When I first checked into the hotel, I had a very noisy room next to me. I surmised that the hotel was hosting this family until more permanent and suitable living accommodations could be found. The two children were screaming all the time. I was frustrated because it was impacting my sleep, but not angry. I’m sure the parents were having a tougher time than myself. Having my own children, I had some empathy. They too must be tired; they too must be stressed I thought.
On the second night of my stay the fire alarm went off around 1:00 am. I had my usual first reaction, false alarm, it’ll stop. But it didn’t. After the tenth request to evacuate, I thought I better act. The RULE of course, is to leave your personal belongings behind. I didn’t do that. I packed my suitcase and headed down the stairwell.
As I stood outside in the freezing temperatures I could see people passing judgement on me. No one, except me, had a suitcase. Their looks of disgust at me said it all: “What a jerk for taking his suitcase”. As I stood there, I saw the family exit with their two children, nothing to protect them from the cold. Luckily within a minute, they were able to sneak back into a side door for warmth, just before I could open my suitcase and hand them the comforter I had packed.
You see I was operating on a principle dictated by the context, call it empathy, kindness, or whatever…not a rule.
- Ralph Renger
Author, System Evaluation Theory: A Blueprint for Practitioners Evaluating Complex Interventions Operating and Functioning as Systems
What a colonoscopy, kidney stone, and peanut allergy taught me about evaluating complex interventions operating as systems
January 24th, 2023
As I enter the mid to later stages of my life I am confronted with the realties of disease. One joy of getting older is the colonoscopy. As those who have had the procedure will tell you, it’s the preparation that really sucks. Nothing like being wed to the porcelain thrown for a few days. But I’m not complaining. I am fortunate and privileged to live in a country where I have access to preventive health services and the good news is that my colon is cancer free. And there is a certain irony that a colonoscopy has me blogging about evaluating system waste.
Last night, within just a few minutes, I found myself on my knees in excruciating pain, crying like a baby and pleading to God to take me so the pain would end. Our neighbors came over and watched our youngest daughter as my wife rushed me to the hospital emergency department. I’m not going to waste your time talking about the lack of empathy in the hospital setting. I’m convinced after a lifetime of interacting with the health care system that it lacks compassion. Perhaps that should be a defined emergent property to which all health care systems should aspire.
Last year, I was working with food health inspectors evaluating restaurant food safety. As we were evaluating the establishment, I noticed that a food order was being repeated over and over as it passed down the line. Why would they bother doing that I thought when they can all read the order on the electronic screen? As I was chatting with Aaron the owner of Snakes and Latte he let me know that it was a safety precaution. A customer had a peanut allergy and this was their way of ensuring that everyone on the line was aware and that this important piece of information was not going to be lost amid the other orders.
So what do these three experiences teach us about evaluating complex interventions operating and functioning as systems? Well, as I was being passed from one health care provider to the next during my colonoscopy I was asked for my name and date of birth each time. At the time I wondered why? And as I lay there in the hospital hopped up on morphine to help manage my kidney stone pain, I was annoyed that every time the staff came into my room they were asking me to confirm who I was. I was miffed, “don’t they know that I’ve been in that same room for 5 hours” I thought. The system evaluator in me was upset with the redundancies.
Then my experience with Aaron came flooding back to me and it all became clear. When evaluating complex interventions, I am always looking to evaluate interdependencies with the goal of making the interactions between components more efficient. Reworking steps is a sign of system waste. But what I learned, is that reworking process steps is an indicator of a potential problem. What we, as evaluators, need to do, is then dig deeper to understanding whether the process being reworked is actually wasteful (for example, resulting from poorly operating feedback loops) or whether it is a necessary redundancy. Especially in systems with many fast-moving parts, like a hospital, I’m thankful that these redundancies are in place for my safety. Getting the wrong medication or the wrong procedure might actually result in something more serious than a wasted process step; a wasted life.
- Ralph Renger
Author, System Evaluation Theory: A Blueprint for Practitioners Evaluating Complex Interventions Operating and Functioning as Systems
What the war in Ukraine makes clear about the motivation underpinning system change
January 9th, 2023
In evaluating complex interventions, I often find myself making recommendations targeting incremental change. My thought was that having some “small wins” can help build confidence in the evaluation and hopefully lead to Lorenz’ butterfly effect (When Lorenz Discovered the Butterfly Effect | OpenMind (bbvaopenmind.com). But the war in Ukraine, has made me question my incremental change philosophy and the motivation underpinning system change.
In the stroke of a pen policy makers made major changes to an immigration system, that for decades they claimed needed fixing. So why is it that less than a year ago the same Polish policy makers who erected barbed wire and shot at war torn Syrian immigrants welcome in Ukrainians?
Polish soldiers keeping out Syrian refugees
So why is it that the USA policy makers who erected barbed wire and continue to force immigrants who traveled by foot for thousands of miles to wait in tents in freezing temperatures and to be victimized by border gangs, welcome Ukrainian immigrants across the same Mexican-USA border? It was gut wrenching for me to watch a Nicaraguan mother and her child who had been waiting a year in a tent, just meters from a better life, watch a Ukrainian mother cross unimpeded with her child into the USA.
Border crossings scene in El Paso, TX
The circumstances of the immigrants seeking entry into Poland, for that matter all of Europe, and the USA are similar. All are fleeing war. So why, the differential treatment? The only discernable difference I can see is that the Ukraine refugees are white.
The iceberg profile from the Haines Centre for Strategic Management can help shed some light as to why major system change can happen in the blink of an eye in some circumstances and not in others.
Everything systems comes from the base. The base of many systems is rooted in racism. There I said it. But I didn’t need to say. It. We just need to listen what leaders of white countries are saying openly. For example, Trump kicked off his presidential campaign appealing to the nationalism “They’re sending people that have lots of problems, and they’re bringing those problems with us. They’re bringing drugs. They’re bringing crime. They’re rapists. And some, I assume, are good people,” Trump said in his announcement speech. June 2015. The racism, coded “nationalism” isn’t confined to the USA. “The Hungarian Fidesz government and, in particular, Viktor Orbán are known for their racist sentiment toward “the others”. Prime minister Orbán more than once made clear that he takes an openly racist stance towards anyone who is not from Europe or a “Western culture”. “Viktor Orbán’s racist rhetoric and his propagation of the "great replacement theory" - Quo Vademus (quo-vademus.org)
To be fair, not all of those who wield power are racist or may be aware of their biases and prejudices. Nevertheless, that’s not an excuse. Unconscious or not, its real and exists.
In Europe and the USA the power base is white. The structures, that is the immigration policies in Europe and USA, to no one’s surprise represent the values of the those who write them. The processes, are the steps to implement these policies. We can continue to make incremental changes, but not until we acknowledge that the base of many our systems are racist can meaningful changes to the structure of complex interventions be made.
The war in Ukraine makes two things obvious. When there is motivation for major system change it can happen quickly. The motivation for change is rooted in culture and the culture of many systems is white. I see the exact same problem playing out in the medical systems I evaluate. Non-whites do not receive the same level of access to medical care nor the same quality of care. The culture, structure, and processes of medical system is based on a “one size fits all”. That one size is a white size. “But we aren’t racist, we are treating everyone the same”, they say. While equal, it isn’t equitable.
- Ralph Renger
Author, System Evaluation Theory: A Blueprint for Practitioners Evaluating Complex Interventions Operating and Functioning as Systems
Don’t judge a book (intervention) by its cover (label
December 24th, 2022
In my book I make a point of using the term intervention, rather than program or system. I feel it is better to simply speak about interventions in terms of their level of complexity, rather than label them as a program or a system. Why is using the term intervention advantageous?
Many interventions are labelled as systems; like the transportation system, health care system, tax system, etc. However, often these “systems” are nothing more than a “bunch of stuff” (Meadows, 2008). On the other hand, I have come across several interventions labeled as “programs”, like the HUD HOPE VI program, that upon closer inspection have many, interdependent components that are intended to coordinate and collaborate to achieve a higher function: the very definition of a system.
As an evaluator you can’t assume that because an intervention carries the “system” label that it is in fact operating and functioning as a system. Similarly, if the intervention carries the “program” label you can’t assume it isn’t operating as a system. For this reason, I find it more practical to refer to interventions generically and then gauge the correct evaluation approach according to their level of complexity. If an intervention has very few moving components and they are linearly related, then an evaluation using logic models is likely good enough. As I write in my book, there is no need to overcomplicate evaluations.
However, if an intervention has many moving parts, then we need to evaluate whether the parts are actually interdependent and working toward achieving a function that no component can achieve interdependently (i.e., the emergent property). This is because although interventions with many moving parts have the potential to operate and function as a system, not all do.
In my book, I discuss how a necessary first step in any evaluation is to take the time to understand the intervention design. For those interventions designed with many components intended to work interdependently, I then explain how to first define those interdependencies. Once the component interdependencies are defined, I then explain how to evaluate them by applying several system principles (e.g., feedback loops, cascading evets, reflex arcs). The evaluation of interdependencies will confirm whether the intervention intended to operate and function as a system is in fact doing so. If so, then investing in evaluating emergence (the higher functional purpose of the intervention), makes sense. If not, then evaluation recommendations focus on operational level improvements.
- Ralph Renger
Author, System Evaluation Theory: A Blueprint for Practitioners Evaluating Complex Interventions Operating and Functioning as Systems
Outcomes are Not Emergent Properties
November 28th, 2022
One recurring challenge I encounter when explaining the idea of a system emergent property is that colleagues often want to equate “outcomes” with an emergent system property. “Well, isn’t the emergent property just an outcome of the complex intervention?” is what I am frequently asked.
My reply is that I prefer to use the term emergent property when evaluating complex interventions because it does not carry the “baggage” of program evaluation. Outcomes are often equated with the logic model, as in the immediate, intermediate, and long-term outcomes. By definition these outcomes are chronological, they are time bound. This is because the outcomes were derived, using if-then, root cause analysis, reductionist-type thinking. If you are focused on cause and effect, then by definition one thing precedes the other and there is a time component.
An emergent property is qualitatively different. To emerge, according to Merriam-Webster is “to become manifest; to become known”. What becomes known occurs through the interdependence between complex intervention parts. Equity, quality of life, stability, and so forth are examples of emergent properties for complex social interventions. They are not chronological, rather they manifest themselves when the intervention components are operating as they should: an emergent property is the function of a complex intervention that arises through the product of component interactions (Ackoff, 1994).
I get that there is a certain sense of comfort in trying to equate emergence with something one already knows and is familiar, like outcomes, but they are not the same thing. If you equate emergence with a long term outcome, you miss the opportunity to collect data on the higher purpose of the complex intervention, and the intervention and the evaluation suffers as a result.
- Ralph Renger
Author, System Evaluation Theory: A Blueprint for Practitioners Evaluating Complex Interventions Operating and Functioning as Systems