System Evaluation

WHAT IS A SYSTEM?

A system is “an integrated whole whose essential properties arise from the relationship [interdependence] between its parts (p. 142)."

Ison, R. L. (2008). Systems thinking and practice for action research.

“There's always a flaw in the system” - Beetee (The Hunger Games Catching Fire)

OUR CONTRIBUTION TO SYSTEM EVALUATION

We are proud to be one of the world leaders in system evaluation. Dr. Renger published system evaluation theory (SET) which he developed based on his work in the emergency response sector. Our evaluation family has since published 7 articles on the application of SET. SET combines systems thinking and system theory to walk the client through three steps to evaluate their systems. SET is especially helpful in evaluating agencies working together around a common goal.

WHAT SYSTEM EVALUATION SERVICES DO YOU OFFER?

When evaluating a system we explore your needs in 2 main areas.

1. DEFINING YOUR SYSTEM

Working with leadership we will define who shares your common goal and how are you connected. For more information, please refer to our article on defining systems linked below.

Defining Systems

2. EVALUATING SYSTEM EFFICIENCY

We will examine your standard operating procedures for:

1. System waste: we look for unnecessary duplication of effort (waste). We will then evaluate the source for this waste. Is it a staff training issue? Is your information technology (IT) causing delays? If your leadership supportive and providing the resources needed? Is your staff culture supportive of your goal? For more information, please refer to our article on system waste linked below.

2. Cascading failures: we don’t just look at where we see the problem, but evaluate to determine whether there is an upstream cause. For more information, please refer to our article on system cascading failures linked below.

3. Functional feedback loops: system health depends on functional feedback loops. We examine whether information is getting back to who needs it and whether the feedback is viewed as credible, relevant, specific, timely and sufficiently frequent. For more information, please refer to our article on system

feedback loops linked below.

4. Reflex arcs: is there a way to streamline your operations without sacrificing quality?

Evaluating System Cascading Failures

Reworks: A robust system efficiency measure

Evaluation of System Feedback Loops

PRIOR SYSTEM EVALUATION WORK

Becker, Renger, Mcpherson Inidcators of

Renger Foltysova Renger Booze Defining s

Renger Keogh Hawkins EJA article Robust

Renger SET article in EJA December 2015.

Renger, Foltysova, Ienuso, Renger & Booz

Renger, Krapp, Wood, Williamson 2011 Sys

Renger, R. Evaluation of system feedback

Lessons Learned in Evaluating Infastruct

Comparing and Contrasting a Program vers

Becker, K., Renger, R., & McPherson, M. (2015). Indicators of buy-in to gauge evaluation success. Evaluation Journal of Australasia, 15 (2) 12-21.

Renger, R., Atkinson, L., Renger, J., Renger, J., & Hart, G. (2019). The connection between logic models and systems thinking concepts. Evaluation Journal of Australasia, 19(2), 79-87.

Renger, R., Basson, M. D., Hart, G., Van Eck, R., Souvannasacd, E., Renger, J., & Foltysova, J. (2020). Lessons learned in evaluating the infrastructure of a Centre for Translational Research. Evaluation Journal of Australasia, 20(1), 6-22.

Renger, R., Foltysova, J., Renger, J., & Booze, W. (2017). Defining systems to evaluate system efficiency and effectiveness. Evaluation Journal of Australasia, 17 (3), 4-13.

Renger, R., Keogh, B., Hawkins, A., Foltysova, K., & Souvannasacd, E. (2018) Reworks: A robust system efficiency measure. Evaluation Journal of Australasia, 18 (3), 183-191.

Renger, R. (2015). System Evaluation Theory (SET). Evaluation Journal of Australasia, 15 (4), 16-28.

Renger, R., Foltysova, J., Ienuso, S., Renger, J., & Booze, W. (2017). Evaluating system cascading failures. Evaluation Journal of Australasia, 17 (2), 29-36.

Renger, R., Wood, S., Williamson, S., & Krapp, S. (2011). Systemic evaluation, impact evaluation, and logic models. Evaluation Journal of Australasia, 11 (2), 24-30.

Renger, R. (2016). Illustrating the Evaluation of System Feedback Mechanisms using System Evaluation Theory (SET). Evaluation Journal of Australasia, 16 (4), 15-21.

Renger, R., Foltysova, J., Renger, J., Donaldson, S.I., Hart, G., & Hawkins, A. Comparing and contrasting a program versus system approach to evaluation: the example of a cardiac care system. Canadian Journal of Program Evaluation, 35(2), 240-257.

Renger, R., Hart, G., Van Eck, R., Souvannasacd, E., Renger., J., & Basson, M.D. Lessons Learned in Evaluating a NIH Centre for Translational Research (CTR). Evaluation Journal of Australasia, 20 (1), 1-17