
Evaluation practice
Selected publications, conference papers, workshops and seminars on systems-based evaluation, evaluation practice and environmental water management.
For reference managers such as Zotero, EndNote and Mendeley. Plain text — if your computer asks what to open it with, any text editor will do.
Keogh, B. and McInnes, R. (2022) ‘Evaluation of Sydney Water’s WaterFix® Residential Program, 2019-2021’, Cobalt59, Sydney, June. Prepared for the NSW Department of Planning, Industry and Environment.
A hybrid Systems Evaluation Theory and program logic evaluation of the residential water efficiency program expanded with NSW Climate Change Fund money during the Sydney drought, synthesising scientific studies, customer feedback and management data. The evaluation was peer reviewed and assessed as conforming to NSW Government principles, norms and standards.
View reportARTD Consortium with Slattery & Johnson, Cobalt59 and Alexandra Consulting (2021) Environmental Water Management Program 2014-2019 Evaluation, report for the NSW Department of Planning, Industry and Environment, February 2021. Brian Keogh represented Cobalt59.
A systems evaluation of the NSW Government’s environmental water management program, using critical systems heuristics to frame stakeholder perspectives and map constraints to root causes. The evaluation reviewed 145 documents, conducted 43 interviews and surveyed 125 stakeholders, and was accepted without major adjustment.
View reportRenger, R., Keogh, B., Hawkins, A., Foltysova, J. and Souvannasacd, E. (2018) ‘Reworks: a robust system efficiency measure’, Evaluation Journal of Australasia, 18(3), pp. 183–191. Available at: https://journals.sagepub.com/doi/10.1177/1035719X18796611
This peer-reviewed article introduces “reworks” as a system-level efficiency measure grounded in Systems Evaluation Theory. It shows how evaluators can assess system efficiency in ways that go beyond narrow program outputs and better reflect the performance of complex systems.
View publicationKeogh, B. and Scally-Irvine, K. (2023) Changing the Theory of Change? Integrating change theories from other disciplines to standard ToC practice so we can better inform responses to complex situations and a climate-changing world, pre-conference workshop, Australian Evaluation Society International Conference, Brisbane, October 2023. Available at: https://conference2023.aes.asn.au/images/aes23_workshop_program2.pdf
This workshop examined how theories of change can be strengthened by drawing on wider traditions of change theory, especially in complex and climate-affected contexts where linear models are often too limited.
View workshopRenger, R., Keogh, B. and Atkinson, L. (2024) Systems Evaluation Theory – Practice & Implementation, online workshop, Australian Evaluation Society, February 2026. Available at: https://www.aes.asn.au/evaluation-learning/professional-learning-events/february-workshops/workshop-systems-evaluation-theory-practice-implementation-1-8-may-2024
This professional learning workshop focused on translating Systems Evaluation Theory into practice and helping evaluators apply systems concepts in concrete evaluation settings.
View workshopRenger, R., Keogh, B. and Atkinson, L. (2019) ‘Applying Systems Evaluation Theory (SET)’, paper presented at the Australian Evaluation Society International Conference, Sydney, September 2019. Available at: https://www.aes.asn.au/annual-conference-papers/2019-aes-international-conference-sydney-presentations
This presentation introduced Systems Evaluation Theory and showed how it can be used to define system boundaries, identify systemic inefficiencies and generate stronger evaluation insights for complex interventions.
View presentationKeogh, B. and Maloney, J. (2025) ‘The Dynamics of Influence: Evaluating Social Power’, paper presented at the Australian Evaluation Society International Conference, Canberra, September 2025. Available at: https://www.aes.asn.au/images/images/stories/files/2025_Conference_Canberra/aes25_Keogh.pdf
This presentation explores practical ways to analyse social power in evaluation, with a focus on mapping influence, understanding power relations and assessing whether those relations are changing over time.
View presentationElliott, J., Hawkins, A., Keogh, B. and Scally-Irvine, K. (2024) Demystifying Critical Systems Heuristics, AES Systems Special Interest Group seminar, February 2024. Available at: https://www.aes.asn.au/evaluation-learning/professional-learning-events/special-interest-group-sig/seminar-demystifying-critical-system-heuristics-2-february-2024
This seminar introduced Critical Systems Heuristics as a practical way for evaluators to question assumptions, test boundary choices and examine who is included or excluded in evaluation framing.
View seminarScally-Irvine, K., Elliott, J., Hawkins, A. and Keogh, B. (2025) Panarchy – How do complex systems change and persist over time?, AES Systems Special Interest Group seminar, June 2025. Available at: https://www.aes.asn.au/evaluation-learning/professional-learning-events/special-interest-group-sig/aes-systems-sig-panarchy-how-do-complex-systems-change-and-persist-over-time-online-13-june-2025
This seminar explored panarchy as a way of understanding cycles of stability, disruption and renewal in complex systems, and considered the implications for systems-informed evaluation.
View seminar
Four practice areas. One systems lens.
Have a similar problem?
If you have a particular situation you would like to discuss, please get in touch.
Contact Cobalt59