Designing and implementing results frameworks

Advisory and ELM Platform services for the Otago-Southland Regional Transport Plan, June 2017 – current

The Otago and Southland Regional Transport Committees (RTCs), both committees of their respective regional council, are charged with preparing and reviewing a regional land transport plan for their region. The two committees work closely together and have combined the 2015-2021 Otago and Southland RLTPs into a single document with a common strategic front-end. Evalstars worked in collaboration with regional transport planning managers to help Otago Regional Council (ORC) and Environment Southland (ES) develop an evaluation scheme for the Otago Southland Regional Land Transport Plans (RLTPs), including:

  • An evaluative model to be able to measure the desired outcomes, key problems and benefits of the wider RLTPs
  • Expansion of the logic model to include bids put forward by the participating agencies and prioritised by the Committees for proposed projects bids
  • Wider project logic, connecting problems and benefits with high-level desired outcomes
  • ORC moving towards an outputs and outcomes logic programme model or similar.

We ensured the process of information gathering and decision making was guided by the intent to embed collaboration amongst key personnel. Our work was built on the prior work undertaken by the Regional Transport Committees on the RTLP objectives, and problem statements. As well as the increasing recognition that a more strategic approach planning, managing and reporting to meet customer requirements, leading to changes to the evolving transport system, increases the effectiveness and efficiency of operations management and risk management.

For more information, please visit the Otago Regional Council website.

Ginigoada PNG – Organisation-wide monitoring and evaluation framework (DFAT/World Bank), 2013 – current

Compressed Gini higher resolution
Graduate of a Ginigoada bus programme

Ginigoada is a charitable organisation operating out of the Port Moresby Chamber of Commerce and Industries (POMCCI). Ginigoada required a monitoring and evaluation system that covered their bus intervention programmes for unemployed youth in Port Moresby.

The team worked collaboratively to develop a programme model that operates across four bus interventions and tracks progress through to outcomes and higher-level job specific training. During the first stage of the project, we developed an innovative IT solution, the ELM Platform, to collect information and aid the synthesis of results.

EvalStars is currently working with Ginigoada on the second stage of this project, with funding from DFAT. For more information, visit the Ginogoada website.

High Level Strategic Input into Results Management System for Labour Mobility Assistance Program (LMAP), 2014-2015

This project included developing a framework under which the team could fully develop a Results Management System, this included a revised and agreed ToC. Additional risks were included in a revised Risk Management Plan (RMP), identified through the revision of ToC. Elaboration of the Results Management System (monitoring and evaluation framework) to align it with the revised ToC, including the key high-level impact, outcome and output indicators for the programme. An agreed collaborative engagement model was developed that embedded capacity building (ways of working).

This included an embedded evaluative management approach, integrated strategic and operational planning, capability building, monitoring, evaluation activities and assessment, reflective practice, and reporting. As part of an overall Results Management System approach identified the key data collection tools required for ToC and results measurement framework that could be adapted, modularised and scaled to meet potential future needs across an integrated embedded shared platform.

The Department of National Planning and Monitoring, Papua New Guinea, 2014

The team carried out a review of progress 2011-2015 with the Papua New Guinea Government’s Department of National Planning and Monitoring. The project aimed to bridge the medium-term development plan adopted in 2010 with the Development Strategic Plan 2030 and the Vision 2050.

The team supported data collection from over 38 government sectors to establish an accurate snapshot of where the country is now. The data informed the development of a Bridging Plan based on reliable data that sets realistic and achievable targets for the 38 sectors. The Plan will demonstrate the Minister’s determination to make a paradigm shift towards a greener and more sustainable path to development in PNG while still making progress towards the human development indicators as established by the United Nations.

The team and officers in the Department of National Planning and Monitoring used the ELM Platform to enter data, to store and share policy documents, to store interview data, miscellaneous articles and sources of information and outcomes models.

Department of Agriculture Fisheries and Forestry (DAFF) Australia – Australian-Indonesia Partnership Emerging Infectious Disease Program (AIP-EID), 2012 – 2015

Indonesia - AIP-EID

This programme, funded by AusAID and administered by DAFF, is a partnership between the Government of Indonesia and the Australian DAFF. The programme is designed to strengthen Indonesia’s ability to deal effectively with emerging infectious diseases. The team re-designed the evaluative monitoring system for the AIP-EID to meet AusAID evaluation, as well as the capacity building programme, monitoring and evaluation standards.

As a result of this review, the AIP-EID programme leaders can now track progress, demonstrate success, and use accurate data collected to plan future activities. The programme has strengthened national planning, improved the quality of laboratory and quarantine processes and increased staff capacity and capability in Indonesia and Australia. The benefit of this strengthening is a more secure border with better ability to respond to zoonotic diseases.

For more information on this project, visit the Australian Government – Department of Agriculture and Water Resources website.

Wellington Homeless Women’s Trust (WHWT), 2013-2015 – Pro bono work

In 2013, the Wellington Homeless Women’s Trust had established a new service in Wellington to provide secure, temporary accommodation and support for women who are in the process of moving to permanent accommodation. The Trust has private funding to establish and deliver the service over two years. The evaluation team provided independent evaluation expertise to support the Trust to gather information to inform the development of the project, understand the little-known need for secure accommodation and support for women, and track the difference that the Trust makes in the next two years. For further information on the great work the Trust is doing, please check their website.

Sporting Event System, Sport New Zealand, 2014

We worked with Sport New Zealand to help with developing an approach that would deliver a world leading sporting events system for New Zealand. In particular, we helped to distil the key components and processes of the model and also identify a complete set of indicators and measures to track progress and enable reporting.
The contribution of the team was invaluable in helping model the system attributes and processes and then identifying a measurement matrix that could be implemented over time. The application of this work is currently underway and portions of the thinking have been successfully applied elsewhere in the business.

Ministry for Primary Industries (MPI) 2014

We worked with MPI to develop a draft, partial measurement tables for 10 of the programmes that are currently underway as part of the Primary Growth Partnership. Working in collaboration with the Ministry and programme co-investors, The team helped clarify intended outcomes, develop some fit-for-purpose indicators, measures, data sources, and baselines to support ongoing monitoring and evaluation. MPI is continued this work as part of its overall management of PGP.

Design of Monitoring, Evaluation, Reporting and Improvement (MERI) System for Phase 2 of the Pacific Horticultural and Agricultural Market Access (PHAMA) Programme 2013–2014

We provided technical support to take forward recommendations from the Independent Review of the PHAMA Programme to strengthen MERI in line with DFAT Australian Aid Programme Evaluation Standards. The team facilitated the process to develop workable logic models to track results from input, through activity, to output and outcome levels for the multiple components of this complex programme. Specific attention was given to embedding ownership of results and MERI processes across multiple stakeholders. Competency development at the individual and organisational levels to realise this formed part of the MERI system.

For more information, please visit the PHAMA website

Statistics New Zealand, Evaluating learning and development initiatives – Capability and Change, 2012 – 2013

We provided senior advice and oversight to the development of an innovative framework to track the contribution capability development made to Statistics New Zealand. A unique characteristic of this work was the need to develop an approach that was standardised but flexible enough to be customised by any part of the business, for any form of capability development. The project involved developing practical tools to capture change in individual’s capabilities as a result of learning and development initiatives. The project also required an approach that would ‘tell the overall story’ of learning and development across the organisation on an on-going basis.

College of Mental Health Nurses – Mental Health and Addictions Credentialling in Primary Care (Nursing) Project, 2012 – 2013

We worked with the College of Mental Health Nurses to design and support the implementation of an integrated evaluative monitoring approach to track the progress and results of a new project to credential primary care (generalist) nurses. On-going capability development was a core focus of this project, with the team working closely with the project manager to ensure the College can sustainably track the performance of the project, and effectively communicate this to their stakeholders. For more information on the project’s approach to monitoring and evaluation click here. For more information on the credentialing project itself, click here.

Results Framework Specialist, Ministry of Education, New Zealand 2012–2013.

This project involved the design of a results focused evaluative monitoring framework to demonstrate results for children, schools, and families who receive their services. The team documented preliminary lessons from selected high performing schools, developed an evaluation plan for the Specialist Services Results Framework, and co-facilitated the modelling and indicator development workshops to design the results framework that form the basis of the monitoring and evaluation system. This includes providing technical advice for the aggregation of quantitative and qualitative outcome results reporting across four interventions with 30,000 learners, while taking into account the individual nature of each learner’s gauge of success; and providing Specialist Services with a results framework which went across four service arms.

M&E Advisor, Australia Indonesian Forest Carbon Partnership, November 2012-January 2013.

We provided specialist M&E advisory services to the Australian Indonesian Forest Carbon Partnership. This included the completion of an evaluability/needs assessment of the program for M&E purposes, and the completion of a draft M&E implementation plan that would meet AusAID M&E standards. The second phase of this work was not completed due to significant budget cuts and early closure of the program.

Statistics New Zealand, Transform Collections, May 2012 – May 2014

We supported Statistics New Zealand in the design and implementation of an evaluation strategy and plan to monitor and evaluate one large component of a 10-year change and transformation programme. This included facilitating the development of a programme model with a wide range of internal stakeholders, development of a comprehensive evaluation strategy and plan (including describing a clear process to review and refresh the M&E system), and intense capacity building of internal staff to grow their evaluation capabilities to ensure the sustainable implementation of the M&E system, and on-going peer review of data being collected and reported.

New Zealand Customs – Evaluation of the Pacific Islands Leadership Programme 2012

We worked with the New Zealand Customs Service to design an integrated evaluative monitoring system to track the results of a pilot training programme focused on developing Customs leadership capability in the Pacific. This included baseline, mid-point, and end-point data collection as well as capability building (one-on-one coaching and mentoring) to enable the internal team to collect regular monitoring data. The system provided a rich source of timely data for the on-going progress reporting, as well as informing the formal end-point evaluation. This work helped inform decisions regarding the design and roll out of the programme more widely in the region.

Department of Labour, develop and refine Settlement Unit reporting templates to align with RBA concepts, 2012.

We developed and refined a concise, focused framework of Settlement Services key results, indicators and measures. We also developed a results-focused blueprint for monitoring reporting progress, efficiency and effectiveness drawing on key areas. Developed and provided recommendations to implement a refined reporting template, and any enhancements to Settlement Services reporting practices.

New Zealand Fire Service – National Training, 2011 – 2012.

We supported National Training to develop an integrated system to better report on the contribution training is making to the NZFS. This included the development of a programme model and data-blueprint with a wide stakeholder group, advising on indicator selection to ensure a robust framework (mix of quantitative and qualitative indicators at inputs, activities, outputs and impacts/outcomes), and provision of in-house training on data collection and analysis.

Ola Fou – Praxis, 2011 – 2012

Praxis is a network of Christian practitioners who support the development of young people in their communities. The network operates across the Pacific, New Zealand, and Australia. We supported Praxis to develop and implement an innovative evaluative monitoring system to enable Praxis to assess the progress of one of their programmes – Ola Fou. Ola Fou trains community youth workers in six countries across the Pacific. The evaluative monitoring system included the use of innovative tools based on photos and diagrams to collect and report data in a way that was meaningful, ongoing, and appropriate to the context, whilst meeting the information needs of Praxis and their major funder, Ministry of Foreign Affairs and Trade. For more information visit their website.

Mountain Safety Council, 2011

We provided advice and support to the Evaluation Advisor in the development and implementation of a monitoring and evaluation framework for the organisation. The objective was to improve their ability to report on their impact as an organisation, and to support decision-making at the board level.

Ministry of Health – Innovations Team Health Workforce New Zealand, 2011

We supported the Innovations Team to move towards an integrated monitoring and evaluation ‘way of working’. This included the development of an overarching ‘team’ (program) model, the identification of a preliminary list of aggregatable indicators for individual innovations projects to include in the project level monitoring frameworks, and the development of guidelines to support the team to achieve their new ‘way of working’. A formal capacity assessment (and accompanying report) was also completed.

Youth Quest 2011

We supported Youth Quest to develop a simple and robust evaluative monitoring system, to enable them to track the progress of the troubled youth that enter their programme. This includes the tracking of students during the formal programme as well as afterwards when they are being mentored. The information in this system is generating supported Youth Quest to be more effective as an organisation, and to ensure they were making a difference to the youth they were working with. The evaluative monitoring system also assisted them in meeting the reporting requirements and information needs of their stakeholders.

Enterprise Challenge Fund – Coffey International Development. Monitoring and Evaluation Adviser. AusAID funded program. 2010-2011.

The evaluation team provided the monitoring and evaluation specialist services to the Enterprise Challenge Fund, a three year AusAID program piloting the use of matched-grants to business as a poverty alleviation mechanism. This included the design of an overarching ‘portfolio level’ framework, as well as a linked monitoring and evaluation framework for each of the 22 projects in 8 countries in Asia and the Pacific. Data was collected and reported by Country Managers, with advice and support provided by the evaluation team. Data was also aggregated and reported at the portfolio level on key priority areas. The team provided advice and support to Country Managers on making revisions and updates to their project models and frameworks, as well as completion of data analysis and reporting for the annual report. The team also provided supporting M&E specialist support services, working on all aspects of ensuring the smooth ongoing implementation of the framework, and included fieldwork and capability support to the in-country team in the Solomon Islands.

For more information, visit the Enterprise Challenge Fund website

Papua New Guinea Law & Justice Baseline Collation Project. AusAID-funded, 2008-2009

Papua New Guinea Law and Justice Sector Evaluation Design. AusAID-funded, 2008

AusAID – Quality at Entry Review, Australia Papua New Guinea Incentive Fund (Phase Three), 2008