Evaluations and impact studies


End of programme impact evaluation of the capability development assistance programme with National Qualifications Authorities (Samoa and Tonga), New Zealand Qualifications Authority and Ministry of Foreign Affairs and Trade, October 2016 – March 2017

Samoa parliament building
Samoa parliament building

We completed work for the partnership programme on behalf of the NZQA and the Ministry of Foreign Affairs and Trade, in accordance with their quality standards. The evaluation team used Rapid Evaluation Appraisal Method (REAM), a mixed methods approach for data collection and information gathering in New Zealand, Samoa and Tonga. We utilised a case study approach to assess what was intended overall and then compared what happened in Samoa and Tonga separately. This enabled the evaluation to review how the programme has worked in different settings.

 

Pacific Technical Assistance (PacificTA): Evaluation of the Local Government Technical Assistance Facility for Pacific Island countries, 2016

We completed an evaluation of the Local Government Technical Assistance Facility for Pacific Island Countries (PacificTA). This is an innovative approach to delivering Technical Assistance providing expertise from local government in New Zealand to Pacific Island countries.

The evaluation purpose was to review the implementation of the facility to date, as well as to inform the future design of the facility. This work involved extensive stakeholder engagement via key informant interviews, including fieldwork in the Cook Islands, Samoa, and Kiribati. An online stakeholder survey was also used.

The PacificTA was established in June 2012 and currently ends May 2017. The initiative was developed in response to learnings from New Zealand’s Aid Programme, the Commonwealth Local Government Forum (CLGF) regional Pacific programme and New Zealand local councils’ engagement with Pacific Island Country (PIC) governments. Some New Zealand local councils had formal relationships with PICs and had received requests for technical assistance to support local government in PICs. The specific need identified in PacificTA activity design was a gap in PIC technical expertise to form, develop and deliver appropriate solutions to service delivery at the local level.

PacificTA provides technical support to local authorities across a variety of service provision (e.g. solid waste and water management, urban planning, asset management, strategic planning) where local expertise gaps exist. Most technical assistance assignments are short term, but there are also periodic longer term PacificTA engagements. PacificTA was designed to complement the bilateral support provided under the New Zealand Aid Programme to Pacific Island Countries. Eligible countries include Cook Islands, Tonga, Niue, Samoa, Vanuatu, Solomon Islands, Papua New Guinea, Tuvalu, Tokelau and Kiribati. Fiji became eligible in 2015.

https://www.mfat.govt.nz/assets/Aid-Prog-docs/Evaluations/2016/June-2016/LGNZ-evaluation-report-June2016.pdf 

Smart Cities – Smart Nation project: helping to make our cities smart, safe places for us to live, work and play in, 2016 – 2017

Smart Cities - Water Pump
Water quality sensor

An iterative evaluation of the Smart Cities programme for Land Information New Zealand (LINZ). Funded out of the Treasury Better Public Service Seed Fund, the programme piloted a series of innovative smart-sensing technology projects in joint partnership with Christchurch, Wellington, and Auckland city councils.

An iterative developmental evaluation approach was adopted to support the effective capture of lessons learned throughout the programme, involving a close working arrangement with LINZ and the Territorial Authorities involved.

At the end of the programme the team made overall assessments on the programme’s relevance, efficiency, effectiveness, and sustainability, as well as how effective it was at generating informative lessons. This information was critical to inform future decision-making at both local and central government level on the adoption of these smart technologies. For information on the development of the technology behind transforming Wellington  (YouTube), on the project from LINZ. Click here to read our  Smart Cities Case Study for more information on how we used our ELM to support this project.

Eastern and Central Community Trust (ECCT) – Wairarapa Rural Education Activities Programme – Maths is Fun Programme Evaluation, September 2016 – March 2017

The evaluation team critically assessed the Maths is Fun programme to determine its viability for future delivery of outcomes in relation to return on investment and ECCTs strategic plan. We evaluated how effective, efficient and appropriate the current framework is, working constructively with key stakeholders to ensure the evaluation was conducted in a non-disruptive way. The evaluation also identified opportunities to build the capability of different stakeholders, so that relevant elements of the evaluation framework developed as part of this work, could be built on as part of the ongoing monitoring and evaluation of the programme.

Te Ataarangi He Kāinga Kōrerorero, Te Ataarangi, August 2016 – February 2017

This is a te reo Māori programme run by Te Ataarangi, focused on the revitalisation of te reo Māori in homes across New Zealand. The team completed a collaborative aromātai/evaluation to assess the effectiveness of the programme and to co-design a model which could be used moving forward. The purpose of the aromātai/evaluation was to evaluate He Kāinga Kōrerorero’s contribution to stakeholder goals. The aromātai/evaluation also evaluated the effectiveness of the programme over the last year and identified ways to tailor future inputs. We co-designed with Te Ataarangi as a user-friendly model and developed an online progress tracking system that mets the requirements of the programme and the needs of Te Ataarangi.

Auckland Council, Evaluation of the Sports Partnership Project, Project Lead, January-December 2015

We worked with Auckland Council to evaluate a pilot Sport Partnerships Project (funded by Sport NZ). The evaluation included stakeholder interviews with internal and external stakeholders, and support to the project manager to co-deliver the evaluation. Final judgements were made against the OECD-DAC evaluation criteria. A key focus of this work was also to develop the evaluative capability of the project manager’s own skills in evaluation.

New Zealand Fire Service (NZFS) – Evaluation of the Home Fire Safety Check Programme Evaluation, 2014 – 2015

Fire Service Home Safety - Results ModelThis project evaluated the success and effectiveness of the Home Fire Safety Check (HFSC) programme for reaching and affecting at risk groups. The project assessed the effectiveness of the HFSC in relation to its contribution to the Fire Service Commission’s strategic outcomes. We developed and implemented the evaluation plan which included: conducting organisational data analysis, stakeholder interviews in the field, and a client programme beneficiary survey.

These information sources were analysed and presented at a findings workshop where key NZFS personnel were able to discuss and evaluate the findings of the research. Points from this discussion were integrated into the final report which ensured it was relevant and fit for purpose, providing specific and attainable recommendations for the Fire Service to consider. The team worked collaboratively with the Fire Service’score team and were able to deliver the project on budget and within the timeframes desired by the NZFS. Moreover this project demonstrates the teams commitment to working flexibly and collaboratively with NZFS stakeholders and staff.

Accident Compensation Corporation – evaluation of Alcohol Screening and Brief Intervention Trials, 2014 – 2015

We were commissioned to perform a process evaluation of a Screening and Brief intervention (SBI) initiative for the ACC. The purpose of the evaluation was to provide ACC and other agencies with an independent assessment of the relevance, efficiency, effectiveness and sustainability of the SBI process at Wellington Hospital Emergency Department. The findings informed their decision whether it should be continued and rolled out to other settings and what the conditions, requirements and possible strategy for such roll-out might be.

Ministry of Pacific Island Affairs (MPIA) – Achiever’s Project Evaluation, 2014

We worked with the Ministry of Pacific Island Affairs to evaluate the effectiveness of the Achievers Project, involving online media to enhance student learning. Nineteen secondary schools produced two-minute videos to profile the achievements of Pacific students. The videos were posted on YouTube and received 40,000 views shortly after posting. The number of viewings have increased significantly in recent times. Schools and students considered the initiative an enormous success and hope it will be repeated in the future. We worked with stakeholders to collate evidence of successful outcomes particularly the contribution of the Achievers project to the confidence of Pacific students.

Land Information New Zealand – Evaluation of the Ratings Valuation Regulatory Framework, 2013

LINZ Review of Occupational Regulation of valuers Model

LINZ commissioned an evaluation to assess the effectiveness of the Rating Valuations Regulatory Framework against its original objectives, and to assess its viability for current use as well as for new or emerging purposes. The Rating Valuations Regulatory Framework is the law, policies and related processes and procedures that regulate the valuation of property by local authorities to set rates. The aim of this work was to ensure that the framework is fit for purpose. This may require changes to law (Rating Valuations Act 1998 and related regulations), policies and related processes and procedures. The evaluation team undertook the evaluation and reported its findings to LINZ on 28 June 2013. The evaluation finding informed LINZ’s policy work aimed at improving the Rating Valuations Regulatory Framework. That work will start in the second half of 2013 and will include a public consultation process. For further information, visit the LINZ website.

Whanau Pakari, June-September 2013

The evaluation team completed an evaluation of Whanau Pakari; a unique multidisciplinary intervention programme tackling child and adolescent obesity in Taranaki. The evaluation informed the Taranaki District Health Board’s decision-making regarding on-going funding of this pilot programme. This programme and evaluation was unusual in that a clinical trial was also being undertaken.

New Zealand Customs – Evaluation of the Pacific Islands Leadership Programme, 2012

We worked with the New Zealand Customs Service to design an integrated evaluative monitoring system to track the results of a pilot training programme focused on developing Customs leadership capability in the Pacific. This included baseline, mid-point, and end-point data collection as well as capability building (one-on-one coaching and mentoring) to enable the internal team to collect regular monitoring data. The system provided a rich source of timely data for the on-going progress reporting, as well as informing the formal end-point evaluation. This work has helped inform decisions regarding the design and roll out of the programme more widely in the region.

Indonesian Water and Sanitation Hibah Evaluation – AusAID funded program, 2011

From 2010 to 2011, AusAID funded an innovative pilot of an outputs-based aid Water and Sanitation initiative in Indonesia. The evaluation process helped stakeholders achieve clarity in how the funding mechanism worked. It was also able to highlight some of the early positive impacts of the program to peoples’ lives. The findings and recommendations of the evaluation were incorporated into the design of phase two. To view the AusAID website, including the report and AusAIDs management response to the report click here.

Development West Coast 2010

We completed an outcomes evaluation of the West Coast Tourism Major Regional Initiative (MRI) for Development West Coast. This included an assessment of the progress (at inputs, outputs and outcomes/impacts) of the 21 projects funded by this initiative. These results were aggregated and reported against a high-level outcomes model developed to assess the overall achievements of the MRI. This evaluation assisted Development West Coast in their decision making regarding future funding. Click here for more information.

development wc

Papua New Guinea Law & Justice Sector – Provincial Engagement Evaluation Project, PNG/AusAID-funded, 2009

Australia Papua New Guinea Incentive Fund (APNGIF) – AusAID/Coffey International Development, Evaluation of APNGIF Support, 2008

Department of Labour – Evaluation of two Employment Relations Courses, 2007 –2008