Our team of specialists have been involved in a range of projects in New Zealand and internationally. Our current and past projects demonstrate the knowledge and skills the EvalStars team have in designing and implementing evaluative monitoring frameworks, strategic and operational business planning, evaluations and impact studies, reviews, research and performance measurement and management.
Evaluative services for the Otago-Southland Regional Transport Committee, June 2017 – current.
The Otago and Southland Regional Transport Committees (RTCs) (both committees of their respective regional council) are charged with preparing and reviewing a regional land transport plan for their region. The two committees work closely together and have combined the 2015-2021 Otago and Southland RLTPs into a single document with a common strategic front-end. Evalstars worked in collaboration with regional transport planning managers to help Otago Regional Council (ORC) and Environment Southland (ES) develop an evaluation scheme for the Otago Southland Regional Land Transport Plans (RLTPs), including:
- An evaluative model to be able to measure the desired outcomes, key problems and benefits of the wider RLTPs
- Expansion of the logic model to include bids put forward by the participating agencies and prioritised by the Committees for proposed projects bids
- Wider project logic, connecting problems and benefits with high-level desired outcomes
- ORC moving towards an outputs and outcomes logic programme model or similar.
We ensured the process of information gathering and decision making was guided by the intent to embed collaboration amongst key personnel. Our work was built on the prior work undertaken by the Regional Transport Committees on the RTLP objectives, and problem statements. As well as the increasing recognition that a more strategic approach planning, managing and reporting to meet customer requirements, leading to changes to the evolving transport system, increases the effectiveness and efficiency of operations management and risk management.
English Language Partners (ELP) – Developing an integrated evaluative self-assessment system, 2014 – 2015
Using the DOMETM approach we supported English Language Partners (ELP), a not-for-profit organisation providing English Language services to refugees and migrants, to develop an integrated evaluative self-assessment system that can be applied across 23 centres nationwide. We worked closely with the Chief Executive and an internal working group, as well as with ELP’s regional managers, the Board, their Ethnic Advisory Board, and external stakeholders to develop a meaningful and fit-for-purpose approach that will work for their organisation. We also completed a detailed independent ‘post-amalgamation’ baseline report for the organisation to capture the ‘state’ of the organisation following a national amalgamation. ELP have continued to use the system developed. ELP Outcomes model (PDF)
End of programme impact evaluation of the capability development assistance programme with National Qualifications Authorities (Samoa and Tonga), New Zealand Qualifications Authority and Ministry of Foreign Affairs and Trade, October 2016 – March 2017
We evaluated the partnership programme on behalf of the NZQA and the Ministry of Foreign Affairs and Trade, in accordance with their quality standards. The evaluation team used Rapid Evaluation Appraisal Method (REAM), a mixed methods approach for data collection and information gathering in New Zealand, Samoa and Tonga. They also utilised a case study approach to assess what was intended overall and then compare what happened in Samoa and Tonga separately. This enabled the evaluation to review how the programme has worked in different settings.
Post Project Review: Comprehensive Clinical Assessment (interRAI) in Aged Residential Care (2011-2015) – 2016
We undertook a post-project review of the Comprehensive Clinical Assessment (interRAI) Project, commissioned by interRAI Services, Central Region’s Technical Advisory Services Ltd New Zealand (TAS). The team assessed the extent to which the project met its intended objectives; identifying enablers and barriers; and identifying learnings and recommendations. The review also aims to inform interRAI Governance and managers, funders, implementers, and interRAI users, of project’s progress, to inform decision-making, and to meet accountability requirements. The team used quantitative and qualitative Rapid Evaluation Appraisal Methods developed for and used in evaluation of international development assistance initiatives. This mixed-method research design approach includes gathering information from multiple sources and includes a survey of facility managers from approximately 700 aged-care residential (ACR) facilities, and more in-depth interviews with a purposeful example of ARC providers. The review used a consultative approach undertaken in a professional and ethical manner and adheres to the Australasian Evaluation Society principles for ethical conduct for evaluations, and OECD DAC Quality Evaluation Standards.
- Lessons learned from the introduction of Comprehensive Clinical Assessment (interRAI) in Aged Residential Care – Final Report
- INsite Magazine article: interRAI Review: costs outweigh the benefits
- interRAI New Zealand website
Tongan Seasonal Workers Study, Cardno – Labour Mobility Assistance Program (LMAP), September 2016 – current
We worked collaboratively with our Tongan partners to understand the drivers for Tongan seasonal workers breaching visa conditions while contracted to Australian approved employers under the Seasonal Worker Programme. The study sought to identify potential ways and practical solutions to reduce the incidence of visa breaches through improvements to selection and preparation; employment processes in Tonga and Australia; linkages with and support from communities in Tonga and Australia; and other appropriate approaches. The research involved extensive consultation with government officials in Australia and Tonga, community consultation with the Tongan community in Tonga and the Tongan diaspora in Australia. It also included interviews with other relevant stakeholders. The research included a survey of Approved Employers and will be supported by an action plan for an integrated programme of activities in Tonga and Australia. The research report, which includes recommendations and action plan for both the Governments of Australia and Tonga, has been approved by the Australian
Government and Tonga’s Ministry of Internal Affairs. This will be workshopped with stakeholders in both Australia and Tonga in the coming weeks.
New Zealand Qualifications Authority (NZQA) – Workshop Series, 2016.
We worked with NZQA Assessment Division to develop and facilitated three workshops focused on building NZQA staff capability and capacity in evaluation methods. Workshops covered framework design and aligning evaluation methods; concept for working system and roles and responsibilities; and practical evaluation methods and tools, ethics and implementation capacity.