Evaluation of the RSQM
Between June and October 2013 the RJC carried out a test of the RSQM. The objectives were:
- To test whether the Restorative Service Standards are appropriate, ensuring safe and effective practice and allowing for local innovative practice.
- To assess the impact of the RSQM on restorative services in terms of achievability and resources.
- To test the extent to which criminal justice agencies can evidence indicators 1.2, 2.5, 5.1 and 6.5 which relate to the safeguarding of victims.
An independent evaluation of the test was carried out by the Charities Evaluation Service (CES). The main findings of the evaluation were:
Overall, most service providers believe that the Restorative Service Standards and RSQM will improve the quality of their work.
The standards are comprehensive and achievable but the indicators could be improved.
In relation to the RSQM:
- There was unanimous support from respondents for the principles and emerging practice of establishing a sector-wide quality mark.
- Assessor training was of a high quality and was well received.
- Assessors acted as peer reviewers, with knowledge and experience of restorative justice practice, providing critical insights into the quality mark process.
- Some services believe that the RSQM provides an endorsement of their current practices.
- The quality of evidence provided to support the standards was variable and more work will be needed to ensure relevance and consistency.
- The online application process is user friendly, easy to navigate and well supported with strong guidance and timely feedback.
- Assessors and service providers highly rated the RJC in terms of training, responding to enquiries, feedback and site visits.
- The costs of participation – particularly in terms of time taken – could be prohibitive for smaller projects.
- The RJC will need to clarify the use of the Restorative Service Quality Mark ‘brand’ (when used by whom and how) and the ongoing process of quality control, review and validation.
The CES's report also identified a number of recommendations which are listed below, followed by the action the RJC has taken.
Standards and indicators
1. In places, the standards need to be reworded and in some cases expanded to make sure that they provide a full description of the issues to be controlled and managed by restorative service providers. The indicators require some reworking – in some cases they do not adequately describe what is in the standard, or are not challenging enough to assure quality.
The RJC has reviewed and revised the standards and indicators.
Best Practice Guidance for Restorative Practice (2011)
2. The guidance should refer to 'good' practice rather than the more aspirational 'best' practice. Good practice is understood as the minimum requirements for safe and effective working. The guidance should be amended to clarify its use for individuals and projects.
The RJC will consider these recommendations when the guidance is next reviewed.
RSQM: Health check
3. During the test there were intermittent problems accessing the site and technical issues that need to be addressed.
Technical issues were addressed when they arose during the test. The RJC carried out ongoing tests as amendments to the site were made. It is noted that access to the RSQM website needs to be improved for prison establishments.
4. On the site, guidance could be improved. An explanation of why each indicator has been included and a brief explanation of what good practice looks like at each of the health check grades would give people a better sense of whether or not they are providing too little or too much information.
The RJC needs to provide an explanation of why each indicator was included and consider further the inclusion of good practice examples following feedback from the RSQM assessor team.
5. Assessors and restorative service providers should be offered the opportunity for a 'walk-through' training session on accessing and using the online system prior to the first site visit. (It is worth noting that they already have an assessor demonstrating the system at the first site visit.)
Introducing such sessions would increase the cost of the RSQM. A demonstration of the online system is included in the first site visit, there are online help videos and the RJC can provide telephone support.
RSQM: Assessors
6. The role of the assessor needs clarification. There needs to be a clear distinction between support and scrutiny.
The RSQM ethos is to be supportive to agencies applying for the quality mark. The RSQM assessors are not inspectors. Guidance to assessors has been amended to further clarify the amount of support that can be given and what constitutes consultancy.
7. The assessors need to be recognised by stakeholders as independent, objective peer reviewers.
This recommendation has been actioned.
8. Assessors should set the agenda for site visits.
Agendas for site visits are a collaboration between the agency and assessor. Applicant and assessor guidance has been improved.
9. Assessors need to have a free hand to investigate any aspect of the restorative service that is considered relevant to RSQM standards.
The RJC has improved the guidance to assessors and applicants.
10. Assessor final reports need to be more comprehensive. They should reference the evidence needed from the health check stage, identify measures needed to address gaps and cross reference this with an account of how the project responded to the first site visit.
Guidance to assessors has been improved and report forms amended.
RSQM: Restorative service providers
11. To improve providers' engagement in and understanding of the RSQM process, and the quality of the evidence they provide, providers applying for the RSQM should attend at least one induction session to be given an introduction to the principles, policies and procedures underpinning the RSQM process.
The introduction of such sessions would increase the cost of the RSQM. This is included in the first site visit and the online guidance. The RJC will also be hosting a number of launch events.
RSQM: Planning
12. Planning tools should be developed – particularly for first and final site visits – so that all parties are aware of their respective roles and responsibilities.This should include an evidence planner ahead of each visit to provide projects with clear guidance on the indicators needed to demonstrate achievement for each standard.
The RJC has developed further guidance for assessors and applicants including an evidence planner.
13. Although the guidance contains a useful glossary of terms we propose that RSQM applicant projects will need to demonstrate their understanding of restorative practice at the first site visit.
Guidance to applicants and assessors has been updated.
14. The site visits need careful planning with a clear agenda, a timetable of activities and a statement of expected outcomes.
Guidance to applicants and assessors has been updated.
15. Part of the planning could include provision for witnessing the restorative process – conferences or interviews – so that assessors will be able to judge the application of policies to practice.
Guidance to applicants and assessors has been improved although it is noted that this may not be possible for every application. For example, an intervention may not take place on the day of the final site visit or the participants may not consent to an observer.
16. To help organisations prepare for future RSQM process, they should be provided with more guidance as to what the site visits will entail; as they already get an Applicants' Handbook this may need reviewing to ensure it is comprehensive.
The RJC has updated the Applicants' Handbook.
17. It would be preferable for assessors to have the project's evidence in advance of their final site visit.
Assessor and applicant guidance has been updated.
18. Where a project has referral and/or case load procedures, these should be tested through the use of random sampling which is appropriate to the scale of restorative work.
Guidance to applicants and assessors has been updated.
Sector characteristics
19. The RSQM needs to reflect the diversity of the sector. We recommend the development of a generic RSQM which, where appropriate, also allows for supplementary standards to be introduced.
The RJC will consider this recommendation further as part of the review of the RSQM. Examples of potential evidence have been included in the guidance which are more sector specific.
20. The RJC needs to determine what qualifies as a restorative service by 'scale' so that distinctions can be made between light touch and more significant approaches.
Guidance to assessors and applicants has been updated.
21. The RSQM and the standards should be reviewed, perhaps after three years.
This will form part of the RJC's cyclical review process.