After your diversion program is up and running, how do you determine if it is a success? The most basic measure of success is whether or not your program is meeting the objectives you’ve set for your diversion program. (In previous articles, we've gone over identifying diversion program goals and achieving those goals.) Even if you've identified and seemingly achieved the goals you've set, it's always useful to revisit them as part of the evaluation process.
There are some essential elements to measuring your diversion program outcomes, and it all comes down to the quality of the data you've collected and how you interpret it.
Data Collection
Determining the effectiveness of your diversion program is predicated on the quality of the data you have, and your conclusions will only be as complete as your access to data that encompasses relevant demographics, key statistics, and operational information. Collecting diversion program data begins at the point a case is identified as diversion-eligible and continues until the final disposition of the case. Some data points to consider include:
Percentage of overall caseload referred to diversion
Number of cases diverted by penal code
Percentage of diversion eligible offenders enrolled/failed to enroll in the diversion program
Reasons why offenders failed to enroll in the diversion program
Percentage of offenders who complete/failed to complete the diversion program
Reasons why offenders failed to complete diversion
Additional information, such as the number of staff hours saved and corresponding cost savings to your office, are important data points to capture for grant reporting and funding discussions.
Demographic data has become increasingly important in diversion program administration to address and evaluate equal access issues and disparate treatment of marginalized groups. Capturing this data at the point of case intake will help determine whether there are issues with the referral process or enrollment rates that may be impacting the success of your program.
Robust diversion programs with case management will also be able to track the accuracy of offender contact information, how often and when the participant was contacted by phone, email, text, or by mail both by individual and aggregate data for the program as a whole.
Data Extraction
Naturally, it is not enough to just gather program data. You must be able to extract and compile that data into a useful, standardized format. This brings us to the need for a comprehensive case management system that can produce reports, run data queries, and contain complete case files. A complete case management system, such as the proprietary system we offer at CorrectiveSolutions, will record dates, calculate deadlines, keep track of documents, and contain call logs, allowing you to evaluate outcomes measured against your goals.
Analyzing the Data and Identifying Pitfalls
Once you have calculated relevant statistical information, what does it all mean? What does it mean that a program has a certain completion rate? What does it mean if enrollment is low? If the numbers do not match your expectations, or if the program is not helping you reach your goals, then it is time to put the numbers in context, and identify where the program is lagging.
Here are some potential pitfalls:
Bad contact information: Does the initial referral have enough current information to be able to locate and reach the offender?
Insufficient efforts to reach the offender: Even willing diversion program participants require a lot of handholding throughout the process. It's often the case that increased communication correlates to higher completion rates.
The conditions are too complicated or difficult to achieve: the participant may decline diversion if it is too difficult to complete, or they are unable to finish all conditions before their deadline date. Are there simply not enough providers in the community for the number of participants that need it? Should the diversion program be longer?
Hidden costs: some program requirements may have additional costs that are not included in the program fee, such as third-party drug testing or counseling. It would appear that the program is affordable, but in actuality it is too costly.
Low case referrals: prosecutor turnover or lack of follow-up training can cause a diversion program to be underutilized. Are the program’s low completion numbers due to a lack of case referrals?
Since every diversion program is different, a key to determine its ongoing performance is consistent data collection over time. If the enrollment or completion rates are trending lower over a certain time period, correlation data may help to explain or identify its cause and further probed for causation.
Additional Sources of Data
Some aspects of diversion need to be evaluated using more than just statistical data and can be found by accessing information contained in case notes, program documents, and from the conditions of diversion themselves. For example, how do you measure how well the program holds offenders accountable for their actions? One aspect may be a journal or workbook as part of the behavioral change offender education condition, or perhaps the information shared through case management calls and/or meetings. These activities help guide the offender through the process of reflecting on their behavior and actions.
If your program is managed by a high-quality diversion program administrator, they should have the ability to offer post-diversion surveys that probe the effectiveness of the program and the offender’s insights as to their accountability for their actions. Other conditions that are recorded are community service, apology letters, and victim impact panels, all of which provide the opportunity to analyze an offender's feelings of personal accountability.
Lastly, measuring whether the diversion program is affordable or accessible to offenders regardless of their economic status is important. Indigent offenders or offenders below the poverty line will often stop participating in the program if it is too costly. Indications that the program is unaffordable for some may be that they complete all or some conditions except for the fees or an absence from conditions that have costs like drug testing (i.e., did they fail the drug test condition due to absence, or was it truly a dirty test?). Consider recording failure reasons at the end of the case that can be statistically tallied and analyzed alongside case notes to determine whether your program has a cost issue.
Measuring outcomes is essential for every diversion program. Even if the results indicate your program is not as successful has you had hoped, proper data collection and analyses will provide guidance for making changes that will lead to future success.
How Does Your Office Measure Success?
If you would like to discuss the successes and challenges that come with achieving your diversion program's goals, or are looking to partner with a program provider that can ensure success without being a drain on your office's resources, visit our contact us page or contact Thomas Jonsson directly using the information below.