For agencies that are relatively new to program evaluation, Evaluation Capacity Building Grants provide the time, resources and support needed to develop an effective evaluation framework – the critical foundation upon which any successful evaluation must be built. In partnership with the Centre, more than 20 grant recipients are developing customized sets of program evaluation tools including logic models, outcomes indicators and measures. Mutual Support Systems (MSS) in Niagara is one of those agencies. Though no strangers to the idea and practice of program evaluation, MSS leaders recognized a need to develop a stronger foundation upon which they could build a sustainable program evaluation infrastructure. Read about their projects and progress here.
With an appropriate evaluation framework in place, agencies can move on to implementing the plan that will deliver the results and feedback they need. The Centre’s Evaluation Implementation Grants are supporting more than a dozen organizations that have clearly demonstrated they have the knowledge, skill and capacity necessary to evaluate an existing program or collection of services. In the Eastern region, four agencies are joining forces to evaluate a pilot program that delivers school-based services and parent training for children and families at high risk of developing emotional and behavioural problems and/or complex needs. You can read about this cross-sectoral, multi-site evaluation effort here.
In program evaluation, getting started is often the hardest part. In order to build a lasting culture of evaluation, it’s important to ensure we are asking the right questions, collecting the right information and using the evaluation results to inform ongoing service planning and delivery. Having an evaluation framework that is clear and flexible enough to stand the test of time reduces the amount of financial and human resources needed to sustain ongoing program evaluation efforts, and that’s what was missing at Mutual Support Systems (MSS) Niagara after the agency’s first foray into program evaluation in 2006.
“The first project did not provide the groundwork necessary to facilitate ongoing program evaluation,” says executive director David Schulz. “The primary barrier has been the lack of a fully developed program logic model, staffing resources, and the lack of a structured process to regularly survey stakeholders to inform the evaluation process. Without these components in place the evaluation process is too demanding on limited financial and staffing resources.”
The residential care and treatment program at Mutual Support Systems includes a network of four group homes staffed by treatment foster parents, child and youth workers and case managers. In partnership with the District School Board of Niagara, MSS also operates three low-enrollment specialized classrooms for those children who can not attend community schools due to learning and/or behavioural challenges.
Many of the children MSS serves have debilitating emotional and behavioural symptoms stemming from the trauma of past physical or sexual abuse. They have weekly access to psychotherapy through a partnership with a local counselling agency.
Evaluating such a complex program with a diverse group of stakeholders requires careful planning and extensive communication. MSS is working with the Centre to fully develop its program logic model and evaluation framework. They have identified and engaged all key stakeholders and begun to build lasting communication activities so evaluation results and needs can be shared among the group.
“The program logic model will be linked to the child assessment regime already in place in the program to measure individual child outcomes,” says Schulz. “The focus will be on measuring client-specific outcomes and using those to evaluate program effectiveness, determine areas needing improvement and measure changes in effectiveness over time.”
It wasn’t a simple task. They were working with a very large geographic area, limited human and financial resources and a bilingual population. Cooperation in service delivery has many benefits, but it can also add a layer of complexity that must be managed carefully.
The agencies formed a pilot project steering committee, with members from each of the four agencies and representatives from MCYS and the Ministry of Education. The team reviewed the literature, and ultimately settled on two evidence-based programs that fit the bill – Goldstein’s Skillstreaming and Cunningham’s Community Parent Education program (COPE). Both interventions will be delivered in school or at home as appropriate.
“The selection of the COPE and Skillstreaming programs was based on its relative ease of implementation, bilingual format and non-intensive training,” says program consultant Guy Theroux, who was hired to coordinate the implementation of these programs.
The pilot project began in January 2008, and it is scheduled to end in June 2009. The future prospects of the program will depend on the results of a large-scale evaluation currently underway with the help of an Evaluation Implementation Grant from the Centre. Their efforts will have both immediate and lasting benefits.
“Through the implementation of the pilot project, partnering agencies will be gathering and submitting data by means of the measurement tools,” says Theroux. “An outcome of the pilot project is to increase the capacity of each agency to maintain the evaluation activities beyond the pilot stage. Following the pilot project timeline, a lead agency will be identified in order to host and manage evaluation activities on behalf of the partnering agencies.”
In order to foster a shared understanding of the evaluation and ensure that findings are actually used to improve the program, the group is taking a utilization-focused approach that involves continuous engagement from all stakeholders. This gives each partner a sense of ownership of the evaluation findings, reinforcing their commitment to put them to good use.
Overall, there was general consensus among primary stakeholders that this evaluation was an important investment, and they were committed to including all stakeholders in the process throughout all stages of the evaluation,” said Theroux.
Do you have a topic to suggest for Focal Point? Send it to firstname.lastname@example.org.
Ontario Welcomes Child and Youth Mental Health Champion from South of the Border
There is little disagreement that child- and youth-centred mental health care makes sense for Ontario. The Ministry of Children and Youth Services, the Provincial Centre of Excellence for Child and Youth Mental Health at CHEO (the Centre) and service providers from all sectors around the province are working hard to ensure our evolving system delivers optimal results for those who matter most – children and youth.
It might be easy to talk the talk, but Dr. John Lyons has spent his career developing the tools we need to walk the walk. In July 2008, Lyons joined the CHEO team and became the inaugural endowed Chair in Child and Youth Mental Health, a post co-supported by the University of Ottawa, CHEO and the Centre. John comes to Ottawa via Chicago, where he was a professor of psychiatry and medicine at Northwestern University and the founding director of the school’s mental health services and policy program.With a mantra of keeping the focus on the child – not systems, service providers or even costs – Lyons has helped countless communities develop common visions and build systems that really work. His ideas may not seem revolutionary at first, but they present a fundamental challenge to how child and youth mental health services ought to be delivered and evaluated.
“Many competing pressures make it difficult for us to always do what is in the best interests of children and families,” says Lyons. "While it is impossible to eliminate these tensions, it has been my experience that by finding ways to always inform decisions based on the needs and strengths of children and families at all levels of the child-serving system, improved outcomes are achieved.”
Lyons has published nearly 200 peer-reviewed articles and five books, including Redressing the Emperor: Improving our children’s public mental health service system. In this landmark publication, Lyons describes his Total Clinical Outcomes Model (TCOM) and its associated assessment tool dubbed Child and Adolescent Needs and Strengths (CANS).
For several years, the Children’s Hospital of Eastern Ontario (CHEO) has been using both TCOM and CANS to organize, deliver and evaluate its mental health services and those of its community-based partners. Lyons has been a key player in the transformation process.
“I have had a long-standing relationship with CHEO and the Centre,” he says. “I have always been impressed with the commitment of CHEO to high-quality clinical care and the efforts of the Centre to promote evidence-based approaches to improving the lives of children and families.”
Dr. Purnima Sundar
The Centre is thrilled to welcome Dr. Purnima Sundar, who joined the Centre as a research and knowledge exchange consultant in May 2008. Purnima brings several years of experience doing community-based, participatory action research and program evaluation in the areas of youth and mental health, and diversity/multiculturalism. Here at the Centre, Purnima consults with organizations on evaluation and research, offers support to groups engaged in community mobilization efforts, and is involved in a range of activities to support knowledge exchange. Purnima holds a master's degree in community psychology and a PhD in social work - both from Wilfrid Laurier University. She teaches in the school of social work at Carleton University and serves as a member of the Board of Directors at South East Ottawa Community Health Centre.
The Centre’s growing research and evaluation team has been bolstered with the arrival of research associate Tanya Witteveen. She has experience in coordinating clinical research in child and youth mental health as well as interdisciplinary teaching experience. She has a background in clinical and sport psychology and has experience with both quantitative and qualitative research methodology. Recently, Tanya completed her first half-Ironman race in Ottawa. At the Centre, she is consulting with Evaluation Capacity-Building Grant recipients as well as helping other agencies with the planning, design and development of their program evaluations. Tanya holds an undergraduate degree from McGill University and a Master’s degree from the University of Ottawa.
Heather Woltman is a recent addition to the Centre and is looking forward to working as a research assistant with the evaluation and research team. Heather graduated from Queen’s University with a B.A. (Hons.) in psychology and has experience in organizational development from her previous position as a consultant with TouchStone Solutions Ltd. At the Centre, Heather will be working with the evaluation and research team to support agencies across the province in their program evaluation efforts. She will also be helping to maintain and expand Yoo Magazine, an online health literacy resource for youth developed by senior scientist Dr. Darcy Santor.
The Centre is proud to announce the appointment of Colleen Hannewyk as acting director of strategic planning and operations. Colleen began a 14-month term in June 2008 and takes over for Amy Boudreau who is currently on maternity leave.
Colleen has a Master's degree in business administration and more than 18 years of management experience in community-based health care and social programs. Among some of her positions, Colleen has served as Best Start project manager in the County of Lanark Social Services, executive director/community planner for HIV/AIDS Regional Services in Kingston, and manager of children's services for the United Counties of Leeds and Grenville. She also has extensive experience in early childhood education. Colleen's community perspective and appreciation of integrated services and evidence-based practices are definite assets to the Centre.
COMMUNITY MOBILIZATION AWARDS
Following a successful pilot program in 2007-08, the Centre is ready to accept applications for its innovative Community Mobilization Award from motivated communities across the province. A limited number of awards will help groups of service providers, organizations and other stakeholders collaborate to address a pressing local issue. Successful applicants will receive up to $30,000 over a one-year award term. During that time, partners will work together – and with the Centre – to develop and implement a community-owned action plan that is feasible, sustainable and grounded in the best available evidence. It is anticipated that these awards will leverage both in-kind and financial resources from partners and other sources.
Communities considering applying for a Community Mobilization Award are required to contact the Centre prior to submitting an application. A consultant will assess your readiness to engage in a collaborative project and help you to complete your application. The application deadline for the 2008-09 Community Mobilization Award is January 12, 2009. Detailed information about the award and application process is available online.
The Dare to Dream Program has helped dozens of youth raise awareness of mental health issues in their schools and communities. Grants of up to $5,000 are awarded to successful applicants and their adult mentors, and are available to Ontario schools, community agencies and youth organizations with an interest in child and youth mental health. Past projects have included plays, magazines, films, websites, workshops, awareness weeks and dance performances.
The deadline for the next round of Dare to Dream awards is on October 31, 2008. We encourage you to promote this valuable opportunity within your community networks and among your partners. For information about Dare to Dream, visit www.daretodreamprogram.ca.
Upcoming deadlines for the grants and awards program …
For more information about these or other funding opportunities visit us online at www.excellenceforchildandyouth.ca.
Evaluation Info Goes Online
In May 2008, 20 Ontario service providers were trained as champions for the new and innovative Med Ed tool. Since then, most participants have held further training sessions in their home agencies and the word about Med Ed is traveling fast. Due to popular demand, the Centre is planning to hold a second training session for a new group of Med Ed champions. We are looking for service providers from all regions who are interested in strengthening the way they partner with youth to manage medications and make decisions about their use.
HealthCheckPlus was designed by Dr. Darcy Santor, a senior scientist at the Centre, to facilitate the timely screening and assessment of a variety of health and mental health difficulties with no additional burden to physicians’ time. The system is implemented on a touch-screen kiosk that can be set up in a clinic waiting room or examination rooms. Patients complete a five- to 10-minute on-screen validated questionnaire, and the system immediately generates and prints a one-page report for the physician highlighting symptoms, risk factors and identified needs. HealthCheckPlus will also print treatment guidelines that are related to the screening results.
HealthCheckPlus is currently being pilot tested at several Canadian locations, and a large-scale trial in New Zealand is on the horizon.
On the Road Again
The Centre is gearing up for another exciting year of regional conferences. Planning is underway across the province, and several dates have been confirmed for the fall and winter. This year's meetings will feature both introductory and advanced sessions on program evaluation, and will include opportunities for both informal and formal networking. Each conference will highlight the work of local grant recipients, who will be on hand to share their successes and challenges with meeting participants.