A Good Idea in Settlement & Inclusion

Evaluating your hybrid services, there's a toolkit for that

Posted on: May 29, 2023

The YMCA of the National Capital Region’s project, Wired: Evaluating Settlement Online (WESO), is a research-based program that addresses the challenges/barriers that service providers and newcomer clients face in delivering/using remote and hybrid services. The project is uncovering evidence-based practices that Settlement Service Provider Organizations (SPOs) find promising in delivering remote/hybrid services and engaging clients, creating a toolkit/framework and resources for SPOs to evaluate the effectiveness of your service delivery and adapt your models accordingly to improve services, and creating a network of knowledge and promising practice sharing.

The WESO team has created an evaluation tool for SPOs to assess the quality of your hybrid services and increase client intentions to use hybrid services. This toolkit also includes a data collection tool, in the form of a client questionnaire, that SPOs can use to collect client feedback.

WESO toolkit title page

Their Quality Assessment Toolkit for Hybrid Settlement Services provides a methodology and tools you can use immediately to evaluate the quality and effectiveness of your hybrid service delivery. It also highlights tips and best practices to support SPOs to adapt and improve your hybrid service delivery models. 

The WESO project defines hybrid services as:

Hybrid service refers to a service delivery mode where clients have the option to interact with service providers both offline or in-person and through online platforms. It combines the convenience of online access with the personal touch of in-person interactions, empowering clients to choose the mode that suits them best based on their specific needs at any given moment.

Why does it matter?

Like many SPOs, you may be struggling with questions like:

  • How do you know that your hybrid service delivery is effective?
  • What are you getting right?
  • What do you need to improve?
  • Most importantly, what do your clients think?

In February 2021 I moderated a panel that asked the question: How do I know my transition to virtual service delivery is effective? There were some great insights, tools, ideas, and discussions. 

But the question has hung over our sector during the pandemic and as we more or less formally shift to a hybrid service delivery model.

The WESO toolkit is helping to answer all of these questions (disclosure: I have a small consulting role on the project). 

Most importantly, the analysis you get from the Toolkit will provide you with specific areas where you need to take action to strengthen your service delivery and improve client performance. It will help you to prioritize your efforts, based on what your clients are telling you is important to them and where you perform or under-perform. 

Feedback from your clients is the most essential information you need to collect and analyze to ensure your services are meeting their needs and creating necessary outcomes.

The Toolkit will tell you where you need to improve. You'll know where you’re doing well. You'll also see where you may be over-focusing (i.e. you perform highly, but they’re not as important to your clients). As well, you'll identify areas that are not particularly important to your clients, where you don’t need to focus.

The WESO Toolkit and approach take the guessing out of your evaluation work. Replace anecdotal perception of what is working with evidence from Newcomer experiences to focus on what is actually working.

What do you need to know?

The WESO toolkit is based on the findings of extensive research and engagement with newcomer clients and staff at SPOs across Canada. 

  • A targeted literature review - 250 studies and reports from peer-reviewed publication venues, government guides, and surveys.  120 studies were chosen which underlined two major service delivery themes: conventional (traditional) service delivery and E-service delivery. 28 articles that include both models of service delivery were chosen and summarized in a table that included the dimensions of service quality utilized in each model.
  • 260 settlement staff members from 112 organizations in 12 provinces and territories were surveyed.
  • 15 focus groups with 60 SPO staff
  • 240 Newcomer clients (120 remote/hybrid service users, 210 in-person service users), proportional to each province’s newcomer population, were surveyed.
  • 2 focus groups with Newcomers

The survey focused on 5 dimensions that affect hybrid service quality and drive clients’ intention to use hybrid services in the future:

  1. Acceptance of online services
  2. Staff performance
  3. Website content and Accessibility
  4. Web security
  5. Client barriers to accessing online services

A client experiment was created and implemented focused on Newcomers who had not or had not wanted to access online services. The goal was to find out if a small training intervention might help Newcomers. 40 Newcomers received digital literacy training and the WESO team evaluated their experience and feedback. 

Training covered areas related to everyday practical computer skills including:

  • web search
  • sending emails with attachments
  • joining Zoom meetings
  • filling and signing PDFs

It turns out even a small intervention makes a big difference.

Two-thirds of participants overall agreed that their knowledge and skills improved by taking our training. 85% of participants have had a positive experience accessing online Settlement services after the training.

A vast majority of Newcomers WESO surveyed and trained think hybrid services should continue (85.8%) and 86.3% of Newcomers surveyed appreciate a multiple service delivery mode.

We know that there is no one-size-fits-all-Newcomers approach to hybrid service delivery. We all know that our Newcomer clients are diverse. It isn’t surprising that this diversity should extend to how Newcomers want to access services.

We need to get the hybrid service mix right.

What you can do right now

Getting it right means knowing you got it right, where you didn’t, and how to learn from what you know and incorporate it into your service planning.

The WESO team has created not only the Toolkit, but training and tools to help you implement it in your work. 

The toolkit and training includes:

  • self-assessment tools for organizational readiness and hybrid service quality
  • a data collection tool, in the form of a client survey that SPOs can use to collect client feedback.
  • a ready-to-use Excel template that SPOs can use to generate a report from your collected client surveys. An automated report directs the attention of the SPO to areas that need improvements, areas that have waste, and areas that they are doing right. 
  • an action plan to implement the toolkit for SPOs
  • a number of promising practices that support high- quality service delivery. You and your team and organization can use these to continue exploring and expanding your hybrid service delivery practice:
  • Tips to Remove Barriers for Newcomers to Access Online Services
  • Tips to Boost Settlement Sector Performance When Providing Online Services
  • Tips to Improve Website Content and Accessibility

In addition to the existing toolkit and training, the WESO team has developed three self-paced learning modules to enhance your ability to evaluate hybrid services. These modules focus on key areas of data collection, research methodology, and statistical analysis skills. By completing these modules, you will gain the necessary knowledge to assess your services effectively and make informed decisions for service planning.

The WESO team is running training sessions now. You can attend one of their upcoming sessions. If you're on SettleNet.org, the sector’s Community of Practice, join WESO’s SettleNet.org group for info and updates. You can also access their toolkit materials and more on the Y website.

Summary

The YMCA of the National Capital Region’s project, Wired: Evaluating Settlement Online (WESO), is a research-based program that addresses the challenges/barriers that service providers and newcomer clients face in delivering/using remote and hybrid services. The project is uncovering evidence-based practices that Settlement Service Provider Organizations (SPOs) find promising in delivering remote/hybrid services and engaging clients, creating a toolkit/framework and resources for SPOs to evaluate the effectiveness of your service delivery and adapt your models accordingly to improve services, and creating a network of knowledge and promising practice sharing.
arrow-circle-upmagnifier