It's true, some time ago I did a small deep dive into the power of feedback. It was to focus on a really interesting SSIR series on the power of feedback, including a specific story I thought was amazing. I've seen some interesting things come through my feeds recently about feedback and how it can help make change, so it's time to dive in a bit more deeply.
In this story, according to the U.S.-based Nurse-Family Partnership, getting things right means asking the right questions. The simple act of gathering and acting on end-user and partner feedback resulted in changes to the organization’s systems, services, and even its business model.
They received a small grant that provides recipients with a framework to be more responsive to the input and feedback from the people and communities they serve. Using a simple five-question survey that provided a “Net Promoter Score,” they were able to streamline their client feedback collection. Importantly, the simple act of listening has had a transformative impact on the organization and has meant the creation of new data and client outreach roles and capacities.
(A Net Promoter Score (NPS) is used to measure the loyalty of an organization’s customer relationships. It is calculated based on responses to a single question: How likely is it that you would recommend our company/product/service to a friend or colleague?)"a five-step process from an initiative of Fund for Shared Insight called Listen for Good (L4G)that involves:
Watch this great presentation providing an overview of the "listening" model they used for more insight and information.
What does this have to do with you, you ask?
In my experience and some research, the immigrant and refugee-serving sector continues to face many challenges to technology implementation, particularly around service delivery and data, information, and feedback management. In order to achieve innovation and technology outcomes that will propel the sector forward, investments are needed to build the capacity of sector agencies. Feedback to IRCC from the sector about technology capacity has been consistent: the sector needs adequate resourcing, professional development, time, collaboration with experts, and flexibility to iterate, “fail,” learn, and innovate.
Service-providing agencies collect a lot of data and information about clients, primarily for the purposes of output reporting. There is an opportunity to utilize the wealth of existing and archived data for more strategic practices moving forward. Even though agencies consistently express that they want to do more, they also state that they do not have the expertise or the capacity within their agencies. When they say they do have it, in general it is described as a staff person doing it “on the side of their desk.” This is a common theme in the sector, one which is also recognized by the funder.
But, not in terms of its funding.
Measuring service impact and client outcomes comes up in interviews and the literature as both important, and a challenge for agencies. Many interviewees are unsure of what it is they should be measuring. Measuring outcomes versus outputs is part of the discourse. In a panel discussion, former IRCC Director General Michelle Lattimore described the importance of measuring customer experience in service delivery (jump ahead in the video to 1:02:51 or so to see the really interesting panel she's part of, and for her insights). She suggests that valuing client experiences that are not simply efficient and effective but also build trust with clients is particularly important.
That's right, and sounds great, but...
Mainstream sector actors that have been funded for years indicate that they have patchworks of systems and few in-house capabilities and resources to effectively manage client data feedback looks and information management systems. In many cases they’re using software housed on computers located in their agencies, which are not interoperable with other systems they are or might use in the future. Most settlement agencies indicate that they would need an increase in resources, both technical (technology, design and process) as well as human (new roles, or re-/up-skilling of existing staff, along with time allocated for these tasks).
These sector actors collect a great deal of data about their clients but admit they don’t know how to use it. They’re conditioned to collect and report data to their funder, but it has not been harnessed to inform their service delivery process. A variety of mechanisms and information management systems are used in different agencies. Many are output focused; data collected to provide to IRCC in iCARES for the purposes of statistical reporting. Issues that come up that agencies are not effectively able to deal with include:
Should agencies build internal expertise or make use of centralized resources that can analyze and provide data back to them? Sector folks suggest that it is both. Agencies need to be able to work with their own data, but also with other external data sources and collaborations (such as government data).
But, data/information management capacity & harnessing client feedback require additional resources and external expertise to build sector capacity. Building agency capacity around data, information management to better utilize client feedback requires new skill sets, or roles. Building this capacity will be essential for the future, but service providers do not currently collect outcome data (OK, some of you do, but it's not widespread enough).
Having some resources at an umbrella group or LIP or other centralized neutral sources for aggregate data analysis has also come up as something that would be useful. Another idea would be having a shared researcher who does regular statistical and analyses, including responding to data requests, for local agencies in a LIP catchment area. Ethnographic research is also being looked at as an innovative way to ensure that client’s lived context is being incorporated into program design and decision-making.
Creating new data collection and analysis approaches would require a great deal of effort. Is it IRCC’s role to fund or build this capacity among service providers? I think so. Maybe there's a reason to hope in the recent call for proposals? I suppose we'll know in 2020...
Agencies recognize that they need to shift to user-centred program design sooner rather than later, in order to ensure they are meeting client needs and measuring outcomes. Service evaluation that is done is not tied to the concrete needs of clients, according to some. While clients are surveyed about the services they receive, the feedback is not incorporated in program development.
Discussion about client tracking systems and how effective they are at providing agencies with the information they need to make educated decisions around investments and service gaps are not new. Sector folks indicate that they still struggle with internal data silos, where client data is collected and entered more than once because of different programs or funder requirements in their agencies.
There is also research that outlines the capacity challenges agencies face moving from outputs to an outcomes-focused approach. In 2010, researcher Meyer Burstein outlined a service provider innovations strategy that builds on the unique sector strategic capacities to jumpstart innovation and disseminate best practice information.
A 2012 report evaluated the innovation capacity and practices of immigrant and refugee-serving sector agencies identifying significant barriers to sector innovation (PDF). The report recommended an annual innovation cycle to be implemented by the funder that would also increase knowledge sharing and transfer. Priorities could focus on target groups and services, geographic location, scale, and local infrastructure, in collaboration with IRCC. Both this study and Burstein's earlier report built on previous observations that action be taken based on a shared interest by governments and the sector to build agency capacity to better analyze client outcomes and innovate.
Researchers recommend the sector act collectively, to share resources, protocols, promising practices and investments while funders provide the support necessary for such capacity building, including “research, knowledge dissemination, and experimentation with new technologies and new organizational arrangements” to promote consistent technology development, capacity building and rollout in services to clients. Collaboration between the sector and its core funder, IRCC, is recommended in a variety of research reports, spanning almost 20 years of work. This isn't new to the sector or its key funder.
IRCC doesn't seem to have been listening to the research they've funded or is unwilling to resource sector-wide development in this area. So, it's up to you (and when you learn, you should share with your colleagues!). Finding useful capacity building and training on data collection and analysis is an ongoing challenge for many agencies. Here are some suggestions.
Specific tools like IFRC’s Data Playbook provide hands-on tools to enable organizations to start conversations about data collection and analysis. Open North recently suggested that external resources and experts should be brought in to create data management expertise and a shared data culture within the sector. These approaches and recommendations should be investigated further.
In one promising practice, the Toronto East Quadrant LIP is working with local settlement agencies to aggregate client data they collect to be analyzed by a neutral third party system and shared back to the agencies and community. There is a lot of interesting potential here. Working with the agencies, the LIP identified a gap in capacity, resources, and expertise to analyze client data effectively within their own organizations. It also identified a need to create a safe and neutral space where agencies can focus on collaboration instead of competition, to ensure that agencies are encouraged to share information, particularly service delivery information, for mutual benefit. Keep an eye on the project to see what you might learn.
Dive deeply into that SSIR series on the power of feedback. You will find a lot of practical information and resources. Participate in an upcoming live online discussion on September 17th, Managing User Feedback, Our Next Online Discussion: "User feedback is a critical element of any network-centric resource. Soliciting and incorporating user feedback shares ownership and demonstrates the contribution of your network. In this online discussion, we will explore ways to manage user feedback, track how feedback is incorporated into resources and then accurately report back to contributors."
Read this recent newsletter from Feedback Labs about proof that feedback works. Yes, a place called Feedback Labs exists, and "in order to make feedback commonplace in the social sector, [they have created] feedback tools that are easy to use, and widely accessible feedback training." Check out their free toolkits and crash courses in feedback and building a feedback practice at your org, so good.
And, in case you're thinking, hey, to get good feedback and use it well, I probably need to learn more about data, collecting and analyzing data, etc. (gentle now, it's a deep, but rewarding path), so check out free courses at the School of Data.