This presentation was part of a pre-conference at the June 2019 International Metropolis Conference held in Ottawa.The topic of the day was: How Do We Know What’s Working? Measuring Settlement Outcomes for Individuals and Communities
Through presentations and table discussions, this hands-on, full-day event explored new strategies for measuring immigrants’ economic, social and civic-cultural outcomes at both the service delivery and community levels. The focus was on strategies for determining what works and what doesn’t, as well as new ways of measuring processes and change, attributing outcomes, and assessing community impact. Outcome measurement is not only a tool for ensuring accountability. It provided a basis for identifying promising practices that can be further developed and shared; allowed practitioners to identify practices that need improvement and suggested how to do so; and, at the community level, points to areas in need of particular attention.
In this presentation, ISSofBC's Kathy Sherrel outlined the journey her 350 person, 17 location, and 60 active program organization has been on that started in their settlement department around this concept of
outcomes measurement about two years ago.
Download her presentation slides: Kathy-Sherrell- Journey to Outcomes Measurement - P2P pre conference - intl metropolis june 2019 (PDF)
Transcript (auto-generated by YouTube):
okay so I'm gonna bring the scale way
down or way sideways we've heard a lot
this morning about outcomes measurement
and lots of large scale data sets which
on the one hand can be very exciting but
when we also hear about these this
outcomes measurement being done in some
ways on our behalf as sector
representatives it can also be a bit
unnerving when you hear the term
outcomes measurement I think there's
kind of two ends of a spectrum that we
all fall in there's some of us whose
inner nerds squeal with gee joy and then
there's the people who don't may not
want to admit it but are terrified
because they're not quite sure what this
is and they're not quite sure how to do
it as the government increases the
emphasis on on outcomes measurement it
can be really scary to admit you don't
have it all figured out but you know
what I want to start this conversation I
work for a large scale organization in
Vancouver I think the work we do is
great we have you want data we've got
data we've got the organizational data
we capture our outputs our program
evaluations at the program level our
evaluation forms after services but if
you ask me do we know what the overall
impact of our organization which has 350
people 17 locations and about 60 active
programs I will tell you no and I don't
think we're alone so I want to talk a
bit about the journey that we've been on
at ISS that started in our settlement
department around this concept of
outcomes measurement about two years ago
as I said we recognize we collect a lot
of information than I care is a
fantastic amount of information but
unless you have a organizational
database where you can access it and
analyze it it doesn't do you a lot of
good and I'm heartened to hear and I've
been hearing for a little while that the
looking how to get those data cubes back
to us so that we can use that
information and it doesn't it evens the
playing field with those who maybe in
smaller organizations who don't have the
capacity or that organizational database
when we look at in ISS we started in our
settlement department with really trying
to understand what is the impact we make
we plant them up with a three year plan
focused on settlement clients which
included the implementation of an annual
multilingual survey outcomes settlement
survey refinement and updating of
program evaluation for group services an
ongoing analysis and correction to
ensure that our responses were
reflective of the clients we wanted to
be able to tell that bigger story and to
begin to be able to tell that bigger
story on our behalf and on our clients
behalf before we launched we addressed a
lot of the grappled with a lot of the
ethical and practical considerations of
doing any sort of survey with with
clients including language it's not just
about what languages you translate the
materials into it's about how you
translate words a lot of the outcomes
may seem single simple if you are in
English as a first language but if
you're not they contain high level words
and complex concepts that you need to be
able to make accessible to to all
newcomers both those who come with
English and those who come with no no
literacy skills we wanted to make sure
that our our evaluation whether we were
looking at workshop evaluations or
surveys were accessible so we looked at
things like Likert scales that use
emojis rather than words because the
concept of very much compared to a
little bit compared to neutral was a lot
then a very big happy face - a very big
unhappy face we thought about how we
were gonna do this our organization
serves about 20,000 clients a year so
even if we were looking at settlement we
were looking at large-scale numbers and
ideally you want to be able to use focus
groups or individual interviews but
that's not always possible so we thought
about doing a multi method multilingual
survey we looked at an online surveys
are a quick and easy way to collect
information but as we've heard they also
prioritize those who choose to respond
and who have the digital literacy so we
try to make sure when we do the surveys
that we also have copies in our offices
so that those who come in can answer it
in first language the question of who
implements the survey organizations
don't always have the resources to bring
in outside evaluators and to be able to
pay people to do who are not involved in
service provision to do those surveys
and as you're asking staff to do the
surveys to implement the service you've
got to think about the question around
ethics and power if I'm the one
providing you services what is the
likelihood that you are going to say
that my services were not effective for
me a few years ago when I was in China
really brought it home as I was handing
my visa over at my passport in my visa
over in front of me was a little thing
saying how was your service today given
that they were about to let me in the
country or not it was excellent thank
and we need to think about that we need
to make sure that the answers we're
getting are indicative of what our
clients are feeling so we did planning
we loved planning we thought about the
issues we looked at our outcomes that we
had committed to particularly in IRC see
we consulted with managers around how do
we make the language accessible then we
took what we thought were the more
accessible and we talked to different
staff around did it make sense asking
them to think about from their cultural
background from their language did it
make sense we tested the survey with a
hundred english-speaking clients not the
best to test with admittedly but we
wanted to understand from those who did
speak English and who had access
whether the concepts were accessible we
finalized it we tried to avoid
duplication and we thought about the
issue if you need to understand who it
is that you're serving so we asked
demographic questions we're still
internally grappling with that whole
question of identification and following
people we wanted it to be anonymous
because we wanted people to be able to
respond fully we ended up with 41
questions which looking back way too
we gave staff we talked to staff about
it we had talked about why we were doing
it or at least we thought we did and we
we moved forward we gave them online
links we sent it out to clients and we
figured we were now well on our way you
know we had planned we have tried to
think of everything we knew that there
would be curbs we knew we were gonna
have to adjust but really we thought
we're on our way and that's where our
imagination that reality do you remember
the boardgame snakes and ladders how it
felt when you get to pretty much the end
and you hit that spot in your way back
well that's what outcomes measurement is
like when you're trying to implement it
at an organizational level it is not one
way you can make all of the progress and
sometimes you have these leaps these
moments where everything comes together
and then you hit the square that you're
back at the beginning and there's things
that you should have thought about that
you didn't when we started we wanted to
understand settlement and what our first
survey taught us is we think about our
divisions in our organization our
zatia so we learned that you know what
we can't do if we want to do a survey at
an organizational level we actually had
to think at an organizational level
so our second survey this year was at an
organizational level when we look at
adjusting our course our year to survey
the first thing that we learned we had
to address is outcomes measurement takes
capacity and resources and I really
strongly want to stress that as funders
when we're talking about outcomes
measurement it is not enough to say this
is what we have to do there has to be
capacity building and there has to be
resources and that includes time and
money to be able to effectively measure
outcomes and to do the justice that it
staff buy-in is critical as a researcher
I have long been frustrated with what I
call the take out version of research
you know those phone calls you get that
say I need 10 immigrants to interview by
Friday and if you can have you know for
women for men to non-binary preferably 2
in a wheelchair and I need them all to
speak 10 different languages here's what
I realized I demand as as somebody with
an academic background that when
researchers come to us that they explain
the process that they involve people
from the beginning and I failed I think
we failed but all on part of this we
explained what we were doing but we
didn't actually explain why we were
doing it we didn't explain how this
impacted them that it wasn't about you
know were they doing their job because
that's something we had to address is
for some people when we said performance
outcomes measurement it was a question
of am i doing my job and really making
sure people understood the difference
between outputs and outcomes we began
this process by integrating into all of
conversations around outputs we have a
couple of key slides that will come up
much too I think people's annoyance
pretty much every time we talk about it
because we really want people to start
from the foundation of understanding
inputs outputs and outcomes and
understanding their role in creating
these outcomes and they need to be able
to see themselves and how their work
influences it we also need to adopt a
holistic organization-wide
approach and not only did we do this at
the oh hey sorry not only did we do this
at the survey level but as I'll show on
the next slide we've started to do this
even in our proposals to think beyond
what are the outcomes of this program to
what are the outcomes we're trying to
achieve as an organization client
engagement is necessary and probably the
hardest part about doing this we
recognize when we did it at an
organizational level that link classes
are the ideal because teachers could get
people started and then leave the room
or have somebody in there to answer
questions settlement at the far end of
the spectrum is often chaos driven
people don't necessarily come all the
time they tend to come when there's a
problem and they want to deal with it
so how do you reach those clients how do
you have them respond and even again
going back to staff I and when you're
looking at when clients are coming in
asking people to do more to collect more
information to say to clients you know
what can you just stay and answer this
is really challenging in our
organization settlement has the highest
client numbers by far and yet we were
third in terms of respondents who
indicated they had received settlement
services so we're trying to look right
now and having discussions internally
about how do we overcome that is it
doing field or focus groups
is it having volunteers or paid non
regular staff individuals in our offices
doing the survey but unless we figure it
out we will not succeed and we're still
looking at how do we bring all those
other sources we've heard it today we
have all these sources of information
from I care to evaluations at the end of
workshops to outside evaluations to
focus groups to success stories how do
we actually bring those all together and
talk about outcomes holistically across
different types of information so inside
our organization it's changing the way
we do outcomes measurement this is a
simple document but it was included in
our proposals and it represents in some
ways our first step at really trying to
overcome the internal silos that we have
in doing the client pathways we wanted
to show right from wrap through
settlement employment language that it
didn't matter what the individual key
indicators that you are looking at in
your program we are all reap trying to
reach the same goal and all of us
regardless of funder contribute to those
outcomes so for this document while it's
IRC C language what we also realize is
if we step back we could look at our
provincial outcomes we could look at
different funder outcomes they all fit
within this so we're looking at adapting
this slightly and then actually posting
it in every office that we have so that
people can see they're part of that
bigger process and understand where the
goal line is I think as we talk about
outcomes measurement there's some larger
scale considerations we really have to
be able to grapple with and to have
these conversations at multiple levels
the agency capacity to collect and
analyze data very significantly across
and I had a buyer and appreciate some of
the work that's being done by the
settlement outcomes group including
bringing together the settlement
outcomes working group of which Bubba
and I both participate as well as the
project out of University of Alberta on
increasing capacity to measure refugee
outcomes I think these are great but we
need to start having these conversations
and having these trainings about
outcomes measurement not just at the
senior manager and above level but every
person in every organization needs to
understand why we're doing this and to
buy into it as I said it can't be said
just a reminder it takes time and
resources not just to implement it but
to develop it and to really think
through it's kind of like the ongoing
Deus Ex it's like the ongoing rabbit
hole every time you think you've got it
you come down another rabbit hole
attribution is always a challenge and
especially over time positive outcomes
are easy to tell the story but what two
negative outcomes tell us is it about
failure or is it about we ask the wrong
question and in the absence of
consistent indicators how can outcomes
be measured at the national level I
think the biggest message that I have is
to me we're all in this together we all
need to be able to talk openly about
what we are doing what we don't know
what questions we have rather than being
in our own silos and developing things
in in isolation if we work together to
build the sector ride capacity share
best practices and explore ways to
measure outcomes everybody gains
