OK. So, I'm not an academic, but consider myself a decent researcher. During some work a few years ago on the digital messaging practices of immigrant and refugee-serving organizations and newcomers, I came across some really interesting research that just struck me as a great model. It was in the form of a Master's degree (which makes me want to write a post about the importance of community, student, grey area, and other non-academic (read: PhD's) research, but that's for another time): ‘As Important to Me as Water’: How Refugees in Rome Use Smartphones to Improve Their Well-being (2017).
St George's research introduced me to the Capability Approach, really interesting work on technology use among refugees, some of their models/frameworks, and her own version of this, the Smartphone Evaluation Framework.
Building from this Smartphone Evaluation Framework theory, I proposed a similar Digital Messaging Evaluation Framework. Each section is outlined and explained below, incorporating useful components of other researchers' ideas and approaches.
In this post, I'm sharing my half-baked idea, which I honestly wish I had shared pre-COVID. It could have generated some interesting conversations. Any way, here you go. You can also access this post as a PDF.
Introduction
"The paradox is this: ICTs and particularly the internet are widely regarded as groundbreaking inventions that have changed the way millions of people live their lives, and yet researchers and practitioners in the field of ICT and development often struggle to prove specific impacts of the technology to funders." Dorothea Kleine, ICT4What ?— Using the Choice Framework To Operationalise the Capability Approach To Development
Digital messaging is simple, on the surface, but is just the tip of the iceberg when it comes to digital capacity:
The following framework comes out of this research (which I'll add to this site soon, when I have a bit more time):
Gigler, B.S. (2011). Informational capabilities - the missing link for the impact of ICT on development. E-transform knowledge platform working paper series(1).The World Bank.
Gigler, B.S. (2015). Development as freedom in a digital age: experiences from the rural poor in Bolivia. Washington DC: World Bank.
Kleine, D. (2010). ICT4WHAT? Using the choice framework to operationalise the capability approach to development. Journal of International Development 22(5), 674-692.
Diaz Andrade, A & Doolin, B. (2016). Information and Communication Technology and the Social Inclusion of Refugees. MIS Quarterly. 40. 405-416.
Abujarour, S & Krasnova, H & Wenninger, H & Fedorowicz, J & Olbrich, S & Tan, C & Venkatesh, V & Urquhart, C. (2016). Leveraging Technology for Refugee Integration: How Can We Help?
Abujarour, S & Krasnova, H. (2017a). Understanding the Role of ICTs in Promoting Social Inclusion: The Case of Syrian Refugees in Germany.
Abujarour, S & Krasnova, H & Díaz Andrade, A & Olbrich, S & Tan, C & Urquhart, C & Wiesche, M. (2017b). Empowering Refugees with Technology: Best Practices and Research Agenda.
Abujarour, S & Krasnova, H. (2018). E-Learning as a Means of Social Inclusion E-Learning as a Means of Social Inclusion: The Case of Syrian Refugees in Germany.
Vivienne St George, T. (2017). ‘As Important to Me as Water ’ : How Refugees in Rome Use Smartphones to Improve Their Well-being.
Mergel, I., & Bretschneider, S. I. (2013). A three-stage adoption process for social media use in government. Public Administration Review, 73(3), 390–400.
These researchers started with economist Amartya Sen’s Capability Approach and combined it with McClure's notion of informational capabilities to create a useful framework for the immigrant and refugee-serving sector to build and evaluate settlement agencies' capacity to use technology to serve clients.
McClure's informational capabilities concept refers to:
ICT capability – ability to use ICTs in an effective manner
Information literacy – ability to find, process, evaluate, and use information
Communication Capability – ability to effectively communicate with clients, colleagues and others in their professional network
Content Capability – ability to produce and share content with clients and others through the network.
These capabilities include the skills, knowledge and attitudes of the Settlement worker. All of these capabilities exist within an Information and Communications Ecology or environment, much of which is outside the control of the individual. This includes agency structure, policies, and even further external forces, such as funding, government policies, and includes the informational capabilities of these actors, as well as those of clients.
Informational capabilities are a person‘s capability to transform their existing informational capital, such as level of access to ICTs (their opportunity structure) into real opportunities to achieve the things they value or need to to. In our case, to serve clients effectively. In our framework, informational capabilities refer to front-line worker's freedom and ability to use ICTs within the institutional and socioeconomic setup of their agency. It is important to emphasize the significant differences between informational capital, ICT capabilities and informational capabilities.
All of this occurs within stages of adoption. It is important to acknowledge that different agencies are at different stages of capability and adoption. Usefully, a couple of researchers have created a model of three phases of adoption of technology (in their case social media) in government. The three stages, are “experimentation, constructive chaos, and institutionalization” of new technologies.
I believe this model, and how it can flow from one stage to the other, is an accurate portrayal of how digital messaging technologies are currently being adopted within Settlement agencies, and can be useful for our analysis.
“In the experimentation phase, individual innovators who have some experience with technology from other work or non-work settings begin using it in the workplace, and the use spreads from worker to worker and thus may be used in a variety of informal ways. In the constructive chaos stage, workers begin to recognize both benefits and risks of the technology use, which may create tensions within the organization, and organizations respond with attempts to standardize the technology use through practice or policy standards. Often in this stage organizations seek answers that other similar organizations use, or draw upon their past policies related to ICT use to drive current standards. Sometimes reactionary standards are developed in response to misuse, or concerning use, of technology.
Finally, in the institutionalization stage, the agency has developed a set of standards, processes, and enforcement measures to control the use of the technology. Agencies that know about their workers' use of social media are grappling with how to move from the first two categories to the later given the perceived benefits of social media use, while also grappling with the risks and challenges related to social media use.” Mergel, I., & Bretschneider, S. I. A three-stage adoption process for social media use in government.
Additional researchers have taken these approaches and focused on the importance of acquiring informational capabilities — the ability to transform access to ICTs into real opportunities to achieve the things one values. In our case, to serve clients using technologies that they are using or have indicated a preference to use to communicate with service providers. This approach was recently applied to the use of ICT and, more specifically, smartphones among refugees. Vivienne St George, T. (2017). ‘As Important to Me as Water’: How Refugees in Rome Use Smartphones to Improve Their Well-being (2017).
The Digital Messaging Evaluation Framework
Opportunity Structure (external to people's capabilities – the information and communications ecology)
This is the environment within which front-line workers and managers within are attempting to use digital messaging to serve clients. This structure addresses contextual factors influencing settlement agencies' smartphone use to serve clients. These are conceived of as structural influences which impact on people’s opportunity to use ICTs to expand their capabilities to serve clients.
“Technology is embedded in the wider set-up of institutions, policies, programmes, norms and discourses. As such they need to be analysed as firmly and historically engrained.” The sense of choice vs the use and achievement of choice is heavily influenced by agency capability to see potential of tech and to integrate tech within existing policy, practice, management and human resource frameworks. That in turn is heavily influenced by funder ability to recognize this potential. These are the work, agency and funder information, communications, and funding environments that front-line workers exist within.
Agencies:
funding
recognition of technology as (not) legitimate service tool/channel
leadership digital literacy
changing and diverse use of ICT by clients/newcomers
data literacy
security/encryption literacy
including technology in assessment - not asking clients for preferred method of communication at intake
changing staff numbers and complement (based on immigration patterns/trends) – how to know the right way to invest in new technology/phones?
Lack of funder support/literacy - funder digital literacy and ability to evaluate the request for support of technology in budget, or entire technology-mediated projects
Workers
overwhelmed already with day-to-day workload
Concerned about more work and boundaries
digital, data, security literacy
people, not tech focused
responsive to client tech use, not proactive – while different clients use different technologies, difficulty prioritizing, and this changes over time
don't have tools (including hardware), training and evaluative support to implement tech effectively
perception as “para professional” - no regulatory or standards frameworks in place
find out about client communications preferences anecdotally
management/funding supports vary inconsistently
access to training, professional development, orientation and support
Funders:
lack of digital literacy in the context of human services
pressure to audit rather than serve
not clear on what the sector wants in terms of use, support and funding for technology to serve clients
internal disconnect about utility of technology in service delivery
institutional pressures
lack of clarity about how technology is part of and can mediate service delivery rather than being an indirect service tool (eg. Computer on desk as admin tool for funded staff)
Personal assets/resources
In the second box, personal assets and resources are made up of the informational capabilities:
ICT capability – ability to use ICTs in an effective manner
Information literacy – ability to find, process, evaluate, and use information
Communication Capability – ability to effectively communicate with clients, colleagues and others in their professional network
Content Capability – ability to produce and share content with clients and others through the network.
In the case of settlement agencies, this is mainly in the form of the existence or lack of personal and agency assets and resources, or that may be hidden, informal and ad hoc. There are many assets that agencies and workers have, but many more challenges and barriers when it comes to using technology to serve clients.
Agencies and Funders:
various levels of tech literacy and seeing technology as a legitimate service tool
project-based technology integration – siloed, not all staff, not connected to mission/vision/strategic plans
few policies, protocols, guidelines, onboarding standards, expectations, management/supervisory clarity, etc. - managers and leaders do not feel competent or capable managing staff using technology to serve clients
lack hardware, IT, ICT infrastructure
a history of innovative service delivery with scarce resources
limited knowledge sharing within agencies and across the sector
limited resources, unlimite challenges
limited knowledge/understanding and models of how to manage technology for service delivery
limited sector/collective advocacy opportunities to funders or policy-makers about ICT use to serve clients
Workers:
client centric
commitment to clients and their settlement/integration
regardless of barriers, they serve clients using technology anyway, sometimes using their own personal devices and accounts
varying levels of tech literacy, capacity, interest
strong community connections but lack UX/user design knowledge for technology integration – want to be more flexible in using the tech that their clients use
limited influence on service delivery methods and innovation in spite of experience
limited ways to “pitch” ideas within agencies and to funders
some have access to the technology they need/want, some don't – inconsistent implementation across the organization
limited exposure to methods/tools/practices of how to use technology to serve clients
Levers for change (institutional processes, policies and programs)
The third box, in the centre of the diagram, refers to the kinds of interventions or support that may be required from government to support agency technology integration, management and use, and agencies to support worker technology use.
Within the levers for change there should be an analysis and recognition of the different stages of technology adoption and where a particular agency is. This will guide what they need to do and supports required to move ultimately to institutionalization of of digital messaging (and other technologies) in service delivery.
Agencies, funders and workers:
skills development
leadership support
space and resources to experiment
becoming knowledgeable and competent in user-centred design
policy development
continuous learning/standardized professional development
sharing of “good ideas”/Community of Practice
acknowledgement of digital services as legitimate service, not tech, not indirect
protocols, guidelines and standards
privacy, security, confidentiality standards
digital service standards – incorporating best of from other sectors (but also from what is emerging within the sector at the agency and funder levels)
hardware access and upgrading (Total Cost of Ownership beyond computers)
seeing technology/digital messaging as another service channel that requires the competencies and resources to master it
iteration and innovation – the sector can and should have both – both require funder/program innovation
Managing in a Computerized Environment 3 (MICE3) – building on what the sector has already told the funder it needs in order to effectively use technology to serve clients (i.e. Their informational capabilities)
collaboration and partnerships with the private sector, including civic tech
evaluation frameworks – client outcome oriented, not just technology focused – how technology has contributed to outcomes, access and efficiencies
simplified processes to pitch new technology use in service delivery
agencies move “ask” to funder, funder open to “ask” - better dialogue between sector and funders
revisioning the role of the settlement worker – a blended digital/in-person role
Informational/Service capabilities (Affordances) enabled by digital messaging (and technology more broadly)
In the fourth box, we focus on the informational capabilities most appropriate for the settlement sector.
Affordances are neatly organized by the informational capabilities model:
ICT capability – ability to use ICTs in an effective manner
Information literacy – ability to find, process, evaluate, and use information
Communication Capability – ability to effectively communicate with clients, colleagues and others in their professional network
Content Capability – ability to produce and share content with clients and others through the network.
(Technology affordance analysis is about unpacking the different attributes of a technology so you can determine its suitability for a particular goal, such as serving newcomers.)
Kleine's “degrees of empowerment” idea is also useful here. Kleine writes: "Individuals were aware of some possibilities the new technology offered them, like email and online chat, but not of others, like Voice over IP. This was precisely because their educational resources (including computer skills) and the dominant discourse… stressed some usages over others. For any piece of research focused on a technology which is new to the respondents, the dimension of 'sense of choice' will play a significant role. The 'use of choice' dimension refers to whether or not an individual actually makes the choice and the 'achievement of choice' refers to whether the outcome matches the choice expressed." For workers and agencies, capabilities are impacted by the actual existence of choice, a worker's or agency's sense of choice, use of choice and achievement of choice. This is an important dimension to explore within settlement agency context.
The sense of choice vs the use and achievement of choice is heavily influenced by agency capability to see potential of tech and to integrate tech within existing policy, practice, management and human resource frameworks:
Access to possibilities, or the sense of choice, is essential.
In the sector:
Combine messaging with other technology, such as Google Translate, mapping, etc. (polymedia)
increased technology and information literacy – expansion of sense of what is possible with technology – starting to move from iteration to innovation
worker use with client satisfaction provides use case for leadership
potential for content sharing, replication and re-use within and across agency
more direct learning with clients – example of Chechens showing use of Google Translate as a useful tool to COSTI CSS workers, then adopted by workers within their service framework
increased awareness of potential by also limitations of technology to serve clients
increased appreciation and capacity for security and encryption
beginning to more actively map client journey and where technology intersects and has a role
recognizing that front-line workers have the information, knowledge and experience to push upwards to create change and systems movement
possibilities for increases in:
ICT capability among all settlement actors
information and service and referral capability
communications capabilities
accessibility for clients
appreciation for technology possibilities in service delivery (thin wedge)
recognition that technology doesn't mean a reduction in human resources
recognition of a need to create space for pitches, innovation, policies and guidelines
Outcomes
Overall, we're measuring increases in:
informational/ICT capabilities
service capabilities
integration capabilities with clients
over periods of time and where the agency is at in their digital journey.
The outcomes for workers, agencies, funders are ultimately related to better outcomes for clients. They can be broken into immediate, medium and long-term outcomes, or into the stages of technology adoption: Intrepreneurship and Experimentation, Constructive Chaos, Institutionalization.
Immediate (Intrepreneurship and Experimentation):
increased access for and to clients
less travel
small entry to technology can be leveraged for bigger use later
meet clients in the technology channels and platforms they use
clients communication preferences addressed
increased potential for service scale
formalize informal technology use in agencies, legitimize and recognize as service delivery – funder sees as innovation, really just iteration
increased appreciation for ethics, standards, technology security and need for encryption
Medium (Constructive Chaos):
scale of service – groups, for broadcast or back and forth – is easily created
better appreciation of technology as a direct service tool
increased technology and information literacy among workers and leadership
iteration in service delivery
actual increases in scale of service
polymedia increases
client journey mapping becomes more actively applied
replication of successful models and approaches using technology to serve clients
awareness of potential, but also limitations and downsides, of technology use to serve clients – leading to creation of models, standards, protocols and strategic approaches to integration technology into service delivery
Long-term (Institutionalization):
easily replicable models, standards, protocols and approaches within the sector
innovation in service delivery
enhanced service delivery across channels
consistency in digital capacity and offerings to clients
adoption of models, standards, protocols and strategic approaches – rolled out across the sector with ongoing training, professional development and capacity-building models agencies and workers can access
increase of reaching clients previously unreached or not accessing services
professional development includes technology
knowledge and content sharing, replication and re-use across agencies bringing efficiencies to digital services
Community of Practice develops and is successful – broader than technology, but also encompasses digital capacity
There are likely a number of capacity frameworks out there that can be researched to tease out the specific and actual skill development needed by workers, management and funders. One that has much promise is Nethope's Center for the Digital Nonprofit. Their Digital Nonprofit Skills (DNS)™ framework and Digital Nonprofit Ability (DNA)™ framework expand on the technology adoption for nonprofits, rounding out the specific capabilities agencies and workers need in the digital era.
The the Digital Nonprofit Skills™ (DNS) Assessment which establishes a solid baseline for skills across roles and across organizations. The Digital Nonprofit Ability™ (DNA) Assessment looks at a nonprofit's digital transformation journey to assess their digital readiness.
