Social and Ethical Risks Posed by General-Purpose LLMs for Settling Newcomers in Canada (2024)

What is this research about?

This research explores the potential risks and ethical concerns associated with the use of general-purpose large language models (LLMs) like ChatGPT in the Newcomer-serving sector. It examines how these AI tools could impact immigrants and refugees if used without proper customization and safeguards.

Report abstract:

"The non-profit settlement sector in Canada supports newcomers in achieving successful integration. This sector faces increasing operational pressures amidst rising immigration targets, which highlights a need for enhanced efficiency and innovation, potentially through reliable AI solutions. The ad-hoc use of general-purpose generative AI, such as ChatGPT, might become a common practice among newcomers and service providers to address this need. However, these tools are not tailored for the settlement domain and can have detrimental implications for immigrants and refugees. We explore the risks that these tools might pose on newcomers to first, warn against the unguarded use of generative AI, and second, to incentivize further research and development in creating AI literacy programs as well as customized LLMs that are aligned with the preferences of the impacted communities. Crucially, such technologies should be designed to integrate seamlessly into the existing workflow of the settlement sector, ensuring human oversight, trustworthiness, and accountability."

What do you need to know?

The Canadian settlement sector, which supports newcomers in integrating into society, faces increasing demands due to rising immigration targets. While AI tools could potentially improve efficiency, using general-purpose LLMs without customization poses risks such as biases, misinformation, and vulnerability to malicious use.

What did the researchers do?

The researchers conducted experiments using ChatGPT to simulate scenarios relevant to newcomers, such as seeking employment information, accessing health data, and gathering details about banking services. They analyzed the outputs for biases, inaccuracies, and potential harms across different languages and user backgrounds.

What did the researchers find?

The study revealed several concerning issues:

1. Bias in employment suggestions based on national backgrounds
2. Performance disparities across languages, particularly in health information access
3. Stereotypical representations of immigrants and refugees
4. Instances of hallucinations and misinformation
5. Potential for malicious use through scams targeting newcomers

How can you use this research?

If you work in the settlement sector, this research highlights the need to:

1. Develop customized AI tools tailored specifically for newcomer needs, rather than relying on general-purpose LLMs
2. Implement safeguards and bias mitigation strategies in AI systems used for settlement services
3. Promote AI literacy among newcomers and service providers
4. Ensure human oversight and accountability when integrating AI into existing service structures
5. Invest in technologies to detect and combat AI-facilitated fraudulent activities targeting newcomers
6. Collaborate with policymakers, technologists, and newcomer communities to design fair and transparent AI solutions
7. Focus on enhancing areas where AI can be beneficial, such as language training, information dissemination, and employment support, while maintaining human-centered approaches for sensitive services like resettlement and community connections