This is the fourth session in a 4-part webinar series exploring the transformative potential of AI in the Newcomer-serving sector, with a special focus on how AI can enhance our services and operations. In this session, Fernando Vitorino, Vice President, Information and Communication Technology Solutions & Services, Achēv and Aimee Holmes, Vice President, Data Strategy & Digital Solutions, presented their experiences of integrating AI into program and service delivery.
This webinar, hosted by Peel Newcomer Strategy Group with support of Digital Transformation committee of the Executive Council of Peel-Halton Settlement Partnerships, is designed to share knowledge and discuss the potential impact of Artificial Intelligence.
Speaker: Aimee Holmes, VP Data Strategy and Digital Solutions
Aimee Holmes leads the Online Services team at ACCES Employment, a powerhouse department that integrates an AI-based virtual assistant (chatbot) into service provision, trains staff to use several online tools, develops e-learning modules, and strategically evolves a Salesforce-based CRM to improve how data is collected, shared and analyzed. As the Vice President, Data Strategy & Digital Solutions, she is passionate about harnessing technology to streamline processes, enhance customer service, and save staff time. Aimee has worked in the settlement sector for over 15 years and has an MA in Global Studies.
About Aimee's presentation:
Integrating AI into service delivery requires significant time, effort and resources. As such, it’s critical to define the problem you’d like AI to help solve, engage staff along the way, and select technology wisely. In 2020, ACCES Employment launched VERA (Virtual Employment and Resource Attendant), an AI-based virtual assistant, to improve access to information and services for people seeking employment. In this webinar, you will learn what VERA can and can’t do, usage stats, and the impact it has had on clients, staff and the organization.
Speaker: Fernando Vitorino, Vice President, Information Technology Solutions and Services
Fernando Vitorino is currently leading the Architecture of an AI enabled architecture at Achev. Throughout his career, he held senior level leadership roles in various industries including wireless communications, global logistics services, retail, and information technology consulting. He has also led various business transformation projects to both expand and transform organizations through the implementation of enhanced business intelligence and productivity tools. He holds a Bachelor of Science from Wilfrid Laurier University and a Master of Management Sciences from University of Waterloo as well as ITIL, PSM and PMP certifications.
About Fernando's Presentation:
A brief review of the challenges facing AI governance at an organization, decision making considerations in the selection of AI solutions and finally an overview of AI development at Achēv.
Machine-Generated Transcript
What follows is an AI-generated transcript of Isar's presentation using Otter.ai. It may contain errors and odd sentence breaks and is not a substitute for watching the video.
Jessica Kwik 0:00
So we've This is the fourth and last session of our our of our webinar series on artificial intelligence as it connects to the settlement sector. I'll just introduce myself. I'm Jessica quick. I'm the director at the peel newcomer Strategy Group. We're based at United Way, greater Toronto, and I'm joined by my colleague, Shreya, who's been help so helpful in coordinating this series as well. We're also very grateful to connect what we're doing virtually to the to the land we're on. So I'm just going to start off with a bit of a land acknowledgement. So I'm just going to share my screen just for a moment, I know that in this virtual series, we don't often, we don't really have the chance to see each other. So I'm just going to share a small experience that I had recently. Of course, I didn't test this earlier, so just a moment as I share my my presentation here, here we go. Great. So just to share, I'm just going to put this, make this bigger. That Peel Region is on the treaty lands and territory of the Mississaugas of the credit First Nation, and it's the land that has been the territory and under the care of the Anishinabek, the Wendat Haudenosaunee Ojibwa, Ojibwe Chippewa peoples, and the land that is home to the Metis, and most recently, the territory of the Mississaugas of the credit, First Nation, who are direct descendants of the Mississaugas of the credit. So just noting, just at my office the other day, I noticed this Kestrel that had landed. So hopefully you find some of these moments as well where you can connect to the land. And we're also in the month of truth and reconciliation. So I'm just going to put in the chat box a link to the National Center for Truth and Reconciliation Lunch and Learn series that they're having the week of September 23 so that might be helpful to you and your network to learn more about truth and reconciliation in particular. Great I'm going to turn it over to Marco Campana, who's, you know, such an expert in knowledge mobilization, as well as the field of technology and settlement. And this is one of his last consulting work that he's doing with us and with the sector, as he's moving on to a permanent role with access Alliance. So if you have a moment to congratulate Marco in the chat box, feel free, and we, we really appreciate his generous sharing of curating this series with us, and it's been so helpful as we think about this new frontier of artificial intelligence. So over to you, Marco, to moderate from here. Thanks,
Marco Campana 3:01
Jessica. Thanks for and thank you folks posting in the in the chat. So as Jessica mentioned, this is the final session of a four part series focused on AI in the sector. And if you haven't joined us previously, you will be able to access recordings on the pnsg YouTube channel. Some really interesting session so session one, we had a presenter who talked about their AI project, looking at improving the efficiency of services so an inward looking AI project. And session two, we had someone who is sector adjacent talk about how their organization has been their journey towards AI, learning about AI in applications of marketing, as well as in their service delivery. And in our last session, we had a researcher who talked about the role and implications of generative AI in supporting newcomers, really interesting sessions that I think you'll enjoy. The recordings for if you're interested in today. Today, we're looking at additional local experiences of using AI. So before, before I dive into the introduction, I just want to say we're using slido to manage the Q and A today, because there's so many of us, it might get a little bit unwieldy in the chat, so I've put in a link to the to the slido page, the Q and A page in the chat, and I'll periodically update that there, but you can dive in there as presenters are speaking, and post questions, but also upvote questions, because I know people will have similar questions as each other. So I'm going to let each presenter fully introduce themselves and their work once they dive in, but just a few words about them and what they're going to talk about today, both represent organizations that are doing really innovative work with technology, including AI, and I know that I'm excited to hear from both of them. So first we're going to hear from Amy Holmes, who is access employment, VP of data strategy and digital solutions and leads their online services team. Now, Amy has been sharing access as AI work in progress over the last few years, which has been incredibly valuable for this sector. They're pioneers in this space, and today she's going to continue that, that tradition of sharing and talking about the time, effort, resources needed to integrate. AI into service delivery. Now in 2020 access launched Vera, their AI based virtual employment and resource attendant, and she's going to share today what Vera can and can't do, some of their usage stats. You get a sense of how it's been used and the impact that it's had on clients, staff and the organization as a whole, after Amy Fernando vittorino is achieves Vice President of Information Technology Solutions and Services, and he will provide us with a review of the important and replicable, I think work that achieve is doing to face the challenge of AI governance in a sector organization. So he'll outline how they're making AI solution decisions and where AI development is at Achieve Amy, I'll invite you to start sharing your screen as I finish my remarks and Fernando, once Amy has presented, please go ahead and just share your screen and dive right into your presentation. We'll hold the Q amp a until after both our speakers have presented. But as I mentioned, you can use the slido space to start posting your questions into it while they're presenting, and we'll moderate that based on there. And thank you all for joining us. And over to you. Amy,
Aimee Holmes 6:12
thank you so much, Marco. And thank you very much to the build newcomer Strategy Group for inviting me to participate today. When I heard about the series, I thought, oh, I want to be part of that. I want to help, you know, continue to spread the word about Vera in particular, but also learn from others who are presenting in the series. And I really appreciated the presentation that I saw that Marco and Darcy did. Unfortunately, it was the only session that I was able to attend, but I thought it was very candid and it was very relevant for the audience. And so that's what I've tried to do today in preparing this presentation for you, is to try to think about what you might find interesting. What problem did we try to solve? What did we do? Why did we make that choice? What did we learn? What surprised us, and if you're on the same path, what recommendations might we have for you? So just as a very quick preamble, I have, I have been quite sick in the last week, so I may need to pause momentarily to cough every now and then, but I appreciate your patience. Okay, so, as Marco mentioned in his introduction, I am Amy Holmes, and I work at access employment. And access is an organization that has been around for several decades, and we operate in seven locations within the GTA. We also offer pre arrival services for individuals who have been approved to come to Canada, both who are interested in starting their business or may be interested in working in a particular sector. And our focus is primarily on employment services. So we serve over 56,000 people annually across our different locations, through our 30 plus programs and through our online offerings. We also are very grateful to have an a network of employers that we work with as well that support the the job seekers that we're working with to secure meaningful employment. So let's get to the challenge, the problem that we wanted to potentially solve through an AI based solution. So back in 2018 we were experiencing a bit of a bottleneck at the onset of service. So we had a number of people reaching out to us. We had a high volume of inquiries that were coming in through walk ins, emails, phone at the time. On our website, each of our programs had a list of contacts, or there may be a contact at the bottom of a flyer, and that was hard to maintain as staff came and left, but that was the model we had, and staff would receive. You know, five inquiries a day, sometimes more, asking about, when does your program begin? What are the eligibility requirements? What does the program cover? How do I know it's the right fit for me these types of things, and so staff were expending effort on a regular basis to respond to these types of routine inquiries, basically answering similar questions in the same or a similar way, repeatedly. It also meant that if we somebody wanted to begin their service journey with us, they would have to come to one of our physical locations in order to do that and to get the fulsome information they needed. And for some individuals, that was going to be that was hard, it might cost money, it would take time, it would I. Um may not be convenient in terms of the time of day for them, if they were working a survival job or had other responsibilities that made it hard to come to one of our physical offices between the hours of nine and five, and given the volume of inquiries, we were also not responding to those inquiries in a way that was as timely as we would have wanted. So we knew that the response time also was presenting a barrier for people. So this was the problem or the challenge that we wanted to to look at AI into helping us solve. And so what we came up with was a chat bot solution. So this is an interactive chat bot on our website that is available 24 hours a day. So it would alleviate some of those initial barriers to access accessing information and services, because it was available at any time of the day. So even if staff were asleep, Vera does not sleep, and nobody would need to come on site in order to begin that service journey. They would be able to do so from their mobile device or the library or the comfort of their home. And so Vera, which is an acronym standing for virtual employment and resource attendant, is our solution. We were very fortunate to have secured a grant. We, once we, you know, identified our challenge and that we wanted to build a chat bot, we then had to go looking for some resources. And we were very fortunate to secure a global grant through Accenture, and that was just over a million dollars back in 2019 and that contribution allowed us to put together an internal team as well as hire external vendors. So the internal team were the ones that identified all of the potential questions that users might have and that Vera should be able to answer. They were the primary liaison with all of our staff in order to gather this information and confirm we had the right eligibility criteria for each of the programs, as well as to do the initial set of training with staff as the chat bot went online, they were also the primary liaison with external vendors, of which we had two. One built the chat bot itself, so brought it from ideation through to launch that development process took about a year and a half. The other vendor that we worked with was one that helped us connect the chatbot to our Salesforce based database, and I will go into a bit more detail about why that was important to us. So right off the bat, we engaged our staff, and this ended up being a best practice that we would recommend to other organizations that are considering using this type of technology as we know, one of the primary considerations and concerns about AI is that it can displace workers, and our goal from the start was never to reduce our staffing count as a result of putting this chatbot in place. It was to alleviate the time and effort that our current staff require to respond to these types of routine inquiries at the outset of service, so that they would no longer have to spend that time and effort responding to those queries, and they could put their efforts towards more complex tasks and allow them to have more one on one, facing time with their clients. So we asked our staff, what types of questions do you typically get? How are they composed, you know, how are they asked? How are those questions asked? And then, how do you respond to those questions? Because then we would be able to structure the dialog in a way that would sound natural in the responses as well. And what are those endpoints in those conversations? Once people have found out that piece of information, what's their next typical question? Or what would you typically offer them to do as a next step in that case, and in some of those situations, that would mean the handoff from the chat pod interface to a person. So our internal team really focused on gathering these types of questions and letting staff know what the benefits to them would be as a result of having a chat bot.
Aimee Holmes 14:54
Another best practice that we would like to focus on is about the technology that you. Choose because there are different AI based solutions. And depending on the challenge that you have and the solution that you hope AI will solve you've, you're probably going to have some different AI based solutions. So in our particular case, some of the things that were important to us in making our technology selection was that we wanted to be able to have the flexibility to structure the dialog in a way that would be very akin to what somebody would experience had they come in through one of our doors, or they had called us or emailed us. So we needed it to mimic that same intake flow. So it didn't really matter that they came in through a different door, they would begin that service journey in a similar way. We also wanted the chat bot to have an option in terms of conversation type, so it is it has a menu right off the bat, and users are welcome to select an option from that menu, and many do, and then that guides them down to the next menu of options and whatnot, and they're able to follow those prompts in order to get to the information that they Need and any calls to action. But we also knew that, given our clientele, not everybody speaks English as a first language, and maybe the the common the common queries that we were offering them, wouldn't meet their particular needs, or they didn't see their need reflected there. So we wanted to offer people the ability at any point in time to be able to enter in their own question, in their own words, and the chat bot should be able to interpret that and provide an accurate response. And one of the choices that we made is that we didn't want the chat bot to be able to try to come up with the best answer on its own, so to speak, by maybe searching the internet for that answer. We wanted to really constrain the potential responses, and so we composed over 100 responses, and now that's over 150 of typical responses. So the chat bots job is really to try to find the right vetted response, and on a monthly basis, we look to see how well the chat bot has performed that task, and I'll get into how we how we look at that and evaluate that performance. But in any case, we wanted to make sure that users had the ability to put in their own questions too, and so they were able to get to the information that they needed as quickly as possible. The third thing that we wanted to take into consideration was that the the chat bot would be able to integrate with our Salesforce based database. This is because our staff already used this database to field inquiries through more traditional channels, so we didn't want them to have to go somewhere else in order to get the queries from Vera, it should all be in one place so that it's convenient. And again, doesn't really matter which doorway, it's the fact that this individual has made a service inquiry. Fourthly, we wanted to use technology that is reliable and reputable. And we went with IBM Watson assistant and Watson discovery technology, which were some of the first chatbot technology providers out there, and had a reputation for having good natural language processing capability, and that NLP, or natural language processing, is what allows Vera to interpret the user's spontaneous entry or their unique question and try to field the right response appropriately. In the last four years since Vera launched, I can think of, really a handful, less than 10 times that we've been down for unexpectedly and that, I mean, sometimes we pushed a change that then messed something up, and we figured that out quickly and reverted it, fixed it properly, And then pushed it back to the live version. But generally, we have not had very many situations whatsoever where we have been unexpectedly down. And finally, we wanted something that we would be able to maintain from a cost perspective in the long run, because we knew that that particular grant. The money was finite, and so we needed something that was cost effective in the long term, the subscription. We need to have a subscription, and so that's one of our ongoing costs, and that subscription is usage based. So the more people that use the chat bot, the more it costs to run it. Okay? So let's get into a little bit about what Vera can do. So this is the main menu that users are greeted with when they begin their conversation. And we do find that the vast majority of individuals do select from this menu, programs and services being the most popular topic, and so at that point, users are offered the chance to do a bit of a screening so they could find out which programs are right for them. And basically what that means is they're going to answer a series of questions about themselves, their professional background, their goals, their language skills, their educational attainment, where they live, and other criteria that then in real time, are cross referenced against our program eligibilities list, and what they receive as a list of the programs for which they may be eligible, or sometimes people have already heard about a specific program, And that's the direction that they want to go. And so they're able to learn right about about that particular program immediately, and can make a contact with that team. And there's also the ability for people to register for an information session, and so they're able to see the upcoming information sessions for particular programs, select the one that works best for them and register. And all of that information that they submit to us and their registration for an event is captured in our database upon submission, as I mentioned before, they can also type their own unique query. For example, how do I prepare for an interview? And then they're going to be directed to our prepared response for that type of a question, which may again include another call to action, such as if they would like to register for a session that is related to that particular topic, or we may offer them resources on that topic as well that come from our own library or settlement.org or some other authoritative source, so kind of like a Google search engine, but we very much constrain the resources that can be searched and surfaced. Another option is that they may be interested in particular job search resources, and so we might have an a video about that, or an E Learning Module, or, as I mentioned, a workshop, so they're able to get that information right away. We also through, we subsequently got funding through an IRCC SCI project that helped us build out Vera's knowledge base related to settlement topics as well. And so in this case, we worked with settlement.org and Kath Cath, sorry, Catholic cross cultural services to develop a series of new responses in Vera related to these types of topics that you see here, such as legal advice or health cards, citizenship, that kind of thing. And so we've got short, snappy responses to that with links to authoritative sources for more information on those topics. For employers, we've got some potential information and resources, and the call to action here is for them to provide their name and contact information so that we can follow up for further information. And as well, for current clients and alumni, maybe they're coming back for service again, or they need help with something? Again, we'll collect their name and and be in touch with them. And those queries are rooted directly to if they are a current client, to their current employment consultant. Okay, so looking at some of our impact stats since the Chatbot launched in the fall of 2020, there have been over 130,000 conversations with the chat bot. In fact, the monthly average has really gone up significantly. I was looking at some statistics yesterday, and it's over 5000 conversations a month now. When it when we launched, it was about 1800
Aimee Holmes 24:42
looking at the year over year increase, there is a consistent increase over time in the number of people using the chat bot, which says to us that people are comfortable with this type of technology. They are trusting of the. Technology, and they're open to getting information in this way. And in fact, we find that over 50% of the conversations are happening during our office hours, when we are open, when people could call or could come in, the Chatbot is proving to be their doorway of choice. Of course, an important metric for us is not just how many people use the chat bot, but how many people go on to become our clients as a result of that conversation. And over time, that conversion rate has been consistent of about 25% or a quarter of all the users, which is actually a very high conversion rate, given the light touch inter intervention that this is, and that's doubly impressive to me, because, as I mentioned those that number of conversations has really increased. So if the conversion rate has also remained consistent, to me, that's a real marker of success. Overall, about 10% of our clientele start their journey with Vera and then another important metric, or key metric, is looking at, okay, so did it solve that initial challenge? Did it help people access information and services? Did it help offset the time that staff were spending on these types of queries. And in the last year alone, that represents over 13,000 hours of staff time, which is several FTE thinking about the number of individuals that that you know would be working full time and offset. That's a significant offset there. So once again, we didn't actually reduce staff, but we were able to offer staff that much more time back in their day as a result of having this chatbot in place. So as I mentioned before, we do have a subscription to run Vera that's something that is a an annual cost, and if we run out of credits, then we have to purchase more before the year runs out. From an internal team perspective, we do have a project manager or coordinator working on this. We have a database administrator who supports the project as well as we need to update program eligibility that the chat bot needs to check, or as we have new programs to add to to Vera, and if we also have we still use that external vendor as well to act as sort of a a support when we need to make more complicated changes to the chat bot. And is an intermediary with IBM, so we can't actually purchase that subscription directly from IBM. We work with this company as an intermediary, and sometimes there are security or stability, upgrades or updates that are needed that this vendor provides to us as well. As I had previously mentioned, we did have an SDI project which helped carry some of these costs through the 2021 to 2024 period. But as that funding has now wrapped as well, we are now supporting the costs of Vera through bits and pieces of different project funding. So we now see Vera as something akin to a marketing type expense, or platforms or online services type of expense, where it's the cost of doing business, and because we have a variety of programs, we're able to spread that cost out over those programs. So what did we expect? So we knew that maintenance would be required over time as our program suite changed, we knew that we would need to add more responses over time to the chat bot, because we knew that, you know, users needs change over time. So for example, as we started to see more Ukrainian newcomers coming to Canada, we needed to add responses to support people who said they were here under emergency travel measures. Or the cube, you know, the CUA T folders, for example. So we needed to be able to route those people directly to our programs for those individuals, we knew that staff would need training, and we knew we'd build up our own internal capacity to use this type of technology, and that we hoped anyways, that it would streamline access to information and services, and we didn't know at the outset how we would measure its success, but we knew we needed to do that, and as those statistics show, we've done a lot in that regard, and that we've assumed that we'd probably have to evolve the type of information via. Collects over time, if we saw that it wasn't sufficient enough for to help our staff make a service determination. What surprised us? Well, the chat bot improved its accuracy very quickly over time, so we didn't need to spend as much time after about the initial six months, it didn't require as much remediation or and by remediation, what I mean is retraining the chatbot to find the right response to a particular query. It got better and better as time goes on, and that's how this technology is designed to work. The more opportunities it has to respond to queries in the same types of queries. The better it gets at doing that, so the more accurate and confident it becomes. What we underestimated was That's how much training staff would need. So we did an initial blast of training and produced a guide, but there needed to be more consistent dialog with teams because there's turnover, or because the types of queries changed, or programs needs changed as well. So we really needed a consistent dialog with our programs so that we would better understand their needs and see how some how something that worked well in theory was actually working in practice, and maybe make some adjustments as to the type of information that the chat bot was collecting, for example. So when we launched, we asked clients, what's you know? What field did you have work experience in? But we didn't ask them what field you want to work in now, and because that might not be the same, that was a helpful piece of information for our staff to have. So we've added that we also found that despite the fact that the queries were in the same place as they found other queries, sometimes they weren't being tended to as quickly as expected. And so we've built in some notifications, either within the Salesforce system or email notifications to let people know that they've got some inquiries to follow up with. We are more reliant on the external vendor than I think we expected to be at this point in time, and some people choose to use the chat bot over speaking with a staff member, as I mentioned before, you know, more than half the people, or half the conversations take place during our office hours, so we were surprised by that, but that's good. We've, you know, it's good that we're giving people a choice that they're interested in.
Aimee Holmes 32:35
And I, as I had mentioned before, yes, we're really impressed that our conversion rate has remained consistent despite the increase in the number the significant increase in the average number of conversations. And then finally, one of the things that I think was a really exciting change about this project was that it became more than Vera as a result of building this internal team to be a link between the business or the services that we offer, and what we're trying to achieve as an organization and technology, we really uncovered a unique niche within our organization to support our teams to be able to get their work done more efficiently and more effectively. And so the online services team has grown over the past number of years, not only in number of staff, but also in the scope of the things that we now support the organization in and it has been truly transformational. There is now a real sense of camaraderie and dialog and excitement, I think about the opportunities that technology can offer us as an organization to streamline our work and save staff time and improve data accuracy and reliability, ease reporting, make data more available to our staff, collect better quality data from the outset, adhere to our equity, diversity and inclusion goals by collecting more of this type of information and responding to programs who tell us this is a sticky point in our process, or I don't know how to get this information. We're not currently collecting this information, but our funder needs it. How do we do that? And so there's a real synergy now, I think, between the way that we're using technology to support our business goals, and that can only have a benefit for our clients and and funders. So that's been a really exciting thing. Okay, so I hope I haven't gone too far over but thank you very much. That's the end of my presentation. I look forward to your questions
Jessica Kwik 34:52
later. Thank you so much, Amy, and we'll hand it over to Fernando. You. Can
Fernando Vitorino 35:05
everyone see my screen? Okay, excellent. Amy, that was a very informative I really enjoyed that. I was actually impressed. 25% conversion rate. That's incredible. It's a hard thing go from a high touch service to what folks consider to be a low touch service. But of course, as you improve the product becomes a higher touch service, as a system learns so, very impressive, incredible. How are you folks? Of it danced, I'll try to be probably a little less impressive in terms of our journey, but we'll talk a little bit about where we're at, how we got there, and obviously, hopefully give you folks some ideas around how in your organization you can adopt some of the lessons learned we have. It's been a very smooth transition, it seems there at Access compared to what I'm about to show you, because we've gone through quite a bit of a learning so let's start with the purpose. Why? Why bother doing AI so for us, or, sorry, I achieve, those folks not familiar with achieve. We do employment services. We do newcomers settlement, just like access does. We also do language assessment, language training, women's program, youth program, focus programs. We also have an IT pillar, where we provide services to other spouse, such as comes to connect and the heart system. So it was important to us that we build solution services that are cross functional as best as possible. Just like all you folks, you have finite budgets, but you don't know if they're going to be continued or not. And so it was important that we build things that they're easy to sustain and that can be useful across one of the 51 programs that we currently run today. So we had to ensure that for doing it, it's enhancing client experience, it's servicing the spouse, both for internal and external use, improving our operational effectiveness and quality. And just like access or focus wasn't necessarily on reducing headcount, it was just allowing our staff to do more. I'm sure, just like you folks, our staff have a pile this big and this much time, so it was an exercising how do we help them be much more effective and then obviously achieve our long term growth and sustainability, especially as those call for proposals come in and you're not sure funding is going to be available for all those wonderful programs next year. So we talked a little bit about enhanced decline experience. So the key for us was ensuring that clients access services in the way they want. Access Services, not necessarily coming to our location, necessarily going to our website, maybe it's our phone service, and so forth. So we've launched something called an omni channel approach, which is whatever the service you're looking to receive. We want to be able to allow you to get it either through our app, our website, in person, phone and so forth. We need to make sure that what we do is reducing the administrative and overhead for ourselves and our service provider organization that we partner with. And the value of this exercise, we lose a lot in terms of what's successful and what's not successful, because it goes into usually the wonderful work being done by our staff, and they don't have time to share it collectively. So with these tools, we're going to start to see patterns of behaviors that allow us to improve the delivery of services we move forward. So start with a governance or, more importantly, how do we keep out of trouble? And I'm getting lots of advice from board members, my fellow SLT and everyone else, of all the things you got to worry about in AI. So good news, I said, No problem. Let me find out what the AI defined standards and legal framework is. If you find it, please let me know it doesn't exist. Okay, so here's the good news, you're not out of compliance. The bad news is you don't know what compliance is, so we're struggling a little bit of what we should do and not do, and this is part of, how do we keep our clients data safe? How do we ensure that we're being transparent? How are we doing the right things? And like I said, this is this is a tough place to be in when you're trying to jump into our industry, in this area without a roadmap. So we came out with some guiding principles. And the good news is there is some guiding principles. And the Government of Canada did publish the guiding principle for use of AI in government. And I'll let you read through them at your own leisure. But the key ones that we came up with were the following six, transparency. The key is, before you use a piece of AI technology, it's important that the consumer, the end client, was using it, or, sorry, a benefit of that technology is aware that you're using. The Accountability you have to ensure that if you're using this technology, you're using it responsibly, and responsibility means ensuring that you know how that data is being stored, where it's being stored, who it's accessing it, fairness as we go through it, and I'm gonna take a pause here. You've probably heard the term generative AI, or machine learning, which is an older term, and AI. And I'm not sure if everyone here knows a difference, so I'm going to try to explain the best I can. One of it. AI is really just a bunch of algorithm, bunch of decisions that someone makes. And the example I provide is, think, when they're teaching cars how to drive, the paper bag blows across the front of the car. When they first launched these these tools, the car would break me. They'd say, Whoa, there's something in front of me. It didn't know how to differentiate between a paper bag and, let's say, a person, and so it had to be taught every iteration or every concept of a paper bag or a plastic bag or a piece of whatever that's coming across. So that is a long process generative AI is it understands the concept of a bag that's something you can hit, and it can keep going, and you don't have to break and it gets there by learning over and over and over so it gets smarter. And I'm sure actually went through the same process. The more interaction you have with particular target audience, the more effective it's going to be. Here's the problem, and we saw that with when we'll talk about one of the examples we have in VLA is the people training it generally are out of the eastern United States, white, male and in university educated. That's your target audience for most AI training. So your challenge is, does that allow you to extend that knowledge to other groups, maybe, maybe not. So fairness is a tough one, because you're using technologies and tools that have been built by other folks for the most part, and they may or may not reflect the groups that we support. And so it's a tough one to jump in with privacy, no different than any other way you manage your client data, you have to consider the privacy of IT and security. Who has access? Why do they have access? Where is it being stored? All that good stuff, and probably the most important education ensure that people know how it's being used, how they can use it, and how can make their other jobs better. So, solution approach. So how do we do this with a limited expertise and budget that we have? We are not experts in AI. Very few people are, and I say that not in any dismissive way, just because AI has evolved probably in the last 12 months, more than is involved ever. So there are very few folks who truly understand how best to approach it. And of course, in our industry, we have very limited funds, not just for capital to actually put it in, but the cost to manage it, as Amy indicated, is pay as you go. So effectively, the more popular it gets, the more expensive it gets, which is a wonderful problem and a very ugly problem to have. If you've got something wonderful, and you find out later on, you can't use it anymore because you have no additional operational funding to support it. So what did we do now? I'm not trying to sell you AWS services. I just we picked these folks for a few reasons. Number one, they had some great funding opportunities for profit, but number two, they take an approach which is very distributed, meaning you kind of pay for the little pieces that you want. And that's the key with them. They built a lot of little pieces, and they have a support network that's quite large, so we're not tied to this particular company, but effectively, that's where we started. Just like access name their product, Vera. We called ours Valerie. We don't have an acronym for it, and honestly, I called it Valerie because sound like HAL from Odyssey, 2001 for all the geeks in the audience, I'm sure that might be funny for everybody else don't read by majority of development services we based on best practices. What does that mean? So we didn't try to get too innovative. We looked at what was out of the box and try to consume those services to get us started.
Fernando Vitorino 44:32
Obviously, our focus is on building our own Integration Services team. We're starting to develop a support team. As we build these, we build all our Runbooks to make sure that we can share knowledge across the IT department as well as across the production, support, sorry, program, delivery, audience as well. And the key for us was reusable, so we try to build everything common. So an example is. We're trying to build a tool set that offers voice recognition for registration. We didn't build it for language services in mind or language assessment in mind, or language or EO. We try to do it as common as possible so that we could redeploy and obviously, as we developed our Gen AI tool, our focus for our client base is newcomers to Canada with low to moderate English and French. So we try to look at tool sets that manage against a CLB level four audience. So as I said before, common solutions as much as possible, and API. There we go. The IT guy. I'll try not to use too many acronyms, application, processing, interface. What does that mean? A separation between the service offered by the vendor and how we consume it. And we do it for the main reason that it makes it more and more cost effective, and it keeps us agnostic. So if we need to unplug AWS and move to another service later on, there's not a big issue. We basically say, no problem. We'll move over to IBM. We'll move over to Azure. We try to build things that are common. So whether or not you're accessing the service through website, phone, kiosks, it's the same service. And so for us, it was important that we had this as a service hub, meaning all the tools were collected in one managed area, and basically this is the traffic cops to ensure that access to data, even though it's coming in through a cloud service, even though it's being managed through or sorry, engaged through third parties, we still have the final say in terms of who comes in and what they see. So what have we learned? So far, it's really expensive if you're not careful. So what we tried to do is go for that low hanging fruit. So we looked at things that AI does really, really well, chatbots, or virtual assistants, was a great one. That is something that everyone does, meaning that a lot of folks find a purpose, and it's a lot easier to deploy. And the other thing it does really well, things like pattern recognition, so part of our we'll talk a little later on, is things like fraud detection, because that's an easy thing for an AI engine to detect, because it's looking for the same pattern, state behaviors, and then cybersecurity. Why do we pick this? Because it works. I'm not reinventing the wheel. I'm not worried of whether or not I'm too far into an innovative sphere that I don't have the expertise in. I have real world examples to fall back on. We learned we had to have a service hub, because you may not be aware in it, the greatest vector, greatest source of intrusions is, of course, employees. They leave USB keys around you, share passwords. You do all kinds of silly things. But second is APIs, if we build them, and then we forget about them, and we forget to lock them down. So it was important that we have a second layer of defense, our third layer of defense against intrusions. And the good news is, because we are building APIs for other services, and it it was easy just to re, re leverage that expertise change management. It's important, as in your own organizations, we found in ours, it takes time, and it's important that your clients understand what you're doing, your staff understand what you're doing, your partners understand what you're doing. Because AI is scary, and we should move slowly, and we should move with tremendous amount of information presented. I love creating these business advisory committees. I do with hearts comes to connect language assessment teams, and basically this is advice because I don't have the answers. I don't engage the clients directly, and it's important that everyone's at the table to understand how it's going to impact them. Going back to those principles of fairness and equity, it's important that everyone has a voice, because you want a solution that's going to work for your entire client group and all the stakeholders. Take an agile approach. Move slowly. Take bite sized pieces, deliver little pieces, little wins, get it working, get people comfortable, then move on to the next so what have we done, or what are we planning on doing? So I said before we start with the easiest one, cybersecurity. So for us, I'm sure all your organizations have firewalls and and tools for keeping the bad guys out. No different for us, we implemented an AI tool, which is tool set called forwarding analyzer. So what it does is, out of the gate, it was really dumb, and it basically kept everybody out. So we had to tell it to ease off. Yeah, and what it does, it looks for behaviors. There's probably a term maybe some of you folks aren't familiar with, vertical presence, penetration versus horizontal penetration. Most attacks where folks hold your data ransom aren't a result of a single point of entry. Somebody implements bot or a piece of code, and it just watches you. So for six months, it'll sit there and track email between clients or sorry between staff. It'll track transactions. It'll track anything that gives them access to places that they want to access it, and then bad things happen after that. So you've got a lot of security that's focused on that front door you don't have access or security been watching the interior doors. So that's what a tool, an AI tool, like 40 analyzer, does. So basically, it tracks the people that are inside the building and what they're doing, and I know Big Brother, but what it does is it looks for things that they don't know don't normally do. So example of my Vice President of Finance seems to connect from an IP address in North Korea at three in the morning, downloading all our customer records. He probably wouldn't do that most of the time, in which case, then we would shut it down, but the that person who access it had the VP of finances, access credentials, they had access to the equipment, the software they needed, so it should have allowed them, but in this particular case, it would stop them, because that's a behavior that's not normal for that particular user. So that's a great thing for what AI does, because it looks for behaviors, looks for patterns, and find things that don't fit. And it tells you, you know what this hassle that you guys do for changing passwords? Maybe you don't need that anymore, because we found that's not the biggest issue, but you definitely should have MFA, the next one we're looking at, very similar to what axis talked about with Vera. We're in the progress of building out our virtual assistant, and we're focusing on one particular area or one particular program, or looking to shift that out. We are not quite as far as what what access is doing, but we're very excited, and the key for us is this is this is going to be a great tool to learn about our clients. What are they asking about? What are their concerns? When are they talking to us? What do we get right? What do we get wrong? And this is going to help us understand how to best serve those clients, and to align our staff and our products against what our clients are asking us to do, the type of information you all have in your organization, but unfortunately, it's between the years of a couple 100 different people, so this kind of this consolidates that knowledge into one tool set. So we're very excited about that, hoping to launch that shortly. Another piece we're looking at and something we're building for our clbpa tool, but we built it, like I said, in a way that we're hoping to be able to deploy for other tools as well, is a remote assessment tool. So one of the hard things with a high touch service like language assessment is how do you get more folks through a language assessment process or any type of test monitoring tool without throwing more staff at it. So this is a tool that's watching the behaviors of clients. It checks to see if they have the appropriate access. They have multiple screens open. Do they phone in the background? Are they looking away from the screen? So all the wonderful things that folks traditionally do to cheat or to potentially get a proxy for them to do the test is what we're looking at doing as part of our CLP role.
Fernando Vitorino 53:59
VLA, so we did a proof of concept. This was a fun one. Half the RCC in participation for the call proposal next year, and they asked us to see if this was possible. And this was basically, is it possible to do language assessment remotely? And this is not just the two that are easy, it's all four. So being able to interpret a writing test and a speaking test, and we went through this exercise, and we able to find that we were at least a 95% accuracy, up to CLB level five. After that gets a little harder in both French and English. So this is an exciting opportunity. We're looking forward to getting approval to move forward with that in the next fiscal year, job matching this is another initiative that we're planning on engaging on. So one of the biggest challenges for our job employment counselors is being able to get through all those resumes and being able to assess whether or not the job. Posting fits their needs. And we want to not just, of course, be able to get posting, find our client bank and identify the right candidates, but we also want to, of course, notify them automatically, so they'll say, hey, you've been matched for this job. Please apply right away, and then just have the counselors engaged on following up and typed resume. The other thing we're going to do is also look at success factors. Why are certain clients being selected and others not? And it's a great training tool for our clients to be able to improve the resumes, and, of course, improve their the type of quality of data they're including our phone assistant. So an extension of the virtual assistant, or Valerie, would be the concept of whether or not you're coming in through the kiosk. Their website is just converting this through a different modality, which is when you call in, and of course, it would authenticate, and then they would be able to access all the information they have in our CRM. Can you just let me know when my next appointment is, if you confirm I got my my score. Did class get canceled? That type of that's it. And then, of course, I will turn it back over to the organization. Sorry to the team for any questions. I i
Jessica Kwik 56:23
Thank you so much, Fernando and Amy. I'll turn it to Marco to moderate from here.
Marco Campana 56:29
Yeah, thank you both. Tons of information. We've got some questions in spite of mainly for you Amy at this point, but Fernando, I think you may have blown everyone's mind a little bit there. So I'm letting, I'm gonna let them pull their pull it together and add some questions. So Amy, I'll jump in with you first, and we'll work through what's there. I clearly have some questions as well that if we don't get audience questions in the meantime, the first question again, these should be quick, because I think you actually answered most of these in your presentation. But was the Chatbot built and deployed by access itself, or was this project accomplished by a third party?
Aimee Holmes 57:03
We used a third party vendor to help us develop the Chatbot.
Marco Campana 57:08
Perfect. And what is the rough figure for this kind of AI chatbot you mentioned in the beginning, it was a $1.1 million investment from Accenture, that's right, I'm wondering, what are the ongoing costs of maintaining it. Now, for example, you mentioned that you know, as you're more successful, it becomes more expensive.
Aimee Holmes 57:26
Yes, that's right. So at the outset of this year, we estimated that the subscription cost would be about 22,000 and we are now. I'm trying to think we're over, like, where we should be by this point in the year. So we will need to top that up again so I could, I could foresee it, it being closer to 30 this year.
Marco Campana 57:55
I wonder. I'm just going to build off of that, because it raised a question for me about the idea of, how do you have this conversation with funders, right? So the more successful you are, the bigger your budget will need to be. But we have, you know, five year three year contracts with funders, with typically fairly inflexible budget areas. Is this an area that we need to figure out as a sector about how to have these conversations and build in some flexibility with with funders and with budgets.
Aimee Holmes 58:24
Yeah, it's, it's a really good question, I think so. I mean, I think funders need to understand the value of these types of things. So yes, while perhaps it's going to cost more to do this thing. The benefit is that, you know, we didn't have staff spending 13,000 hours, you know, doing simple things that the Chatbot was able to accomplish for them. And as a result, look at these outcomes that we've had in the programs, or look at our client satisfaction rates and and sort of make the case based on the actual impact, or from a you know, newcomer perspective, look at all the 1000s of people that were able to access information within minutes, whereas, you know, maybe they would have waited two weeks or something to hear back from a person or something. So this is helping the funder achieve their goals as well in getting people access to information and services more quickly. And then, you know, seeing the the value of those services increase as well. We have, as I mentioned, we have built in the costs for Vera for this current fiscal year, since we didn't have any dedicated project funding anymore for this project across the variety of programs and services that we have and we didn't like, and sorry, I should say, when we submitted proposals to IRCC for the most recent call. All that of projects that will begin next April. We didn't hide Vera under, you know, sort of like marketing cost or something. We put it front and center in those budgets because we wanted to communicate that this is an important and viable way that individuals access our services and come back again and get more information and and are supported along this service journey. So this is something that, you know, we wanted to clearly communicate, that this is part of what we do now, how we do it? Yeah, so
Marco Campana 1:00:36
it's part of your core budget cost, yeah, integrated into very and you mentioned various programs. It's sort of yes, because it's not just in one program area. That's right, yeah. And Fernando, I want to bring you into that part of the conversation, actually, because similarly, you've talked about breaking silos. You want to build for every department, but you also have worked on a project that received some IRCC funding, and as you mentioned, you're waiting direction from IRCC. What's that conversation like with with a funder, in terms of this is not a one and done process. This requires ongoing investment. And how do we, how do we incorporate that into the conversation?
Fernando Vitorino 1:01:13
Not easy. So one of the things we we're looking at within access maybe just unique to us. It is a separate pillar. And so we we've introduced the models to start charging programs per usage. So a lot of our shared services cost, whether it's Azure or recognize AWS, we do want to per billing cycle by actually by computer. So we kind of bundle these costs into each of the different programs, and then they pass it on to the funders through one of the IT costs. Having said that, we also get direct funding for specific initiatives. And so effectively, we've got some funding for MasterCard as an example. We've got something funny by RCC and ipsd and so forth. The other thing is to consider is, as you folks go out in your own AI journey, a lot of the players out there do offer funding to not for profit. So we have received funding from Azure, and we have direct funding from AWS and from one of our service partners. So lot of opportunities and folks who put together proposals. You know, talking about value is easy. I think when you start to put these together, it's, there's a lot of opportunities to get, get you folks off the ground.
Marco Campana 1:02:34
Great. Thank you both for that. There's a follow up question in the chat, actually, and it's related, and I want to actually, I did the math. 13,000 hours is the I think I did it. If you, if you assume 35 hours a week, 52 weeks a year, that's the equivalent of seven full time staff. Thank you. With the explanation about saving work hours for staff, the rising cost of subscription. Does the question of cutting the number of staff rise in these conversations with funders? So not internally, but our funders saying, Hey, how can we cut your staff? Or, how can we, you know, create efficiencies in how we fund you? Amy, maybe I'll ask you first that you've been doing this for a while.
Aimee Holmes 1:03:13
So is it the question that's in the chat there? Yeah, yeah. I guess the question, you know, no, I mean, like I mentioned, for our sake, that was never our goal at the outset, and when we reported back to IRCC at the conclusion of our SDI project, you know, we emphasized how much more we were able to do with current staff as a result of having offset the time that they spent on these types of routine inquiries. So that, you know, it was clear that there is a real cost to these types of tasks for the average organization right now, and that, you know, supporting organizations to go down this type of pathway is actually good business, because, you know, then the staff are able to focus on the things that get people into this line of work in the first place, and that they feel passionate about, and that really make A difference for the individuals that seek our help. Oh, you're muted.
Marco Campana 1:04:27
Of course, it's always one of us in one meeting. Fernando, do you have any anything to add?
Fernando Vitorino 1:04:30
Yeah, no, I echo Amy's sentiment. I think you know the challenge for us. You know, we maintain nine locations, but we operate between 830 to 430 that's not when your clients are operating so you know, when you have a chatbot that can provide a service at two in the morning, language assessment at three in the morning, this is clearing backlog. This is clearing waitlist. This is offering services at the time your clients needed that does not require you that the value is not reducing staff. It's increasing the value. Staff can offer 20%
Marco Campana 1:05:03
nice I wonder if part of our conversation with as a sector, as well as with with our funders, is that what we're finding is that these tools allow our staff more time to do the work that's really needed in terms of service delivery, instead of the administrative so those, those seven, I mean, in a couple years ago, your presentation was three full time staff now it's seven full time staff. Hours saved. I'm assuming you haven't cut those staff from your organization. Instead, those staff have been freed up to do the actual work of serving their clients directly. Amy, agreed. Yeah, perfect. Related to some of the some, some interesting questions are, and I think this is an important point, what are some of the steps you're taking or considering to make these AI tools easier to use for clients who have low digital skills? And I wonder if, in addition to that, in the context, and I think you mentioned it, Amy, these tools are not for every client, but these tools are for clients who do have that context. But is there a responsibility as a sector for us to ensure that we are helping those with low digital skills gain more digital skills to be able to access these types of 24/7 service modalities?
Aimee Holmes 1:06:20
It's a good question. The chatbot that we have is available on our website, and it's also what responds if somebody was to integrate, sorry, have a conversation with us on Facebook Messenger. I had forgotten to mention that earlier, so it is for individuals who can at least get that far as somebody who either doesn't have a digital device or access to the internet or the ability to navigate those tools is not going to interact with Vera anyways, because they won't have gotten that far. But as you said, Marco, an individual doesn't have to start their service journey this way, they always have the option to close the chat bot on most of the pages on the website, it does not open automatically. However, on program landing pages it does, and in fact, it's kind of a fast forwarded version of the conversation. It says, you know, looks like you're interested in this particular program. Would you like to connect with a member of the staff, see if you're eligible for this program or register for an information session, and for some programs already, like, fill out our registration form if you're interested. But then there's always that option of, you know, no, I'm just browsing or something, right? You know, go away kind of thing. But to get to the, you know, the question about responsibility? Yes, I do think so. I think that organizations have to meet clients where they're at, and it is an ongoing conversation that we're having as well. In fact, we're thinking about, we're planning to add more questions to our registration form to try to get a better picture of aggregate digital literacy needs, and using that information as one of the the things that help us make an appropriate program determination for people, it's already happening, but it's happening in more of a verbal context, but we We want to sort of back that up with information as well. And we, we don't offer at our particular organization, we don't offer remedial digital literacy skills training. And I don't know if every organization has a responsibility to do that, but they do have a responsibility to know who else does that in their community. Can support individuals to access that as they as they need to. One of the things that I think, and this relates to another question I saw in the chat, was about languages, and if Vera responds in other languages, or is available in other languages, not really we, we have trained her to be able to respond to some French inquiries, because we do have French programming. And so, you know, she has some French responses as well that she can offer that we've composed. But generally, if people are writing in another language, we invite them to come into our location where they would then be better connected to an individual. The IBM Watson technology does have the ability to do some translating and whatnot, but we didn't feel confident enough to to make that a consistent feature on the chat bot, maybe it's something that we can reconsider again, because it's been a couple of years since we looked at that, but at the time, we weren't satisfied with the quality
Marco Campana 1:09:51
Fair enough. And Fernando, I wonder actually going to start backwards, because you mentioned your phone assistant is planned to be available in multiple languages. And I wonder. How you're addressing that the imperfections of machine translation, even though it's improving by leaps and bounds,
Fernando Vitorino 1:10:07
it's easy to address digital literacy. Spend a lot of money. So I'll give you the hierarchy of costs, video, audio, text. So when we talk about prescription costs, subscription costs. If you are interpreting video, really expensive. You're interpreting audio, significantly less expensive, and text dirt cheap. I mean, for a couple $1,000 a year, you can write your own chat bot. And the same thing with the other modality, which is real time versus batch. If you're willing to let this sit for a couple of seconds, it's a heck of a lot cheaper. I just put that out there. So if you want address digital literacy as easy, you can pay for a service, put it within your own chat bot, or purchase one directly, tie it into recognition or connects or whatever tool set helps you use and tell it to interpret as many languages as you want in real time with a load digital literacy requirement, I push a button and speak it'll cost you more, so I just give you that option. Is, if you want to make it easy, yeah, spend more money. I know that's a flipping answer, but that's the right now. That's the reality, because to try to get a tool to work at CLB level three or lower, or working with folks who have very limited digital skills, this is a high touch community, you probably are going to have to be forced into a direct face to face. So spend money or throw people at it, which is also spending money. So,
Marco Campana 1:11:46
right? But as Amy kind of mentioned as well, is that there's for those who for the service, even in an omni channel capacity, even where language might be might be translated in a phone Assistant, there's still always going to be a role for the the human intervention, in the referral, going back to the conversation Fernando in particular, that you brought up in Amy on talking about the costs, what would you recommend as a and the idea that there's money out there if you if you can find it, if you have the capacity and the human resources to find it, what would you recommend as a key project or a starting point, I guess, for a smaller organization to try AI as a bite sized project. And I want you to think here in terms of something that they might be able to build as a bite sized project, but also something, and I think Fernando, you kind of alluded to this, something that they can use that's already been proven, because that's what you're building on. You're building on proven technologies, for example. So what would you recommend as a starting point for, let's say, an organization that doesn't have, you know, a robust IT and online services department, that doesn't have a robust staffing to go after the big pieces of funding, but wants to build a proof of concept that they can use to then go after that funding or to figure out what might work for them.
Fernando Vitorino 1:13:01
So for an AI perspective, I think there are two things that offer the grace value. The first is client interaction, and the second is operational, administrative effort that can be reduced. So you have to look at your own organization and say, what are the things that we're spending a lot of time doing that's not directly servicing our client, and how is our client not being serviced in the way they want to be? I, for example, 24/7, and so forth. So a chat bot is or a virtual assistant is a really good starting point. And for the sole reason that there's a lot of different partners out there, even if you don't have an IT department that can help you with that, the only thing I always recommend, when you engage a partner, don't ever outsource your project manager. You don't outsource your data. Own the data, own the person who has control over that data, how it gets implemented. That's the expertise that you can bring on board and you can hire. And as from an operational perspective, those are hard, because when you start to look at what are the things we do all day, you have to take away what is that's the competitive advantage? What's the value you bring to your organization? Is it going through hundreds of resumes and looking at them? Or is it in adding expertise of helping people find jobs, and if you can automate those tasks, then you're going to be able to spend more time on the things that you should be doing? Are you helping your customer? You helping your customer? And if you've done some, if you have something your organization that's tedious, I guarantee some other organizations doing the same tedious things. Those are the things you start with. Go to an organization, go to the marketplace, whether it's AWS, IBM, Azure, 50 other folks, and say, Do you have out of the box a service? And my recommendation is, implement that service. Don't try to change it. Don't try to hit 99% of the tediousness. Replace 40% 20% and then build on bite size. Get in the game. Implement that one piece. Use and then add on it from there.
Marco Campana 1:15:05
Amy, did you want to add anything?
Aimee Holmes 1:15:06
I loved what you just said. Fernando, I just kept nodding and nodding here like a bobblehead doll. But yes, I think you really honed in on some really important things there around, what is tedious, what is routine, what is you know, taking away from as opposed to adding to. And, you know, I think it's really common for people to kind of just get used to the way it is, or something. So, you know, it might help to, you know, ask people to keep a time diary for a period of time. Or, you know, have a day, or, you know, a session where you're you're really trying to take a fresh, unique look at how you do the work that you do. And I liked what you had to say too, about thinking about access to the services, or the way that people receive those services. Or, you know, you can get feedback. You know, look at your surveys and things like that, from from what clients are telling you along the way, another thing that came to mind for me is you probably already have some subscriptions to different product, products, zoom, you know, being one of them. Or maybe you use Spark higher or job scan, or some of you know some other tools within those different platforms, there's more and more AI features coming online, like we see that all all the time now. So maybe there's something that you already have access to, that you may not be using, that wouldn't even cost you anything more, because they're building it into the existing costs of the subscription that you have. So this might be something that would help you take minutes in meetings or provide a summary at the end of a conversation with a client, or to compose text for email templates or for a presentation, or to suggest images for something that you're working on. If you've got a canvas subscription, there's a neat way to use AI there to kind of create your own images. And, yeah, just look for time saving measures within the things you already pay for.
Marco Campana 1:17:19
I love that, and I appreciate you're both also focusing on internal uses, not just client facing. And perhaps that's an important way to answer the question for someone who's looking for a bite sized project, and the idea of risk and security and confidentiality, is that if you use it internally first to become comfortable with it as a as an administrative tool, as well as, as you say, use, I look at the tools you have like Microsoft, you know, the AI copilot, and you know, there's various every tool now is building out some AI companion version. So to look at those themselves as internal tools to become comfortable before you do anything that might be outward facing and a little bit riskier when it comes to use of client data, there's an interesting question in the chat and I want to focus on you first, Fernando, because I think you've kind of addressed this in some of your upcoming tools. But some people wouldn't want to interact by typing with a chat bot, but they might be much more comfortable with voice interactions with the chat bot. And when you look at your VLA and your virtual assistant, sorry, your virtual assistant and your phone assistance. Is this something you're looking at where people can actually be asking questions with voice instead of text?
Fernando Vitorino 1:18:29
Yeah, absolutely. So the 50,000 foot technology answer there. So I mentioned the concept of the API. The API is the engine, right? The interpretation translating it from text or voice is is a common service, meaning that once you go through the trouble of putting the logic behind how to answer questions, how they answer questions, if you have that architecture separate, where we partner with just make sure to always separate that, then that's a small ship. I'd like to do it by voice, great. I like to do it by text, wonderful. I like to do it by some type of forgot those things are IBR, what do you call them? Systems for the Deaf. I forgot the term, I apologize. Whatever tool you the modality becomes irrelevant, because effectively, it's the heavy lift. Isn't answering the question. And same thing with the response, whether or not the response comes back to the text or a voice in whatever language, that's an easy exercise, if, if you've built it Amy,
Marco Campana 1:19:31
is that something you're looking at for Vera to be, for folks to be able to interact via voice, eventually, it's
Aimee Holmes 1:19:35
a good question. Actually. No, we hadn't, we haven't explored that particular way to input an answer, but now you got me thinking.
Marco Campana 1:19:47
Fernando said earlier, if you want to spend the money, you can do anything, right?
Fernando Vitorino 1:19:51
So, we've got, we've got, sorry, I couldn't remember. TTY.
Marco Campana 1:19:55
TTY, yes, thank you. The the phone systems for hard of hearing. Um. Um, oh, okay, yeah. So a last question that's come in, and then I'll try to throw one at you as well. Is data ownership, slash privacy, slash confidentiality, a concern with AI apps in terms of, you know, external corporate entities possibly storing this data. And Randall, I'll start with you, because you said, own the data. So in this context, what does that mean? Because you're going to be working with cloud vendors in this space, right?
Fernando Vitorino 1:20:25
I'll try to answer this as shortly as possible. So data, data in a service is not data. What do I describe that? So if you use copilot as an example, your data gets put into this large language model they own, and they're using it, and they're selling it. And so any identifiable piece of information you have that now belongs to Microsoft, if you're using an off the shelf solution, I always ask two things, where do you store your data? And I want proof, and it's got to be in Canada. And number two, how do you anonymize data within the organization? Metadata is fine, but how do you anonymize it if they don't have two very clear and answers that you understand, forget it that you understand, that don't use the service, because you basically saying, all my client data belongs to someone else, and if that client accepts that great even, for example, a job matching tools. I've indicated them. I don't want to see resumes with people's names on it and addresses. You can put everything else, and that's the only you're going to guarantee that data isn't used. If you have a large IT department, you can build your own language, LLM, and keep it on your own premise. But for everybody else, your only hope is keep your data anonymous, or ensure that the partner you work with keeps it anonymous and keeps it in Canada.
Marco Campana 1:21:48
And I've thrown into the chat a reminder that everyone who's IRCC funded has to adhere to IRCCs privacy and security requirements. Now there's there's no language there about AI necessarily, but there is language about the use of client data, for example. And so I think that's really important. And going back to and going back to that bite sized question for a smaller organization, again, one of the risks of using external models is you can't, you don't want to just be uploading either organizational intellectual property or or contact or personal information, identifiable information for clients, because in most cases, that data is being used to train models, and even when you can turn off, don't train the model. Do you trust that model? For example. So in chat, GPT, the most famous and popular, you can go in and say, do not use my my data to train your models, for example. But I think that's really important, that when we're looking at these, these tools and technologies, the idea of being very clear about how they're using, who has access to the information? You know, some note takers, for example, say that you know it's it's all it's all locked in. It's encrypted. But our IT department may, from time to time, need to access your data, but it's not being used to train a model. Okay, well, what's your threshold as an organization when it comes to privacy, confidentiality and security, and how comfortable are you? Are you with the policies and terms and conditions, which also change over time? So you've also got to be ready to keep that in mind. In your case, Amy, I don't think Vera asks for much information, except when it goes to the point of doing an intake and registration through because you can do that through the system, right? So how are you ensuring that that stays secure
Aimee Holmes 1:23:25
for sure? Yeah, I mean, by design, we didn't want Vera to ask for very much, because it shouldn't. It shouldn't take a long time to have a conversation with a virtual assistant, and so we're certainly not collecting any sensitive information. We don't need address, we don't need date of birth. We don't need, you know, sin number, any of that kind of stuff. It's like, name, phone number, email address, and then some background information, like I mentioned about language level or work experience or more goals or something like that. Nevertheless, it is still, you know, with a combination of those things, it's it's still some personally identifiable information. So the technology that that we're using is allows it to be stateless, so that means that the information is passing directly into our Salesforce database and doesn't get retained by IBM. And we made that choice, you know, for the reason that we would own the data and we'd be able to control and we know where that's being stored, which is in Canada. And we we don't ask any other questions in the the chat bot or encourage people to share information spontaneously. You know to try to limit any personal information that the user might opt to share within the chat bot, because if they have entered a query in that in that sort of free text field or whatever they're able, you know that information is is used to train. Train our model. It's not supposed to be used to train, you know, IBM, other models, or whatever it's supposed to be, you know, contained within our instance. But nevertheless, you know, we don't need users to be, you know, writing in my name is so and so, you know, really, it should be a question, like, I need help finding a job? No, that's great.
Marco Campana 1:25:21
I mean, that's great from a tech perspective, but you're also talking about what I see more and more organizations doing, which is almost staggering the intake process. Because intake process, because it can be incredibly cumbersome and create a lot of friction for people who are like, Why do you need all this? I just want to talk to somebody so staggering a small intake that gives you enough information to set up the next appointment where you get more information move on to the needs and asset assessment, and kind of build that out over a period of time. Is also something that I think is technology is giving us as a lesson, because friction in the interaction, people are not used to it anymore. I want to go and I just want to sign up, and I want to get instant services, for example. And so how do you make sure that you know you're getting enough to provide them with something, maintaining data privacy, but also not turning them off from taking the next steps, for example. So we're almost at the end. So I just want to say thank you both so much. Folks will get the recording. I want to throw in the chat really quickly an upcoming series that's being run out of Atlantic Canada by Arisa and i Sans. It's a seven webinar series that actually starts next week, AI's impact on settlement and language learning some really interesting sessions coming out there, which will guide you to even more folks like Amy and Fernando doing really interesting work around AI in settlement and in language. And thank you so much, both Amy and Fernando for for taking the time. I know we could have this conversation over hours and hours and hours, but for folks here, this is these are the great starting points for you. I recommend going through the entire series, because messages you're hearing here are kind of consistent throughout. There's some really interesting threads that you can pull out that it is possible you should maintain all of the the things you already know about, privacy, security, confidentiality. You should engage your staff and your clients in any of these processes, and you should iterate. You don't have to be disruptively innovative in your organization. Build off of what works for you, but also build off of proven technologies, for example. So thank you both very much. And Jessica back over to you.
Jessica Kwik 1:27:18
Well, thank you all. Amy Fernando for giving us a sense of, you know what, what's possible and what you've been able to achieve so far. It's really impressive. And I think the theme, even though you've had so many achievements, is to try to start small and be sort of incremental in that approach, and use a bite sized approach. And so thanks so much for sharing your approach as well as thinking about the limitations and where you're approaching things cautiously as well. And so just really appreciating Marco, how you always encourage us to share as we go and sharing what we're learning. So on that note, we do have a survey for feedback, for folks to respond to, and I think that helps us understand where you'd like to see I know the eresa series is a great jumping off point to learn more, but if there are other areas for learning that you'd like to see into the future, we'd love to hear your thoughts on that. So thank you so much again, everyone for joining. It's been such a great turnout, and we'll be sharing a follow up with the slides as well as the recording. So thank you again, and hope you have all have a great day bye.
Subscribe to get the latest posts sent to your email.
Please take this short survey to help improve the KM4S web site. The survey is anonymous. Thank you for your feedback! (click on the screen anywhere (or on the x in the top right corner) to remove this pop-up)