Skip to main content

Valuing Data Accuracy, User Feedback, and Humans in the Loop in AI Adoption

What does it take to effectively deploy artificial intelligence across multiple bank functions? Is phased, situational deployment preferable or, in fact, inevitable? How can financial institutions scale existing efforts when new technology constantly resets the board?

As generative AI (gen AI) adoption for personal and professional use grows, and interest in agentic 

AI builds, banking industry consultants and practitioners see some important themes emerging:

  • Incorporating user feedback is vital to every project. 
  • Improving data quality is a never-ending task. 
  • Keeping a human in the loop is essential, at least for now

In this article we explore with industry experts where organizations, departments and their clients are in their AI journeys today; which use cases have been most successful and which need adjustments; and what factors might move the needle for banks and an industry staking big investments on AI-driven transformation. 

Data as lifeblood and questions in the black box

For Graham Tasman, partner and national banking industry leader at Grant Thornton Advisors LLC, the curiosity, bold investment and careful experimentation around gen AI are typical of how financial services approaches new technology.

“Financial institutions certainly have the benefit and long history of making outsized tech‑enablement investments relative to other industries as they recognize information is the lifeblood of their business,” says Tasman. “The familiarity in analytical tools, algorithmic coding, extensive modeling and various forms of automation means that banks have a lot of proxies to help guide their AI program’s health and progress.”

Banks, for example, use complex modeling to assist with compliance, risk management and forecasting. Applying similar principles to AI is in some ways just a natural extension of this, Tasman says.

“Obviously, a major difference with AI in the generative and agentic realm is the ‘black box’ nature [of the large language models] and uncertain impact of those constructs, both in the institution’s operations and, externally, in a customer‑facing context,” he says.

Institutions that have successfully deployed AI first put in place a strong governance program for its development and use, Tasman says. Good governance establishes clear guidance on who can use AI, how solutions are approved and developed and what kind of ongoing monitoring is required.

On the ground, Tasman sees financial institutions using AI to write code and assist software development teams, hastening deployments. They are also using it in call center functions to improve consumer experience and better understand customer tendencies, with a goal of boosting cross‑selling opportunities.

Marketing teams are using AI to guide customers toward products while carefully monitoring how the AI prompts are working to avoid unintended outcomes, Tasman says. “For example, institutions would recalibrate a learning model recommending new credit if it results in a customer overcommitment when viewed retrospectively in a regulatory exam,” he says.

Tasman and others see AI capturing efficiencies, especially in documentation‑heavy tasks. This includes compliance and underwriting support roles and accelerating back‑office transaction‑matching and reconciliations.

In addition, fraud and security departments are using AI to monitor transactions and detect money laundering, Tasman says, because AI can assimilate large amounts of transaction data and detect complex behavior patterns that conventional, parameter‑based systems may otherwise miss.

Still, keeping humans in the loop is important. “As with any automation, AI‑led transaction monitoring could generate false positives. A bank would still need to validate each case through human verification,” he says.

A focus on quality and training

Cleveland‑based KeyBank has been using AI across many functions, according to Dean Kontul, EVP and chief information officer at the bank. In 2022, KeyBank launched the AI‑powered virtual assistant “MyKey” for its contact center. Since then, interaction volumes have more than doubled, while agent call volumes have declined, Kontul says. Customer satisfaction has also improved, with the contact center Net Promoter Score rising from +40 to +54.

“In our contact centers, we’re piloting AI‑assisted tools that provide agents with chat or call summaries when a case is transferred, improving context and reducing handle time,” Kontul says. “We are also piloting a use case to enhance knowledge retrieval for procedural documents and to help categorize and summarize complaints. All outputs remain subject to human review.”

But the bank has put AI to good use in other internal areas as well.

For back‑office support, there’s “Penny,” KeyBank’s internal voice assistant. Penny recalls and automates employee support tasks—such as password resets, RSA token management and incident history retrieval. The system has already gone through an upgrade; recent enhancements have improved voice clarity, navigation speed and call routing intelligence.

For risk management and compliance, KeyBank uses an AI‑enabled tool to monitor regulatory changes through summarization and customized alerts. Teams also use an AI‑powered digital worker to automate the initial review of sanctions screening alerts within its compliance operations.

To support its engineering staff, KeyBank deployed multiple AI‑powered coding assistants, and personnel throughout the bank have permission to engage with Microsoft Copilot, using role‑based selective access to balance value, security and cost, says Kontul. The bank is also preparing to release a Copilot Studio use case built on curated HR resources, allowing teammates to quickly find answers to common questions, such as PTO policy.

Kontul says that as AI features become more universal across providers, users like KeyBank can potentially save money. The bank is exploring using different providers for different applications.

Like most organizations, KeyBank has learned a lot through early adoption. To drive adoption, training and supportive change management are critical.

The bank continues to emphasize the purpose and benefits of these tools so teammates integrate them into daily routines, while also investing in and maturing governance tools and processes.

“Content quality also matters—‘Garbage in, garbage out’ is real,” Kontul says. “If source documents are inconsistent or poorly structured, AI outputs suffer. We’ve used early iterations to clean up content and improve data organization, which in turn improves results.”

Managing expectations around accuracy is also essential. Kontul underscores the importance of defining acceptable AI responses and guidance for when human judgment is required. “AI is at a strong starting point, delivering clear efficiency gains and improving user experience,” he says. “However, the return on investment has been gradual. To truly move the needle, we believe the next phase must focus on more robust, end‑to‑end solutions that go beyond productivity enhancements to deliver measurable cost savings and business impact.”

Building on experience

As early as 2018, Bank of America launched its AI‑driven virtual financial assistant, “Erica,” within its mobile app, says Tom Ellis, head of consumer, retail and preferred banking technology.

During testing, the team found a strong preference for texting rather than speaking, Ellis recalls. BofA still offers voice capability, but early lessons from Erica told the bank to focus primarily on text.

“In retrospect, it makes sense that someone probably doesn’t want to ask aloud, ‘What’s the balance on my checking account?’ or other key financial questions in a crowded room,” he says. “That is why garnering client feedback throughout the development process is so critical for us to continue to deliver innovative banking solutions at scale.”

Clients have now interacted with Erica 3 billion times, with 20 million clients actively using the assistant. Erica’s capabilities have expanded to support individual and corporate clients. In 2020, BofA launched Erica for Employees for mobile password reset, device activation and other tasks. A 2023 upgrade added access to payroll and tax form locations, with future updates planned to answer employee questions about BofA products and services.

For AI‑powered search and summarization, BofA offers commercial customers solutions such as Intelligent Receivables and CashPro Forecasting to predict future cash positions across accounts and institutions.

Let the business case lead

BofA call center employees now have generative AI tools to summarize customer call recordings, with the aim of increasing efficiency and accuracy. The bank also uses AI for content generation and supports more than 17,000 software developers with gen‑AI coding tools. More than 50 AI‑enabled fraud detection models are in place to identify suspicious activity more quickly and accurately.

“We already see value in AI implemented at scale for our clients and associates, and there is immense opportunity to further leverage AI for business value,” says Michelle Boston, BofA’s CIO and head of data management technology and enterprise architecture. “AI deployed responsibly can move the needle and act as a force multiplier.”

She has some cautionary advice: Don’t start with an AI strategy and then build a business strategy around the technology. When the business case leads, banks set themselves up for success.

“We begin every innovation journey by asking, ‘Does this solve a client or employee need?’ and, if so, determining if it can be delivered at scale,” she says. “Our responsible AI strategy focuses on human oversight, transparency and accountability for all outcomes.”

Katie Kuehner-Hebert is a contributor to ProSight.

Related Articles

Is banking ready to trust artificial intelligence (AI) to teach? Supplementing learning and development programs for financial institutions with trained…

Banks are finding it harder to treat hurricanes, floods, wildfires, and other extreme weather events as distant environmental problems. The…

The deposit picture is getting a little better, but nobody should mistake that for a boom. In ProSight’s latest “State…

Join Us in Strengthening and Advancing the Industry

We’re helping financial professionals build a stronger future and act with confidence.

Want to come along?

Connect with UsBecome a Member

Smiling man with gray hair and beard wearing a suit and glasses sits at a desk in a modern office with glass walls.