How to Track AI Model Value and ROI

The explosive growth of AI investments

The hype around AI is seemingly unstoppable. On the vendor side, over half of the companies on the Forbes “Next Billion Dollar Startups” list are AI-related. On the buyer side, and fueling the hunger for AI products and services, are organizations buying into said hype. Research by Cisco finds that nearly all (98%) of organizations currently “feel urgency around AI,” with exactly half dedicating 10-30% of IT budgets to AI. And, a new forecast by IDC says global spending on AI will more than double by 2028 with a compound annual growth rate of just under 30%. Bain & Company sees even greater growth, saying the total market for AI will grow by up to 55% annually “for at least the next three years.”

That’s a lot of money—IDC’s $632 billion or Bain’s $990 billion by 2028—on technologies that have yet to show equally lofty benefits.

However, organizations are more than willing to boost AI investments and continue rapid AI deployments as the fear of missing out (FOMO) on AI’s promised benefits is almost as risky as the prospect of competitors realizing AI-powered benefits first. 

As AI investments grow, there is proof of a fast-paced value multiplier as success drives more success: PwC found that top-performing companies are twice as likely to realize AI’s value than lower performers. That’s not just business value; it’s also the complete package of generative AI development, implementing responsible AI approaches, and adopting AI-specific operating models. And top-performing companies are investing more than one-third more into generative AI budgets, signaling the aggressive expectations and the need for middle- and low-performers to keep pace or risk falling even further behind.

Key takeaways

• AI investments are booming, driven by FOMO and the promise of huge results. Early adopters are seeing significant returns, and outsized AI success is driven by even more AI investments.

• Quantifying AI's ROI is challenging but crucial. AI is showing value across industries and use cases, but many organizations struggle to quantify the financial benefits of improvements like increased customer satisfaction and employee productivity.

• Accurate ROI calculations require considering the full scope of costs including hardware, software, personnel, and data, and to account for factors like uncertainty, changes over time, and the value of all AI projects in aggregate.

Where AI drives the most value today and tomorrow

Of course, AI is proving valuable across many use cases and nearly every industry. Chatbots, data analysis, content generation, and software development are some of the most popular use cases related to AI, but many more advanced applications are quickly taking shape.

Here are just a few examples of how organizations are putting AI to work today:

• Retailers use AI to personalize customer experiences and optimize inventory across locations.

• Automakers use AI to advance self-driving cars and automate operations.

• Utilities partner with energy consumers in using AI to improve energy efficiency and automatically adjust demand.

• Healthcare providers use AI to route insurance claims, extract data from complex medical documents, and automate billing and payment reminders. 

• Financial services firms use AI to automate low-risk loan approvals and predict the best products and services to offer potential clients.

However, while AI deployments continue to grow, quantifying the value of those investments has yet to catch up. Qualitative measures of faster cycles, happier customers, and freed-up workers are generally held up as the best available metrics. In fact, Gartner finds that half (49%) of those surveyed say they have “difficulty in estimating and demonstrating the value of AI projects.”

Overall, the EY AI Pulse Survey shows that organizations are realizing returns of 40-80% on various AI investments, such as productivity, cybersecurity, and innovation. According to the survey, the biggest ROIs come from AI deployments in customer satisfaction, employee productivity, and operational efficiencies. What’s even more encouraging is that those spending more on AI initiatives are realizing up to 55% higher returns on their investments.

Calculating the return on AI investments

Calculating the return on any investment requires knowing the resources spent and benefits realized. Let’s look at those separately.

In general, the investment in AI is relatively easy to calculate. Let’s consider the investment side of the ROI calculation first:

Calculating AI’s ROI: Investment types

• AI products and services invoiced by software vendors, consulting partners, and other organizations or consumed by internal resources that deliver and manage AI and related services. This can be as simple as adding a monthly per-user AI subscription charge to your CRM application or as complex as deploying an in-house AI development team to build, deploy, and maintain custom AI applications.

• Hardware, storage, and processing power to house and run AI investments such as cloud storage and processing from Amazon Web Services, AI development platforms like Google Vertex, and APIs and licenses for OpenAI’s GPT or other popular AI models.

• Training and human resources costs to attract and retain AI developers and related talent, deliver upskilling efforts to increase user adoption and ensure effective usage of AI tools, and employ data scientists and architects to ensure AI models are equipped with the best available data.

• Data required to train AI models or alterations and transformations of existing historical organizational data in preparation for AI model training. This may include data cleansing operations and scrubbing data to remove personally identifiable information, employee information, or other sensitive data.

Some organizations may have additional multipliers to account for investment risks, opportunity costs, uncertainty, and time. Each organization should adhere to accepted internal, industry, and regulatory requirements, if applicable. 

With AI, the returns today are not quite as quantifiable as these calculations require. Benefits like increased customer satisfaction, efficiency, data quality, and accuracy are worth paying for, but putting a value on them proves difficult. Or, at the very least, quantifying the value of these benefits is more of a judgment call than a calculatable figure.

When calculating the benefits of AI, it is important to begin with quantifiable metrics. Let’s explore examples below.

Calculating AI’s ROI: Metrics to use 

• Increased productivity for processes requiring humans to work across multiple applications, transfer data, and look up related information. Using AI to automatically enter or copy information across systems can save human workers valuable time, especially when processing dozens or hundreds of items per day.

• Process speed and time savings for workflows where AI automates repetitive tasks and reduces the time required by humans to complete those tasks. A good example is triaging incoming customer support requests. Using AI to answer simple questions, gather customer and order information, and route requests to the proper human agent can save a few minutes per interaction.

• Improved customer satisfaction can be directly measured and correlated with shifts in revenue, profitability, upsells, total order value, and more. Using AI to answer order questions, guide customers to complimentary products, and streamline returns can lead to higher satisfaction, which, in turn, can be connected to increases in revenue, profitability, and more.

As hard metrics on AI’s ROI are collected, don’t lose sight of the soft, qualitative benefits of AI. While things like employee satisfaction, upskilling, and organizational agility may be unquantifiable, they are valuable benefits nonetheless. 

Common mistakes in calculating AI’s ROI

Justifiable, documented, and trusted metrics form the foundation of a good ROI calculation. However, organizations can make some common mistakes when calculating ROI for AI initiatives. 

#1 - Lack of benefits realization certainty

When using too simplistic or absolute outcomes, the calculated benefits can far exceed the actual benefits unless uncertainty is taken into account. 

For example, AI may be deployed to provide upsell recommendations to customers, with a trial showing it increases the average purchase amount for customers by $25 per order. The mistake would be in extrapolating that to every customer order and expecting a comprehensive increase in order values. That would miss potential errors in AI-driven recommendations that might occur more often than expected or for certain product categories. Or, it wouldn’t account for errant recommendations that reduce customer satisfaction or turn into increased returns or calls to customer support. 

A best practice is to account for uncertainty by subtracting the cost of errors and related fallout from the expected benefits. This safety margin can also increase confidence in the ultimate ROI calculation. Another best practice is to evaluate AI models for quality and accuracy to help determine the potential impact of AI errors.

#2 - Not accounting for ROI changes from AI over time

It’s common for organizations to be excited about potential AI benefits immediately upon deployment and after the initial benefits are realized. But, with AI innovations coming at a rapid pace, the benefits can shift dramatically over time, either up or down, depending on AI improvements, implementation complexity and downtimes, and changing data. 

For example, models trained on stale data may not account for shifting patterns, demographics, or other data, market, economic, and other attributes. To adjust for such changes, AI models may need to be retrained or tuned, which incurs extra costs. Not adjusting for those changes can lead to benefits decay. 

The best practice is to monitor and prepare for adjustments and ongoing maintenance of AI applications, data, and models to keep AI investments performing at expected levels.

#3 - Tracking AI as siloed, independent projects

As detailed above, AI is garnering immense attention and investment. It’s far too large for most organizations, which leads some to focus on calculating the ROI of independent AI investments. That fails to aggregate the benefits that, hopefully, combine the overachievers to far outweigh the misfires. 

For example, a generative AI project to draft and send personalized emails to overdue accounts might save hours per day for each collections officer. On the other hand, an AI application to automatically determine the next steps for sales representatives might inadvertently recommend incorrect cross-sell opportunities and fail to increase sales as expected.

The best practice is to take a broad, comprehensive view of the entire organization’s portfolio of AI investments to calculate the overall return on the organization’s entire AI investment. 

Tracking returns on AI investments

Organizations of all sizes, industries, and maturity levels are deploying AI at a rapid pace and with significant investments. Good business practices require leaders to measure and evaluate the returns those investments are driving.

When calculating AI ROIs, it’s important to understand and define AI use cases, determine clear and measurable metrics, account for the breadth of investments across people, processes, and technologies, and confidently quantify the benefits delivered by AI investments. 

With a strategic approach and the best practices listed above, organizations can understand, manage, and improve the value of every AI investment. 

Achieve AI that delivers real ROI with Alation

As organizations pour more capital into new AI-powered tools that demand better access to better data, they are challenged to improve how those AI investments deliver real value. For many, improving the ROI of AI begins with data.

Alation simplifies data discovery, enabling teams to deploy AI innovations based on trusted data confidently. The result is faster, more accurate AI outcomes for faster realization of benefits and a larger ROI.

Schedule an Alation demo to learn how data quality, transparency, and compliance lead to more ethical, compliant, and beneficial AI.