We’re excited to bring Transform 2022 back to life on July 19th and virtually July 20-28. Join AI and data leaders for sensible conversations and exciting networking opportunities. Register today!
More than ever, organizations are putting their trust – and investment – in the potential of Artificial Intelligence (AI) and Machine Learning (ML).
According to the 2022 IBM Global AI Adoption Index, 35% of companies today use AI in their business, while an additional 42% say they are exploring AI. Meanwhile, a McKinsey survey found that 56% of people reported adopting AI in at least one work in 2021, up from 50% in 2020.
But can investing in AI give a true ROI that directly affects a company’s bottom line?
According to a recent REVelate survey by Domino Data Lab, which surveyed attendees at the Rev3 conference in New York City in May, many feel that way. About half, in fact, expect double-digit growth as a result of data science. And 4 out of 5 (79%) said that data science, ML and AI are important for the overall future growth of their company, 36% of people consider it the most important factor.
Implementing AI is, of course, no easy task. Other survey data show the other side of the coin of trust. For example, data from a recent survey by AI engineering firm CognitiveScale found that, although executives know that data quality and deployment are critical success factors for successful app development to drive digital transformation, more than 76% of people know how to reach their goals. Not at all. 12-18 month window. In addition, 32% of execs say that the AI system took longer to produce than expected.
AI should be responsible
ROI from AI is possible, but it must be accurately described and expressed in accordance with the business goal, Cognitive Scale CEO Bob Piciano told VentureBeat.
“If the business’s goal is to achieve more long-range predictions and predictive accuracy with historical data, then AI can be implemented in the same place,” he said. “But AI should be responsible for running the effectiveness of the business – suffice it to say that the ML model was 98% accurate.”
Instead, there may be ROI, for example, to improve the efficiency of the call center, AI-powered capabilities ensuring that the average call handling time is reduced.
“It’s the kind of ROI they talk about in the C-suite,” he explained. “They don’t talk about whether the model is accurate or strong or drifting.”
Co-founder and co-founder of Cognitive Scale She Sabhikhi added that he was not surprised by the fact that 76% of people have difficulty measuring their AI efforts. “That’s exactly what we hear from our enterprise clients,” he said. One problem is the friction between the data science teams and the rest of the organization, he explained, not knowing what to do with the models they develop.
“Those models may have the best algorithms and precision recall possible, but sit on the shelf because they are literally thrown to the development team who then have to try to assemble the application together,” he said.
At this point, however, organizations should be held accountable for their investments in AI because AI is no longer a series of science experiments, Piciano noted. “We call it going from lab to life,” he said. “I was at the Chief Data Analytics Officer’s conference and everyone said, ‘How do I scale?’ How can I industrialize AI? ”
Is ROI the right metric for AI?
However, not everyone agrees that ROI is the best way to measure whether AI has value in an organization. According to Nicola Morini Bianzino, EY’s global chief technology officer, thinking of artificial intelligence and enterprise in terms of “use cases” that are then measured by ROI is the wrong way to go about AI.
“For me, AI is a set of technologies that will be deployed everywhere throughout the enterprise – the use case will not be isolated with the associated ROI analysis,” he said.
Instead, he explained, organizations must use AI everywhere. “It’s almost like a cloud, where two or three years ago I had a lot of conversations with customers who asked, ‘What is ROI? What’s the business case for me to go to the cloud?’ Now, after the epidemic, that conversation doesn’t happen anymore. Everyone just says, ‘I have to do that.’
Also, Bianzino pointed out that the discussion of AI and ROI depends on what you mean by “using AI”.
“Let’s say you’re trying to apply some self-driving capabilities – that is, computer vision as a branch of AI,” he said. “Is that a business case? No, because you can’t implement self-driving without AI.” The same is true for a company like EY, which ingests large amounts of data and advises customers – which cannot be done without AI. “It’s something you can’t separate from the process – it’s built into it,” he said. Said.
Furthermore, AI, by definition, is not productive or efficient on the first day. It takes time to get the data, to train the models, to develop the model and to enhance the model. “It’s not like one day you would say, I’m done with AI and 100% value is there – no, this is an ongoing capability that gets better over time,” he said. “There is really no end to what can be produced in terms of value.”
In a way, Bianzino said, AI is becoming part of the cost of doing business. “If you’re in a business that involves data analysis, you may not have AI capabilities,” he explained. “Can you differentiate the business case of these models? It’s very difficult and I don’t think it’s necessary. To me, it’s almost like the cost of infrastructure to run your business. ”
It is difficult to measure the ROI of AI
At the end of the day, what organizations want is a measure of the business impact of ROI – how much it contributes to the bottom line, says Kegel Carlson, head of data science strategy and evangelism at enterprise MLops provider Domino Data Lab. But one problem is that this can be totally disconnected from how much work has been done in developing the model.
“So if you build a model that improves click-through conversions in percentage, you’ve added a few million dollars to the organization’s bottom line,” he said. “But you could also create a well-preserved maintenance model that helped warn a piece of machinery that it needed maintenance before it happened.” In that case, the impact of the dollar-value on the organization could be completely different, “however one of them could be a very difficult problem,” he added.
Overall, organizations need a “balanced scorecard” where they are tracking AI production. “Because if you don’t find anything in the production, it’s probably a sign that you have a problem,” he said. “On the other hand, if you’re getting too much into the product, it could also be a sign that there’s a problem.”
For example, the more data science teams deploy, the more models they have on the hook for managing and maintaining, he explained. “That’s why you deployed so many models in the last year, so you can’t really afford these other high-value models coming your way,” he explained.
But another problem in measuring AI’s ROI is that for many data science projects, the result is not a model that goes into production. “If you want to do a quantitative win-loss analysis of deals in the last year, you want to do a rigorous statistical check,” he said. “But there’s no model that goes into production, you’re using AI for the insights you get along the way.”
It is important to keep an eye on the activities of data science
However, organizations cannot measure the role of AI if data science activities are not tracked. “One of the problems right now is that so little information science activity is actually being collected and analyzed,” Carlsen said. “If you ask people, they say they don’t really know how the model is performing, or how many projects they have, or how much code data your data scientists have committed in the last week.”
One reason for this is that highly disconnected tools are required for data scientists to use. “This is one of the reasons why Git has become more popular as a repository, a source of truth for your data scientist in an organization,” he explained. MLops tools such as Domino Data Lab’s offer platform that supports these various tools. “How organizations can build this more centralized platform … is important,” he said.
AI results are at the top of the mind
Wallaroo CEO and founder Vid Jain spent nearly a decade in the high-frequency trading business at Merrill Lynch, where he said his role was to scale machine learning and do so with a positive ROI.
The challenge was not really to develop data science, clean up data, or create trade repositories, now called data leaks. So far, the biggest challenge has been to take the models, make them work and deliver them with commercial value, he said.
“Delivery of ROI is very difficult – 90% of these AI initiatives do not generate their ROI, or they do not generate enough ROI to make the investment worthwhile,” he said. “But this is a top priority for everyone. And the answer is not one thing. ”
One basic point is that many people believe that managing machine learning is not much different than running a standard type of application, he explained that there is a big difference, because AI is not static.
“It’s almost like handling a farm, because the data is alive, the data changes and you’re not done,” he said. “It’s not like you create a recommendation algorithm and then the behavior of how people buy becomes stable over time. People change how they buy. All of a sudden, your competitor gets a promotion. They stop buying from you. They go to the competitor. You just have to be more discriminating with the help you render toward other people. ”
Ultimately, every organization needs to decide how they will align their culture with the ultimate goal around AI implementation. “Then you really have to empower people to drive this change, and then make people who are important to your current business lines feel like they’re going to get some value out of AI,” he said.
Most companies are still at the beginning of that journey, he added. “I don’t think most companies are there yet, but I have certainly seen in the last six to nine months a shift towards becoming serious about business results and business value.”
AI’s ROI remains elusive
But the question of how to measure the ROI of AI remains elusive for many organizations. “There are some basics for some, like they can’t even produce their model, or they can but they are becoming blind, or they are successful but now they want to scale,” Jain said. “But as far as ROI is concerned, there is often no P&L associated with machine learning.”
Often, AI initiatives are part of the Center of Excellence and are captured by ROI business units, he explained, while in other cases it is difficult to measure.
“The problem is, is AI part of the business? Or is it a utility? If you are a digital native, AI can be a part of the fuel on which the business runs, “he said. They have to wrestle together. ”