Infrastructure Management: Cost Savings From Improved GPU Utilization

Evidence and data. Prior to the investment in Red Hat AI, the interviewees’ organizations lacked standard approaches to AI development and infrastructure platforms to manage the necessary resources. Interviewees said their organizations sought to scale the complexity and volumes of AI models in production environments and required more GPUs each year to do so. However, their legacy infrastructures provided limited ability to monitor and manage GPU usage, resulting in low GPU utilization rates despite additional spending. With Red Hat AI, their organizations gained better views of where and how GPUs were being used to make smarter allocation decisions. This visibility improved GPU utilization and enabled their organizations to develop and deploy more AI models with existing resources before purchasing additional GPUs.  

• In addition to cost savings, more transparency into GPU utilization meant that interviewees’ organizations were better able to meet business needs as they changed over time. The head of DevOps at a financial services organization stated: “[Flexible infrastructure resources are] not only a cost saver but also a way to do more with the power we have. This gives us flexibility to change resources depending on our priorities. We don’t have unlimited power; we must use limited on-prem resources. Infrastructure management therefore becomes much more meaningful to meet the business objectives of the financial institution and deliver value.”

• This same interviewee also described how Red Hat AI provided more transparency to optimize existing GPUs. They said: “Because OpenShift uses Kubernetes natively, it brings us the ability to manage our resources easily. For example, if you are working on a single model for one month and you need data access for the month, I can change the resources and I can assign the necessary internal resources and GPU to complete the development.”

• The CIO at a government organization stated that prior to Red Hat AI, their average utilization rate per GPU was around 30% to 40%. With Red Hat AI, that utilization increased to between 70% and 80%. Since each GPU [server] cost upward of $400,000, the increase resulted in significant cost savings for their organization.

• The same interviewee noted that GPU utilization did not come at the price of model performance and pointed to model latency improvements of 10% to 20% in parallel with making their GPU utilization more efficient. They stated: “With Red Hat AI, you can visualize latency trends and see how models are responding to maintain target ranges. And I can reuse all my existing OpenShift governance and monitoring stacks, so depending on what I want to bring in [in terms of data sources and performance targets], I end up with lower latency, no egress costs.”

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• The composite organization calculates that it needs four GPU AI server nodes (two GPU per node to total eight GPU) in the first year of the investment to meet business goals. They also anticipate that the volume of GPUs required will double in each year of the investment as it builds more complex models and deploys them to production.

• Prior to Red Hat AI, the organization has a GPU utilization rate of 30%, which means it must purchase more GPU servers to meet the scale and performance expectations set by the business.

• Each GPU server node costs $300,000.

• With Red Hat AI, the organization improves GPU utilization to 70% in Year 1 and up to 80% by Year 3.

• As a result, the organization purchases fewer GPU servers each year to meet business and AI development goals.

Risks. The third section details the potential risks that can impact the benefit. These can be both qualitative and quantitative. State the risks in bullet form. Refer to Best Practices.

• The business goals and associated AI strategy set at the organization will determine the volume of GPU servers expected annually.

• The AI development maturity and prior environment will determine the starting utilization rate for GPU servers.

• GPU costs may vary depending on market fluctuations and providers.

Results. To account for these risks, Forrester adjusted this benefit by 10%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $3.0 million.

Data Scientist Time Savings

Evidence and data. With Red Hat AI, users at the interviewees’ organizations benefited from self-service access to the latest models and resources. Additionally, the AI development platform provided access to predictive and generative AI-specific capabilities, including frameworks. Therefore, platform users accelerated time to begin development and benefited from building better, more accurate models. As a result, users at interviewees’ organizations spent less time on model training and rework. This time savings compounded year over year as the quality of the models continued to improve. The interviewees’ organizations then redistributed the time savings to building more complex AI models.

• The head of DevOps at a financial services organization indicated that data scientists were responsible for more than just enabling model creation. Rather than spending time creating the workbench and storing the necessary data, they wanted to focus on more interesting modeling and development work. The DevOps leader said: “In 10 minutes, if you want to develop a new model, your repository is ready, your workbench is ready, and your infrastructure resources are available. Your working environment is ready and totally isolated. The data scientists reallocate that time to develop more accurate AI and ML models. They can start more projects, or they can take more time for themselves to upskill or hone existing skills.”

• The CIO at a government organization estimated that their users saved 40% to 50% of their time with Red Hat AI by having access to a familiar set of tools and environments that required less of a learning curve. They stated: “Efficiencies for data scientists come from having full visibility and control and that they can leverage all existing Kubernetes and OpenShift skills. So with other [AI tools], the data scientists would have to spend time learning different cloud UIs, software development kits, etc.”

• The same interviewee indicated that their data scientists redistributed their time savings to building more accurate models that required less human correction or retraining: “[With the time savings], data scientists are able to target items more effectively and enable better timing of actions with a higher precision so that there are fewer false positives. I also automate my retraining to include new corrected data and to tighten guidance.”

• The CIO at a government organization also indicated that having more accurate AI models compounded the time savings. They stated: “Even if we have a 10% to 20% accuracy gain — and we saw a 90% accuracy improvement in some areas — it means less rework on the IT side. This time savings compounds as we build more trust with the models, which also increases adoption of the models we build.”

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• The organization has 80 data scientists who are Red Hat AI users. Users could include AI engineer resources at other organizations.

• Data scientists spent 15% of their time on average on model training and rework prior to Red Hat AI.

• With Red Hat AI, data scientists save 40% of their time in Year 1 and up to 60% of their time by Year 3 on those tasks due to improved model accuracy.

• The average fully burdened annual salary for a data scientist is $200,000.

Risks. Data scientist time savings may vary depending on the following:

• The volume of users, the resource types expected to use Red Hat AI, and their fully burdened salary ranges.

• The AI development maturity and environment prior to Red Hat AI will impact the time spent on model training and rework.

Results. To account for these risks, Forrester adjusted this benefit downward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $2.5 million.

MLOps Efficiencies

Evidence and data. For the interviewees’ organizations, MLOps teams were most often responsible for environment provisioning and preparation to support AI model development. With Red Hat AI, MLOps teams easily reproduced environments to improve development scale and accelerate timelines. The time savings enabled MLOps resources to support more users with their AI projects more quickly, which in turn, accelerated time to business value for their organizations.

• The CIO at a government organization agreed that due to MLOps efficiencies, their organization reduced its preparation and provisioning stage by 25% to 30%. They noted: “I would say on the provisioning side that there is a lot of time saved there, the GPU driver setup, model serving setup, compliance reviews, debugging, monitoring setup. Sometimes you might spend a couple of hours or even days per model on the model serving setup. On average, there is a 25% to 30% reduction in different projects.”

• The CEO at a telecommunications organization said that time savings across environment provisioning tasks for AI model development ultimately helped accelerate development timelines from up to five days per project to 10 minutes in some cases. The head of DevOps at a financial services organization also agreed that reducing the time spent on operational tasks required to begin AI model development reduced overall timelines from one week to 10 minutes.

• The CIO at a government organization saw that time savings for MLOps teams also translated into more control that helped reduce compliance issues. They said: “I have the full pipeline control from an MLOps stance. I have more automation and portability, and I don’t have as much lock-in. So I can do MLflow, GitOps, in an integrated continuous integration/continuous delivery.”

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• The composite organization supports 100 AI projects with Red Hat AI in Year 1 and doubles the volume of projects year over year, adding 100 more projects in Year 2 and 200 more projects in Year 3.

• Prior to Red Hat AI, the provisioning and preparation phase of each project took three days on average. The composite assigns a single MLOps resource to complete these tasks per project.

• The organization sees a 75% reduction in time spent on pre-AI model development work, which impacts the internal MLOps teams responsible for completing provisioning tasks and accelerates overall development timelines.

• MLOps resources have fully burdened annual salaries of $175,000, which is divided by 266 working days to achieve an MLOps resource cost per day.

Risks. MLOps efficiencies will vary depending on the following:

• AI model development maturity and scale at the organization in terms of the volume of development projects completed each year.

• The resource types and counts involved in provisioning and preparation tasks for AI model development as well as their fully burdened annual salaries.

• The average number of days spent on predevelopment operations for AI model development prior to the Red Hat AI investment.

Results. To account for these risks, Forrester adjusted this benefit downward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $407,000.

Business Value: Top-Line Impact

Evidence and data. With Red Hat AI, the interviewees’ organizations leveraged internal efficiencies for users and their supporting teams to accelerate AI development timelines. Additionally, with more transparency and control over the infrastructure, they built more complex AI models, such as those that use genAI, with the necessary power to ensure performance standards. With data and cybersecurity demands met, the interviewees’ organizations deployed AI on-premises at scale and more models to production environments. Altogether, the organizations benefited from AI models deployed for external and internal use cases that drove business value. 

• The head of DevOps at a financial services organization built a loan approval model that enabled employees to make fast decisions in an environment with rapidly changing requirements. They indicated that this model would not have been possible to build with other tools, as they lacked the level of control enabled with Red Hat AI.

• The AI innovation manager at a manufacturing organization built multiple AI models for customer-facing use cases that together generated more than $500,000 in additional revenue for their organization. In addition to external use cases, the interviewee discussed an internal use case that included a natural language processing model that performed root cause analysis to reduce nonconformities in the manufacturing lifecycle, saving their organization more than $1 million annually on maintenance and replacement part costs.

• The CIO at a government organization set aggressive performance targets for the models built with Red Hat AI, such as a 95% source quality score, a less than 3% hallucination rate, and a time to answer of 3 seconds or less for short queries. The same interviewee discussed the security component involved in building models with sensitive data. They said, “People are wanting to combine a variety of different data sources and some of it was a challenge from a security perspective.” The interviewee noted that with Red Hat AI, they were able to utilize more data to better tune AI models and achieve better outcomes: “Today, I’m able to give a better fine gradient. So I’m giving better warnings, making better operational decisions, and seeing better outcomes, including more simplified processes.”

• The CEO at a telecommunications organization built an externally facing model for their customer service function that reduced incident response times by 50%. They theorized that this improvement would impact the top line of the organization through better customer experience scores and related KPIs.

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• The composite organization has $100 million annual revenue.

• Given the AI models in production, the organization improves the top line by 0.5% in Year 1, up to 2% in Year 3.

• With a 12% operating margin, the top-line improvement leaves the organization with an additional $60,000 to $240,000 in profit each year.

Risks. Business value from top-line impact may vary depending on the following:

• The size and industry that the organization operates within.

• The volume of AI model projects geared toward revenue-generating use cases.

Results. To account for these risks, Forrester adjusted this benefit downward by 5%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $317,000.

Unquantified Benefits

Interviewees mentioned the following additional benefits that their organizations experienced but were not able to quantify:

• Additional infrastructure team time savings. Interviewees whose organizations were not previously OpenShift users mentioned additional IT ops team time savings from more transparency and standardization after consolidating to Red Hat AI. The head of DevOps at a financial services organization saw less need for ancillary reports to monitor activities across the infrastructure. They stated, “[With Red Hat AI], I can see who is managing models, which models use which libraries, model versions, etc. through a simple query.” The same interviewee described how the additional insights were put into action, stating: “I know which model was built when and in what version, which code and where it came from. I can report against it, identify who can change it, and log changes that have already happened and who approved of those changes.” Finally, this interviewee tied together infrastructure time savings with the ability to scale AI development, stating: “Being on Red Hat AI means we have access to tested and proven versions of approved frameworks and models, which makes it easier to scale up and scale down and manage resources, which saves time on the infrastructure side.” The CIO at a government organization agreed that infrastructure time and cost savings were key factors in enabling AI development at scale, and stated, “[If we had used other AI development tools for our AI projects], we could have theoretically seen 50% to 60% more work to get the tool approved, to get that application built, to get it even into development, let alone production environments.”

• Reduced risk of shadow IT/AI. Streamlining AI development and deployment tools to a single platform meant that interviewees’ organizations benefited from fewer delayed tool approvals and saw faster tool and environment provisioning. By having the approved tools and infrastructure in place, data scientists and developers accelerated time to start projects, which mitigated the risk of those resources obtaining their own, nonapproved tools to meet business objectives.

• Heightened model security and facilitated governance. The head of DevOps at a financial services organization contextualized the importance of governance for AI development at large enterprises, and stated: “If you’re an enterprise and are working with more than 2,000 developers and more than 100 data scientists, you have to control the AI and report against it. You have to give that level of proof of accountability to the bank and the financial institution you are serving.” The same interviewee noted that although internal governance drove their current governance, external regulations were on the horizon: “External regulations [regarding AI/ML models] might come into play in the future and the same reports and transparency that we benefit from today for internal purposes will be critical in meeting those requirements. The approach we’ve taken with Red Hat AI is our safeguard for the future.” The AI innovation manager at a manufacturing organization pointed to emerging global regulations such as the EU AI Act as a sign that more AI regulation was imminent and agreed that Red Hat AI would enable their organization to meet the demands of such external regulations in the future.

• Better employee experiences for data scientists and the business stakeholders they serve. Internally, there were less politics regarding resource consumption for AI development efforts. The head of DevOps at a financial services organization stated, “There is transparency into resource assignments — [we] can see who is using what and for how long, and everyone can see it, which means less politics. Before, there was no governance, so whoever got to work earliest would have access to the resources.” Data scientists were then able to better meet business stakeholder expectations and accelerate time to model deployment.

Flexibility

The value of flexibility is unique to each customer. There are multiple scenarios in which a customer might implement Red Hat AI and later realize additional uses and business opportunities, including:

• Encourage more innovation. Interviewees’ organizations took advantage of Red Hat AI improvements as part of Red Hat’s product roadmap, ongoing updates to the platform, and better control of existing capabilities to arm users with more tools and more data to encourage AI and ML projects. Externally, the organizations scaled AI projects to cover more use cases. They also added more complex AI (including genAI models) and GPUs to fuel more end-user focused model development. The head of DevOps at a financial services organization said: “We are focusing on scaling up to include other business cases, especially genAI use cases, at the bank. We want to enhance the business cases.” The CEO at a telecommunications organization agreed, stating, “Our organization plans to scale the AI solution to other critical areas, such as satellite management, and eventually turn the implementation into a commercial product for other telecom service providers.” Additionally, many organizations were moving toward enabling data scientists with access to GPUs, which would enable more flexibility to further accelerate innovation. The AI innovation manager at a manufacturing organization stated: “We hope to give developers direct access to GPUs. Right now, the AI team is in between that process to determine the business value and allocate the GPUs.”

• Continued AI development and deployment flexibility. Many of the interviewees’ organizations sought Red Hat AI as their platform approach to AI development specifically to meet on-premises deployment demands. However, the interviewees recognized the need for a flexible future of deployment methods, including planning for hybrid AI model deployments. As such, they were especially keen to avoid vendor lock-in and increase the collaboration and partnership with their selected vendors. The AI innovation manager at a manufacturing organization stated: “I’m looking forward to what Red Hat AI can bring for hybrid cloud AI deployments, with edge cases especially. We have demonstrated with proof of concept (POC) this year that the platform is capable of this, we are just waiting for the business to be ready.”

Flexibility would also be quantified when evaluated as part of a specific project (described in more detail in Total Economic Impact Approach).