The Total Economic Impact™ Of CloudBees
Cost Savings And Business Benefits Enabled By CloudBees
A Forrester Total Economic Impact™ Study Commissioned By CloudBees, March, 2024
Efficiency metrics are an essential focus for DevOps leaders, and rightly so: Positive performance in areas such as throughput and time to release often leads to business value.1 The software industry is moving more and more toward the use of flow metrics, as opposed to snapshot-style agile or DevOps research and assessment (DORA) metrics, to offer the most visibility into movement of projects through a queue. With this in mind, software development pipeline tools are poised to become important growth and revenue drivers.
The CloudBees Platform, comprising continuous integration (CI) and continuous delivery (CD), enables organizations to accelerate software delivery. CloudBees offers a stable software development and deployment pipeline with built-in automation and on-demand service and support.
CloudBees commissioned Forrester Consulting to conduct a Total Economic Impact™ (TEI) study and examine the potential return on investment (ROI) enterprises may realize by deploying CloudBees.2 The purpose of this study is to provide readers with a framework to evaluate the potential financial impact of CloudBees on their organizations.
To better understand the benefits, costs, and risks associated with this investment, Forrester interviewed six representatives at organizations using CloudBees. For the purposes of this study, Forrester aggregated the interviewees’ experiences and combined the results into a single composite organization, a large financial services organization with 3,000 developers and revenue of $25 billion per year.
Return on investment (ROI)
426%
Net present value (NPV)
$30.9M
Interviewees said that prior to using CloudBees, their organizations struggled with the lack of standardization, stability, and scalability inherent in their software delivery solutions. Before they deployed CloudBees, interviewees described examples of issues at their organizations:
- Outages happened frequently and sidelined large portions of their workforce for hours at a time.
- Routine processes, such as requesting predeployment approvals or creating credentials for new users, were time-consuming manual tasks.
- Each eight- to 10-person developer team had its own processes and its own customized tools.
Multiplying this fractious scenario across organizations that have dozens or hundreds of distributed teams, plus an enterprise support team of many engineers, results in an organization unable to predictably grow and scale.
After the investment in CloudBees, the interviewees said their outages were reduced to near zero, and with many common processes automated, their organizational efficiency and productivity skyrocketed. These improvements also supported growth, with total deployments of software applications notably increasing in the years following the CloudBees migration by as much as 60% over a five-year period.
Key Findings
Quantified benefits. Three-year, risk-adjusted present value (PV) quantified benefits for the composite organization include:
- Savings from reduced outages. Using CloudBees for continuous integration improves the stability of the software development pipeline, drastically reducing the frequency, duration, and impact of system outages. By Year 2, total lost developer hours are reduced by 99%, saving the organization more than $4.5 million. Over three years, this is worth more than $10.6 million.
- Savings from deployment automation. Using CloudBees for continuous delivery automates predeployment preparation and approval processes at the composite organization, which used to involve several people for several hours for each of the 20,000-plus deployments per year. This efficiency reduces the organization’s annual predeployment costs by 70% in Year 1 – higher in future years when greater efficiency enables the organization to ramp up its deployment productivity. Over three years, this brings more than $27.5 million in value to the composite organization.
Unquantified benefits. Benefits that provide value for the composite organization but are not quantified for this study include:
- Improved product quality. The implementation of stable, repeatable processes reduces the opportunity for human error, and automated predeployment testing makes it less likely that problematic code will be pushed into the pipeline. With improved product quality comes a better end-user customer experience.
- Increased flexibility and innovation. Standard workflows allow teams to communicate and share best practices. With all employees tapping into the same set of tools and processes, individual developers are no longer tethered to a single team, which enhances collaboration and innovation.
- Improved user experience. All employees benefit from the flexibility to move among product teams, which facilitates professional development. Specific to the enterprise team, process automation reduces organizations’ dependence on their pipeline support staff to put in overtime hours, work overnight deployment shifts, and receive after-hours emergency calls. As a result, engineers and programmers enjoy a better work-life balance, helping organizations attract and retain talent.
- Improved security. Standardization of processes and user credentials, as well as the ability to run security scans and software updates efficiently and unobtrusively, all help an organization improve its security posture. This is particularly significant for organizations in regulated business sectors, but any industry that stores customer information can benefit.
Costs. Three-year, risk-adjusted PV costs for the composite organization include:
- Implementation costs totaling $1.4 million over three years. A combination of internal staff time, third-party consultant engagements, and use of CloudBees’ professional services team supports a seven- to 12-month migration project.
- Ongoing costs totaling $5.8 million over three years. CloudBees licensing costs average $550 per user for the composite organization’s 3,500 users. In addition, the organization continues to engage with professional-services consultants to complete its migration of legacy applications from older pipelines. Finally, it also invests in the development of internal communication and documentation to point developers to available support and troubleshooting resources and support a move toward a more self-service culture.
The representative interviews and financial analysis found that a composite organization experiences benefits of $38.2 million over three years versus costs of $7.3 million, adding up to a net present value (NPV) of $30.9 million and an ROI of 426%.
Key Statistics
-
Return on investment (ROI)
426%
-
Benefits PV
$38.2M
-
Net present value (NPV)
$30.9M
Benefits (Three-Year)
Savings from reduced outages Savings from deployment automationTEI Framework And Methodology
From the information provided in the interviews, Forrester constructed a Total Economic Impact™ framework for those organizations considering an investment CloudBees.
The objective of the framework is to identify the cost, benefit, flexibility, and risk factors that affect the investment decision. Forrester took a multistep approach to evaluate the impact that CloudBees can have on an organization.
-
Due Diligence
Interviewed CloudBees stakeholders and Forrester analysts to gather data relative to CloudBees.
-
Interviews
Interviewed 6 representatives at organizations using CloudBees to obtain data about costs, benefits, and risks.
-
Composite Organization
Designed a composite organization based on characteristics of the interviewees’ organizations.
-
Financial Model Framework
Constructed a financial model representative of the interviews using the TEI methodology and risk-adjusted the financial model based on issues and concerns of the interviewees.
-
Case Study
Employed four fundamental elements of TEI in modeling the investment impact: benefits, costs, flexibility, and risks. Given the increasing sophistication of ROI analyses related to IT investments, Forrester’s TEI methodology provides a complete picture of the total economic impact of purchase decisions. Please see Appendix A for additional information on the TEI methodology.
Disclosures
Readers should be aware of the following:
This study is commissioned by CloudBees and delivered by Forrester Consulting. It is not meant to be used as a competitive analysis.
Forrester makes no assumptions as to the potential ROI that other organizations will receive. Forrester strongly advises that readers use their own estimates within the framework provided in the study to determine the appropriateness of an investment in CloudBees.
CloudBees reviewed and provided feedback to Forrester, but Forrester maintains editorial control over the study and its findings and does not accept changes to the study that contradict Forrester’s findings or obscure the meaning of the study.
CloudBees provided the customer names for the interviews but did not participate in the interviews.
Drivers leading to the CloudBees investment
Interviews
| Role | Industry | Revenue | CloudBees Users | |
|---|---|---|---|---|
| Manager of software engineering | Entertainment | $1.5 billion | 3,000 | |
| Head of DevOps, cloud, and SRE | Financial services | $24 billion | 3,500 | |
| Technical product manager | Pharmaceuticals | $59.3 billion | 3,500 | |
| Senior manager, engineering enablement | Entertainment | $26.9 billion | 1,000 | |
| Senior manager, enterprise CI/CD | Transportation and business services | $93.5 billion | 5,500 | |
| Director of DevOps | Hospitality | $3.9 billion | 1,000 | |
Key Challenges
In their prior state, interviewees’ organizations ran their DevOps pipelines on open-source solutions. The lack of consistency and formal support left enterprise service teams struggling to standardize processes and troubleshoot problems.
Specifically, interviewees noted these common challenges:
-
Implementations were large, shared pipeline
instances, leading to poor system performance and user
experience. Several interviewees’ organizations
reported that they had many teams — a few dozen to a few hundred — using the
same pipeline instance. This situation arose from a combination of factors,
including rapid organizational growth and a desire to avoid the manual and
often duplicative work inherent in new-team and new-user setup. The result,
however, was that these large instances became overloaded, slow, and
unstable.
The senior manager, enterprise CI/CD for the transportation and business services organization, said: “Before, people had enough freedom to write pipelines and break them. With the shared instances, any developer team could take the pipeline down for everyone. The blast radius was large.”
The manager of software engineering for an entertainment company shared: “Our prior open-source solution was difficult to maintain. We had shared instances that were so large that people would step on each other’s toes. We had hundreds of plugins, most of them community-released, and they weren’t always compatible. When that would crash the instance, it would affect the whole company. Plus, the performance wasn’t great because we had so many jobs running.”
- Their organizations experienced frequent, impactful outages. With these overloaded pipeline instances, outages were frequent, and whenever they occurred, the pipeline was unavailable for everyone working on that instance. Development work would come to a halt for large sectors of the organization — sometimes for hours at a time — while the problem was investigated.
- Their employees performed lots of manual work. Interviewees talked about the manual and often repetitive work inherent in common tasks such as user setup, pipeline updates, security scans, feature testing, and deployment preparation. Furthermore, these tasks were all active, meaning they required hands-on user engagement, and they couldn’t run in the background while other work was happening. Troubleshooting was particularly onerous in the previous environments; when problems occurred, enterprise support personnel had to spend hours searching message boards for possible fixes.
- Lack of standardization created governance challenges. With the high level of manual work required for all of these tasks, it was nearly impossible to ensure that processes were followed consistently. Interviewees used phrases such as “the Wild West” to describe their prior state, in which developers could customize tools and plugins, and processes were rife with human error. Under these conditions, enterprise pipeline teams had very little control over the diverse processes and situations they inherited from various teams, making it difficult to troubleshoot problems when they arose.
- Teams were siloed. With each team using its own customized tools and processes, developers found it very difficult to communicate between teams, which kept innovations and best practices siloed. Further, it was difficult to move individual developers from one team to another since user credentials were so customized that they were not easily portable. This not only hampered innovation but also restricted professional development opportunities for developers.
- Organizations were unable to scale. All of these factors, taken together, created an environment where organizational growth created problems rather than opportunities.
Composite Organization
Based on the interviews, Forrester constructed a TEI framework, a composite company, and an ROI analysis that illustrates the areas financially affected. The composite organization is representative of the six interviewees, and it is used to present the aggregate financial analysis in the next section. The composite organization has the following characteristics:
Description of composite. A $25-billion financial services organization with a large and sophisticated software development operation. It has 3,000 developers, organized in teams of eight to 10 and backed by an enterprise pipeline support team of 10 people.
Deployment characteristics. The composite organization migrates to CloudBees for CI and CD support and purchases 3,500 user licenses, allowing room for growth.
Quantified benefit data as applied to the composite
Total Benefits
| Ref. | Benefit | Year 1 | Year 2 | Year 3 | Total | Present Value |
|---|---|---|---|---|---|---|
| Atr | Savings from reduced outages | $4,255,335 | $4,291,245 | $4,291,245 | $12,837,825 | $10,639,046 |
| Btr | Savings from deployment automation | $9,310,000 | $11,770,500 | $12,420,538 | $33,501,038 | $27,523,056 |
| Total benefits (risk-adjusted) | $13,565,335 | $16,061,745 | $16,711,783 | $46,338,863 | $38,162,102 | |
Savings From Reduced Outages
Evidence and data. The majority of interviewees identified system outages as a major problem with their prior solution and one that the move to CloudBees largely solved for them.
- Two interviewees — the senior manager, enterprise CI/CD, from the transportation and business services organization and the manager of software engineering from an entertainment company — shared that CI outages used to happen once a month, sometimes for an hour and sometimes for several hours, and they would impact all of their teams. After deploying CloudBees, both interviewees said their outages were near zero, and when they do occur, they only affect one team.
- The senior manager, engineering enablement, from an entertainment company said their organization experienced outages in their CD pipeline almost daily for up to an hour, and they additionally had CI outages about once a month for several hours at a time. In both situations, all of their teams were impacted. They reported that a major benefit of moving to CloudBees was development pipeline segregation. They explained: “Before, we had a couple massive instances with 10,000+ jobs. Now, we have one controller per team. The segregated streams are now running faster, and it’s easier to target our troubleshooting.”
- The senior manager, enterprise CI/CD, for a transportation and business services company summarized, “CloudBees definitely improved our stability and ability to recover.”
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:
- Before moving to CloudBees and in its prior environment, the composite organization experienced outages monthly, each lasting an average of two hours. Because the organization was operating on a single, shared instance, its entire developer workforce was affected.
- During the organization’s migration to CloudBees in Year 1, it sees immediate improvement in the number, length, and impact of outages. With the support provided by CloudBees, average outage length is reduced to 30 minutes, and once developer teams are segregated on their own controllers, only a small percentage of the workforce is impacted during outages.
- As the organization completes its CloudBees migration, it sees even more improvement in Years 2 and 3 as the number of outages is reduced further
Risks. Various factors can impact the size of this benefit for individual organizations. These include:
- The extent of the organization’s shared instances under the prior state. If an organization were able to implement some segregation of developer teams in its previous environment, then the improvement in the “reduction of the blast radius,” as one interviewee put it, won’t be as significant as it is here for the composite, which we assume has all developer teams on one single instance.
- The extent of the organization’s outages under the prior state. If an organization experiences fewer and/or shorter outages than what is assumed here for the composite, this benefit won’t be as significant.
- Developers’ other tasks, independent of the pipeline. Forrester assumes that the productivity loss for a developer during a pipeline outage is 90%, indicating that only a handful of administrative tasks can be performed outside of a functional pipeline. However, if developers have other productive work they can focus on during an outage, this could reduce the size of the benefit.
Results. To account for these risks, Forrester adjusted this benefit downward by 5%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $10.6 million.
Savings From Reduced Outages
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| A1 | Annual outage frequency, prior state | Interviews | 12 | 12 | 12 | |
| A2 | Length of outage, prior state (hours) | Interviews | 2 | 2 | 2 | |
| A3 | Extent of developer workforce affected, prior state | Interviews | 100% | 100% | 100% | |
| A4 | Annual outage impact, prior state (hours) | A1*A2*A3 | 24 | 24 | 24 | |
| A5 | Annual outage frequency, improved | Interviews | 6 | 2 | 2 | |
| A6 | Length of outage, improved (hours) | Interviews | 0.50 | 0.50 | 0.50 | |
| A7 | Extent of developer workforce affected, improved | Interviews | 10% | 10% | 10% | |
| A8 | Annual outage impact, improved (hours) | A5*A6*A7 | 0.30 | 0.10 | 0.10 | |
| A9 | Total developers | Composite | 3,000 | 3,000 | 3,000 | |
| A10 | Fully burdened hourly rate per developer | TEI standard | $70 | $70 | $70 | |
| A11 | Productivity loss during outage | Composite | 90% | 90% | 90% | |
| At | Savings from reduced outages | (A4-A8)*A9*
A10*A11 |
$4,479,300 | $4,517,100 | $4,517,100 | |
| Risk adjustment | ↓5% | |||||
| Atr | Savings from reduced outages (risk-adjusted) | $4,255,335 | $4,291,245 | $4,291,245 | ||
| Three-year total: $12,837,825 | Three-year present value: $10,639,046 | |||||
Savings From Deployment Automation
Evidence and data. Interviewees agreed that automation of CD predeployment processes with CloudBees, from testing through approval, saved a tremendous amount of time and money.
- The senior manager, enterprise CI/CD, from the transportation and business services organization reported that their deployment time went from three days to less than a day. The technical product manager from the pharmaceutical company said their deployment time went from weeks to days. And the head of DevOps, cloud, and SRE from the financial services company said their deployment window went from “days to a week” down to a few hours.
- Automation sped up the testing and validation process. According to the director of DevOps for the hospitality company: “We used to have to orchestrate testing and validation manually across a variety of products within our security and quality engineering spaces. We managed to automate and integrate all of those things within the CloudBees pipeline. So now, teams don’t even have to know who to talk to or what to do. It’s a one-time setup — the things we need to run tests against. The tool returns a certain response, and we’ll see a value for what is a good versus a bad value, and we’re able to vet the threshold and say, ‘Yeah, this looks good, this can go into production,’ or it gets stopped in its tracks fairly early in the queue.”
- Interviewees also discussed how CloudBees automated the release approval process. According to the senior manager, enterprise CI/CD, for the transportation and business services company: “As we were trying to push code into production, there was a lot of manual effort. Everything requires approval, and each release would take one or more days collecting evidence and creating compliance packages to get them approved. Using CloudBees for continuous delivery, once the developer merges code into production, everything is automated. From development to production, it’s gone from days to less than a day.”
- The head of DevOps, cloud, and SRE for the financial services company highlighted how automation helped their organization with an important cultural transition. They said: “What we’re trying to enable is self-service through things like release management patterns and onboarding at their own pace by the developer teams. Automation has really sped up efficiency there. And because of that, we’re adopting a different methodology. We’ve offloaded a lot of the work from the centralized build team. You don’t need to get a ticket; you don’t have to wait. The developers not only manage the pipeline itself but also any peripheral requirements like provisioning a certificate, getting a key, or any of those prerequisites to deployments. It’s really changed the culture, how app teams are managing their own releases.”
- All of these time-savings and process improvements freed up members of the enterprise pipeline support team for higher-value work. As the senior manager, enterprise CI/CD, from the transportation and business services company said: “I have a team of 10 people, and I’ve been able to get down to where I only have two people supporting CI and just one on CD. We’ve put enough automation in place that my folks don’t have to put as many resources into supporting and running the tool, so we can spend more time helping developers with best practices and integrations.”
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:
- Before moving to CloudBees, the composite organization required 2 hours for preparation and approval before each deployment, and that process involved five people.
- The composite averages 20,000 deployments per year for 300 developer teams.
- When the organization migrates to CloudBees, automation reduces the predeployment prep time and the number of people required, starting in Year 1.
- As the organization completes its CloudBees migration, it sees even more efficiency improvements in Years 2 and 3.
- Also, in Years 2 and 3, greater efficiency allows the organization’s annual deployments to increase by 5% annually.
Risks. Various factors can impact the size of this benefit for individual organizations. These include:
- Variability in the prior-state release-approval process. While interviewees told a very consistent story in terms of the percentage improvement enabled by CloudBees’ CD automation, the size of the actual benefit varied across companies. Some organizations, and even some teams within organizations, had highly complex release-approval processes — such as the story of the daily, 100-person release approval call — while others were less involved.
- Productivity of the organization. The more annual deployments an organization has, the larger this benefit will be. To some extent, this variable also depends upon the maturity of the organization.
Results. To account for these risks, Forrester adjusted this benefit downward by 5%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $27.5 million.
Savings From Deployment Automation
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| B1 | Time for predeployment prep and release approval, prior state (hours) | Interviews | 2 | 2 | 2 | |
| B2 | People involved in approval, prior state | Interviews | 5 | 5 | 5 | |
| B3 | Subtotal: Person-hours per deployment, prior state | B1*B2 | 10 | 10 | 10 | |
| B4 | Deployments per year, all teams | Interviews | 20,000 | 20,000 | 20,000 | |
| B5 | Fully burdened hourly rate per developer | TEI standard | $70 | $70 | $70 | |
| B6 | Subtotal: Annual predeployment costs, prior state | B3*B4*B5 | $14,000,000 | $14,000,000 | $14,000,000 | |
| B7 | Time for predeployment prep and release approval, improved (hours) | Interviews | 1.0 | 0.5 | 0.5 | |
| B8 | People involved in approval, improved | Interviews | 3 | 2 | 1 | |
| B9 | Subtotal: Person-hours per deployment, improved | B7*B8 | 3.0 | 1.0 | 0.5 | |
| B10 | Deployments per year, improved, all teams | Interviews | 20,000 | 23,000 | 26,450 | |
| B11 | Subtotal: Annual predeployment costs, improved | B9*B10*B5 | $4,200,000 | $1,610,000 | $925,750 | |
| Bt | Savings from deployment automation | (B6-B11) | $9,800,000 | $12,390,000 | $13,074,250 | |
| Risk adjustment | ↓5% | |||||
| Btr | Savings from deployment automation (risk-adjusted) | $9,310,000 | $11,770,500 | $12,420,538 | ||
| Three-year total: $33,501,038 | Three-year present value: $27,523,056 | |||||
Unquantified Benefits
Interviewees also mentioned the following benefits that their organizations experienced but were not able to quantify:
- Improved product quality. The implementation of stable, repeatable CI processes removes human error during the software development stage. Further, the automated predeployment testing built into CloudBees CD makes it less likely that problematic feature code will be pushed into the pipeline. With improved product quality comes a better end-user customer experience, which could translate into user and revenue growth.
- Increased flexibility and innovation. Standard workflows and processes allow all teams to communicate and share best practices. With all employees tapping into the same set of tools and running operations the same way, individual developers are no longer tethered to a single team. This opens up all kinds of possibilities for collaboration and innovation. For example, one interviewee mentioned that some of their developers were being rotated among product teams that had similar features or shared business goals. The senior manager, engineering enablement, for one of the entertainment companies said, “CloudBees gave us the flexibility for developers to work on different teams, which enabled faster feature development and sped up innovation.”
- Improved user experience. Throughout the developer workforce, all team members benefit from the flexibility to move among product teams, which introduces more opportunities for professional development. And specific to the enterprise team, process automation makes organizations less dependent on their pipeline support staff to put in overtime hours and work overnight or weekend deployment shifts. With improved service and support and fewer unexpected outages, there are fewer overnight emergency calls. As a result, site reliability engineers enjoy a better work-life balance. These are all selling points organizations can use to attract and retain developers and engineers in a challenging labor market.
- Improved security. Features that help an organization improve its security posture include: standardization of processes, uniform creation of new-user credentials, and the ability to efficiently and unobtrusively run automated security scans and software updates. Several interviewees said they are now running pipeline software updates according to a regular, recommended schedule. Before, security updates took resources away from productive work for long periods of time, so they would only run updates occasionally. The senior manager for engineering enablement for one of the entertainment companies said: “Our security group would approach each team one at a time to scan all of their software builds, which would take a few months per team. Now, those scans roll out to all services at the same time, which allows the organization to iterate more quickly on security scanning.” These improvements are particularly significant for organizations in regulated business sectors such as financial services, government, and health care, but any industry that stores personally identifiable customer information can benefit from improved security.
Flexibility
The value of flexibility is unique to each customer. There are multiple scenarios in which a customer might implement CloudBees and later realize additional uses and business opportunities, including the possibility of accelerating revenue recognition through increased productivity and innovation.
As mentioned earlier, the automation built into CloudBees CD improves organizations’ efficiency by reducing the person-hours required to prepare for each deployment. In so doing, it also enables an improvement in overall productivity. The head of DevOps, cloud, and SRE from the financial services organization, for example, noted an impressive productivity boost: from 25,000 to 40,000 deployments a year over the five years following full CloudBees implementation. In the three-year financial model for the composite organization, Forrester captures this in Benefit B by ramping up annual deployments under the “improved state” by 5% each year.
The dollar value of this increased productivity is only realized in the sense that it amplifies the efficiency savings year over year. However, with additional data on revenue-generating software applications, an organization could potentially calculate the revenue impact, as opposed to the increased cost reductions resulting from this productivity increase. As an example, if an application generates a certain average dollar value per user and faster innovation cycles lead to faster user growth, then one could calculate the financial impact of more frequent release cycles for that software product, particularly if it can be shown that revenue is realized sooner than it might have been otherwise.
Additionally, new and improved capabilities added to CloudBees’ solutions can lead to new value opportunities for organizations. Implementing CloudBees’ new high-availability features, for example, can help enable significant scalability to remove bottlenecks from overloaded controllers, speeding up jobs to save time and reduce issues even more. It can also make it easier to upgrade software and release security patches more frequently, helping to enable even greater security improvements and other productivity savings measured above.
Flexibility would also be quantified when evaluated as part of a specific project (described in more detail in Appendix A).
Quantified cost data as applied to the composite
Total Costs
| Ref. | Cost | Initial | Year 1 | Year 2 | Year 3 | Total | Present Value |
|---|---|---|---|---|---|---|---|
| Ctr | Implementation costs | $0 | $1,335,400 | $418,660 | $22,000 | $1,776,060 | $1,576,529 |
| Dtr | Ongoing costs | $0 | $2,332,000 | $2,282,500 | $2,222,000 | $6,836,500 | $5,675,785 |
| Total costs (risk-adjusted) | $0 | $3,667,400 | $2,701,160 | $2,244,000 | $8,612,560 | $7,252,314 | |
Implementation Costs
Evidence and data. The six enterprise team leaders Forrester interviewed shared their experiences with the CloudBees implementation. While all interviewees agreed that the process was quite straightforward, there was some variability in timelines and costs.
- Interviewees spoke of an implementation timeline of seven to 12 months. Almost all of the interviewees’ organizations, with one notable exception, undertook a phased implementation, starting with the top-performing 25% to 50% of their teams and working outward through the organization.
- One organization was forced into an immediate, companywide implementation of CloudBees after a security breach compromised its platform. While this experience is far from ideal — and that atypical approach was not used in the financial modeling for the composite — that company’s experience proves that rapid implementation is possible.
- Interviewees said they dedicated a portion of their team’s time — between two and five FTEs — to the CloudBees implementation.
- Since their organizations had small pipeline support teams of fewer than a dozen people, all interviewees reported utilizing professional services — often a mix of consulting services provided by CloudBees and those provided by third-party consultants — to support their implementation projects. However, the amounts spent on professional services were quite variable, ranging from $150,000 to $750,000, with the scope of that spend dependent on several company-specific factors such as how completely the organization wanted to convert its library of legacy applications to the CloudBees pipeline; whether other, concurrent projects or circumstances, such as cloud migration, required additional consultant time; and the level of complexity and customization in the prior environment’s pipelines.
- The financial services organization reported the most robust effort in migrating legacy applications to CloudBees from their former pipelines. They budgeted 300 person-days for this portion of the project.
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization’s implementation of CloudBees:
- The organization spends $175,000 on professional services from CloudBees, which is toward the higher end of professional-services spending reported by CloudBees for new-customer onboarding. In addition, the organization contracts with other third parties for $250,000 in consulting services in Year 1 and an additional $50,000 in Year 2.
- The organization dedicates five FTEs to the implementation in Year 1 and two FTEs in Year 2.
- The composite organization takes a typical, phased approach to migration, with the project substantially completed in Year 1.
Risks. Various factors can impact the size of the implementation cost category for individual organizations. These include:
- Professional services spend. As mentioned, customer interviews revealed a wide range of professional services budgets. Forrester took a conservative approach to calculating these costs in the financial model, leaning toward the upper end of the range. However, an even higher spend is also possible, particularly for larger organizations with more complex CI/CD processes that choose to migrate all of their legacy applications as part of the CloudBees migration.
- Implementation strategy and timeline. For organizations that choose to lean more heavily on their internal teams vs. third-party vendors, implementation costs will vary based on the length of time required for the project. This approach is also riskier from a completion perspective, since other projects and day-to-day priorities will compete for staff time, potentially delaying completion or causing the organization to roll back its goals.
Results. To account for these risks, Forrester adjusted this cost upward by 10%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $1.6 million.
Implementation Costs
| Ref. | Metric | Source | Initial | Year 1 | Year 2 | Year 3 |
|---|---|---|---|---|---|---|
| C1 | CloudBees professional services | Interviews | $0 | $175,000 | $0 | $0 |
| C2 | Third-party professional services | Interviews | $0 | $275,000 | $75,000 | $20,000 |
| C3 | Internal FTEs assigned to migration | Interviews | 0 | 5 | 2 | 0 |
| C4 | Fully burdened annual salary per developer | TEI standard | $0 | $152,800 | $152,800 | $152,800 |
| Ct | Implementation costs | C1+C2+(C3*C4) | $0 | $1,214,000 | $380,600 | $20,000 |
| Risk adjustment | ↑10% | |||||
| Ctr | Implementation costs (risk-adjusted) | $0 | $1,335,400 | $418,660 | $22,000 | |
| Three-year total: $1,776,060 | Three-year present value: $1,576,529 | |||||
Ongoing Costs
Evidence and data. When it came to the ongoing costs of managing the CloudBees solution, interviewees’ experiences were more consistent.
- Interviewees’ licensing costs were, not surprisingly, directly related to their organization’s size. For the composite organization, CloudBees CI and CD product licensing costs equated to roughly $550 per user for an environment of 3,500.
- The senior manager, enterprise CI/CD, from the transportation and business services organization noted that their company incurred occasional ongoing professional services costs through CloudBees when their use of that support team for special projects exceeded what was included in the service contract. As an example of the scope of those charges, they said their last engagement cost about $50,000.
- Some interviewees also talked about additional investments in internal, post-migration communications programs to drive adoption and success. In particular, the manager of software engineering from one of the entertainment companies noted that a traditional training program wasn’t necessary or helpful, but investments in internal documentation that point developers to available self-service solutions within the CloudBees system was valuable. They said: “We have enough turnover between new employees and vendors that I don’t think a training program would be effective. Where we’re looking to invest is in documentation. We’ve been creating this, and we need to figure out a good way to direct [employees] there. We’re looking at integrating that information into our internal chatbot.
Modeling and assumptions. This section explains how the modeling is done.
- The composite organization has 3,000 developers and purchases licenses for 3,500 users to allow room for growth.
- The organization incurs some additional ongoing third-party professional services costs. As implementation is completed in Year 1 and usage matures in Years 2 and 3, these costs decrease over time.
- The organization also invests in the development of documentation and internal communications programs to support the transition to a self-service culture.
Risks. Various factors can impact the scope of the ongoing costs for individual organizations. These include:
- Ongoing professional services spend. As was the case for the prior category covering implementation costs, the use of professional services is probably the most variable input. While Forrester included a conservative amount (line D2 in the table below) for ongoing consulting engagements, some organizations may see lower or higher amounts.
- Level of investment in documentation and training. This can also vary according to the communications culture of an organization and the organization’s goals for moving to more of a self-service culture following CloudBees migration. Again, Forrester included a small line item in the model for these investments, but some organizations may dedicate more resources to those efforts.
Results. To account for these risks, Forrester adjusted this cost upward by 10%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $5.7 million.
Ongoing Costs
| Ref. | Metric | Source | Initial | Year 1 | Year 2 | Year 3 |
|---|---|---|---|---|---|---|
| D1 | CloudBees licenses (users) | Composite | 0 | $3,500 | $3,500 | $3,500 |
| D2 | Cost per license | Interviews | $0 | $550 | $550 | $550 |
| D3 | Licensing fees | Interviews | $0 | $1,925,000 | $1,925,000 | $1,925,000 |
| D4 | Professional services | Interviews | $0 | $125,000 | $100,000 | $75,000 |
| D5 | Development of documentation and internal communications | Interviews | $0 | $70,000 | $50,000 | $20,000 |
| Dt | Ongoing costs | D3+D4+D5 | $0 | $2,120,000 | $2,075,000 | $2,020,000 |
| Risk adjustment | ↑10% | |||||
| Dtr | Ongoing costs (risk-adjusted) | $0 | $2,387,000 | $2,337,500 | $2,277,000 | |
| Three-year total: $6,836,500 | Three-year present value: $5,675,785 | |||||
Consolidated Three-Year Risk-Adjusted Metrics
Cash Flow Chart (Risk-Adjusted)
Total costs Total benefits Cumulative net benefits-
The financial results calculated in the Benefits and Costs sections can be used to determine the ROI and NPV for the composite organization’s investment. Forrester assumes a yearly discount rate of 10% for this analysis.
These risk-adjusted ROI and NPV values are determined by applying risk-adjustment factors to the unadjusted results in each Benefit and Cost section.
Cash Flow Analysis (Risk-Adjusted)
| Initial | Year 1 | Year 2 | Year 3 | Total | Present Value | |
|---|---|---|---|---|---|---|
| Total costs | $0 | ($3,667,400) | ($2,701,160) | ($2,244,000) | ($8,612,560) | ($7,252,314) |
| Total benefits | $0 | $13,565,335 | $16,061,745 | $16,711,783 | $46,338,863 | $38,162,102 |
| Net benefits | $0 | $9,897,935 | $13,360,585 | $14,467,783 | $37,726,303 | $30,909,788 |
| ROI | 426% | |||||
Appendix A: Total Economic Impact
Total Economic Impact is a methodology developed by Forrester Research that enhances a company’s technology decision-making processes and assists vendors in communicating the value proposition of their products and services to clients. The TEI methodology helps companies demonstrate, justify, and realize the tangible value of IT initiatives to both senior management and other key business stakeholders.
Total Economic Impact Approach
-
Benefits represent the value delivered to the business by the product. The TEI methodology places equal weight on the measure of benefits and the measure of costs, allowing for a full examination of the effect of the technology on the entire organization.
-
Costs consider all expenses necessary to deliver the proposed value, or benefits, of the product. The cost category within TEI captures incremental costs over the existing environment for ongoing costs associated with the solution.
-
Flexibility represents the strategic value that can be obtained for some future additional investment building on top of the initial investment already made. Having the ability to capture that benefit has a PV that can be estimated.
-
Risks measure the uncertainty of benefit and cost estimates given: 1) the likelihood that estimates will meet original projections and 2) the likelihood that estimates will be tracked over time. TEI risk factors are based on “triangular distribution.”
The initial investment column contains costs incurred at “time 0” or at the beginning of Year 1 that are not discounted. All other cash flows are discounted using the discount rate at the end of the year. PV calculations are calculated for each total cost and benefit estimate. NPV calculations in the summary tables are the sum of the initial investment and the discounted cash flows in each year. Sums and present value calculations of the Total Benefits, Total Costs, and Cash Flow tables may not exactly add up, as some rounding may occur.
Appendix B: Supplemental Material
Related Forrester Research
“High-Performance IT: The Essential Technology And Practices ”, Forrester Research, Inc., January 15th, 2024.
“Software Development Market Insights, 2023,” Forrester Research, Inc., January 23rd, 2024.
Appendix C: Endnotes
1 Forrester Research, “Modern Development Metrics That Really Matter,” Forrester Research, Inc., July 3, 2023.
2 Total Economic Impact is a methodology developed by Forrester Research that enhances a company’s technology decision-making processes and assists vendors in communicating the value proposition of their products and services to clients. The TEI methodology helps companies demonstrate, justify, and realize the tangible value of IT initiatives to both senior management and other key business stakeholders.
PDF
