Forrester found and quantified seven benefits for the composite organization. These benefits fall into three distinct categories or areas of value for the composite organization:

For the composite organization, the relative magnitude of each category of benefit is as follows:

Release Efficiencies

Evidence and data. The interviewees said that BMC AMI DevX tools enabled them to eliminate laborious release processes. With their legacy toolchains, they often had to manually manage changes and code promotions. Two of the organizations even had developers in gatekeeper roles dedicated to this work full-time. With BMC AMI DevX Code Pipeline, though, the interviewees’ organizations automated their pipelines, documentation, and many other release processes, thereby saving their developers hours of effort per release.

• The mainframe DevOps lead in financial services summarized: “We had two full-time resources playing gatekeeper. … That was obviously a lot of manual overhead and effort. [But with BMC AMI DevX,] we effectively removed all of the overhead for developers having to document exactly what they are changing … plus the need for gatekeepers to manage the contentions, so we effectively saved two FTEs plus … about 8 hours’ worth of documentation and admin per release, on average. That on its own was already a major, major improvement and savings.”
  
  The same mainframe DevOps lead explained: “It was really a struggle for us [before BMC AMI DevX]. … In a lot of cases, we picked up contentions [(conflicts in the code changes)] quite far down the road, and there were quite a number of impacts, and it got really complicated. In some cases, it could get really, really complicated. If I were guesstimating, I’d say on average it could take about 2 hours to resolve conflicts.”
  
  The interviewee also said that their legacy version control tools made matters worse: “[Our old tool] had a limitation where we could only deploy 10 programs at a time. … So at that stage, we were regularly missing our [maintenance] deployment windows.”
  
  However, the interviewee said that with the BMC AMI DevX tools: “Now, it’s just a completely different level. We’ve done deployments of close to 200 or even more programs, and then we’re talking about it taking minutes. … We’re talking about, in all, five or 10 clicks of the mouse to deploy hundreds of programs and the added benefit of built-in approvals as well. [The BMC AMI DevX tools] really gave us the ability to empower the application owners to own and manage their own deployments.”

• Similarly, the platform lead for mainframe DevOps at another financial services organization said: “We are seeing deployment efficiencies … enabled by the tool [Code Pipeline]. [We’ve] reduced the deployment ‘job’ from about 400 [tasks] to four. … We are seeing deployments go a lot quicker and a lot smoother, and there’s a lot more transparency and confidence in the process. … Now that we’ve moved onto Code Pipeline, our process takes milliseconds. It’s not even seconds; it’s milliseconds. I can’t even measure it [because it’s so fast].”

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• Before implementing the BMC AMI DevX tools, the mainframe team released code to production four times per year. These were planned, major releases containing new features and fixes. (Minor releases and hotfixes were not considered for the model.)

• Before the BMC AMI DevX tools, the mainframe team also had two dedicated release managers. These two developers worked full-time managing contentions and changes to ensure smooth releases.

• In addition, for each release before BMC AMI DevX, other developers also spent a total of 7 hours documenting the changes.

• After adopting BMC AMI DevX tools such as Code Pipeline, the mainframe team automates much of the release process, including contention and change management and documentation. Based on the interviews, Forrester conservatively assumes that the mainframe team saves 96% of the time that it previously spent on each release.

• The average fully burdened hourly rate for mainframe developers at the composite organization before BMC AMI DevX is $97 (see Benefit D: Workforce Development).

• Developers recoup and use productively 75% of the time they save. Realistically, the developers might spend some of the time they save on coffee breaks, in meetings, etc. For the mainframe team at the composite organization, the total time saved is about two FTEs. These developers are reassigned from managing releases to writing new code.

Risks. This benefit may vary from organization to organization based on how much time the mainframe team spends managing releases prior to adopting BMC AMI DevX tooling.

Results. To account for this risk, Forrester adjusted this benefit downward by 10%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $655,000.

Increased Release Frequency

Evidence and data. The interviewees noted that their mainframe teams’ velocities increased without compromising quality (as discussed in Benefit C: Reduced Release Risks). This was due in part to the efficiencies they gained during releases (see Benefit A: Release Efficiencies). However, the interviewees also described additional benefits of the BMC AMI DevX tools at other stages throughout the SDLC — e.g., during program analysis and code review. In the aggregate, these benefits helped the mainframe teams to not only release more frequently but also to accomplish more — to release features and fixes faster than they had been able to before. Multiple interviewees said that their mainframe teams’ velocities increased to be on par with those of teams on non-mainframe platforms (e.g., distributed, etc.).

The interviewees described several ways in which the BMC AMI DevX tools helped their teams increase velocity:

• Visibility into legacy codebases made code changes easier. The interviewees said that the BMC AMI DevX tools (e.g., Code Insights) helped their developers quickly analyze and understand the effects of changes. This was especially important because their codebases were complex and decades old. The BMC AMI DevX tools not only saved the developers time but also helped them be more confident about the changes they were making.
  
  For example, the mainframe systems engineer in financial services said: “Before, we had some sources of code and data that were more than 10 years old. We didn’t know why they were there, what had changed inside of them — it was a complete mess. Now, we have statistics.”
  
  Similarly, the platform lead for mainframe DevOps at another financial services organization attested: “We now have [features] like impact analysis [in Code Insights], which creates visibility into environments that many people didn’t know. [These tools] are helping map out legacy systems that previously weren’t well known or documented. [For example,] how the application sits together, what calls it, and when it’s called — those kinds of things. [The BMC AMI DevX tools] are creating visibility.”

• Superior source code management improved team coordination. The interviewees said that Code Pipeline enabled their organizations to better manage mainframe development. Before Code Pipeline, the mainframe teams had lacked reliable systems of record and sources of truth for their code. As a result, collaboration had required significant overhead, and the developers had tended to work in silos. However, Code Pipeline removed much of this friction, allowing developers to coordinate more easily. The interviewees also said that the BMC AMI DevX tools helped managers better oversee their teams’ development efforts. Both of these factors contributed to higher velocities.
  
  The mainframe DevOps lead in financial services emphasized this the most: “For me, the most important part is Code Pipeline. Before Code Pipeline, we didn’t have a clue [about] why a program was changing or going to production. Why, why, why — we didn’t have a clue. Now, with Code Pipeline, we gain control. We know which source [code] is checked out, where it is, why it’s changing, and what is being changed. We now have a history of what is pushed to production and why it’s pushed to production. We have control of everything. That’s a very, very, very important point.”
  
  The mainframe systems engineer also described how the BMC AMI DevX tools made code review and parallel development not only possible but easy. They said: “It’s easy to do peer review. You can see what has changed between the two versions. It’s easy to just look at those lines [instead of] reviewing the whole program.”
  
  The interviewee added: “[A developer] might create a new feature, wait, and then someone else might change the same source [code] and push it to production. Now, with Code Pipeline, you can’t push something to production without taking the latest version to production. So we can be sure that the developers know [about new changes] and that they take care of that in their source [code]. That’s the biggest, biggest benefit for us.”

• Modern, integrated tooling supported enterprisewide alignment. In addition to improving coordination among mainframe developers, the BMC AMI DevX tools enabled the mainframe teams to better coordinate and synchronize their releases with those of the teams on non-mainframe platforms. Many of the apps at the interviewees’ organizations used both mainframe and non-mainframe systems (e.g., a banking app might have web app UI but still process transactions on mainframe). However, with their legacy tooling, the mainframe teams at the interviewees’ organizations had had tedious, idiosyncratic release processes that had made it difficult to align with their non-mainframe peers. According to the interviewees, the mainframe teams had often been the laggards. The BMC AMI DevX tools changed that, though, by supporting modern development processes. Interviewees from four of the organizations reported that mainframe development velocity was now on par with that of non-mainframe teams. (The interviewees from the other two organizations also believed that mainframe development velocity had improved, but they could only attest to this anecdotally. Their organizations were relatively new to DevOps and so did not have reliable measures of developer output.)
  
  The mainframe system engineer in financial services said: “Now, with these tools and all the APIs they have, we’ve connected our mainframe platform to our enterprise deployment model. We’ve connected the [mainframe] platform just like any other platform. … And that’s what we tell [developers]: The mainframe is a platform like any other one, and you can use it like you use [a desktop computer]. It’s the same.”
  
  Similarly, the mainframe DevOps lead in financial services said: “What we’ve started doing as an organization is PI [(program increment)] planning [together]. We manage dependencies that way so that [development] cadences are constant across the organization.”
  
  Lastly, the CIO in healthcare reported that lead times for mainframe development projects had fallen by about 20% (e.g., from 12 months to 10 months) since their organization had adopted the BMC AMI DevX tools.

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• Adopting the BMC AMI DevX tools helps the mainframe team release more frequently.

  • In Year 1, there is no change, and the mainframe team still deploys to production four times per year — the same release schedule it had with its prior tooling.
  • However, by Years 2 and 3, the mainframe team benefits from process changes enabled by the BMC AMI DevX tools, and the team increases its release cadence. In Year 2, the mainframe team achieves five major releases, and in Year 3, the team achieves six major releases.

• Adopting the BMC AMI DevX tools also helps the mainframe team change more programs per release. By leveraging features of the BMC AMI DevX platform — such as program analysis in Code Insights and an integrated development environment (IDE) in Workbench — the mainframe development team changes a greater percentage of its mainframe codebase per year. Ultimately, the team delivers more work with business value — e.g., new features, fixes, and so on.

  • Forrester assumes that the composite organization’s mainframe codebase contains 20 million lines of code and that a mainframe program averages 2,000 lines of code. Therefore, the composite organization has around 10,000 programs in its mainframe codebase.
  • Before the mainframe team adopts BMC AMI DevX tools, it is only able to change 10 programs per release. This is due to both specific technical limitations of the team’s legacy tools and suboptimal processes that the tools create.
  • With the BMC AMI DevX tools, however, the mainframe team changes far more programs per release: 33 programs per release in Year 1 (a 225% increase), 55 programs per release in Year 2 (a 450% increase), and 100 programs per release in Year 3 (a 900% increase). Features in the BMC AMI DevX tools (e.g., impact analysis, etc.) help the developers quickly and more confidently change legacy code. In addition, the tools enable modern workflows and faster, more efficient processes.

• To calculate the business value of the mainframe team’s increased velocity, Forrester assumes that the composite organization earns $10 billion in annual revenue and that mainframe software drives or supports 10% of the composite organization’s business.³ Therefore, the value of the mainframe codebase is about $1 billion.

  • Before using the BMC AMI DevX tools, the mainframe team’s slower release schedule and limited capacity to change programs means that, in aggregate, the team’s annual development efforts affect only about 0.4% of the composite organization’s mainframe codebase. Based on the value of the mainframe codebase to the composite organization, the mainframe team’s annual changes are worth around $4 million.
  • However, with the BMC AMI DevX tools, the mainframe team both releases more frequently and changes more programs per release. In aggregate, the mainframe team’s development work affects 1.3% of the codebase in Year 1, 2.8% of the codebase in Year 2, and 6.0% of the codebase in Year 3. Based on the value of the mainframe codebase, the changes are thus worth, in total, $13 million in Year 1, $28 million in Year 2, and $60 million in Year 3.
  • The benefit to the composite organization of increased development velocity is the difference in the value of annual mainframe codebase changes before and after BMC AMI DevX tooling.
  • Because the mainframe codebase changes are assumed to support annual revenue, Forrester isolates and calculates the benefit to the composite organization by applying an operating margin of 12%.⁴ Applying an operating margin accounts for the business costs associated with generating revenue so that the final total for this benefit is net-new business value.
  • Forrester assumes that the composite organization’s annual revenue remains relatively constant at around $10 billion per year. That is, Forrester’s model for the composite organization does not account for business growth — neither organic business growth nor growth that might result from the increased mainframe velocity (e.g., new features that drive customer acquisition, etc.). Forrester makes this assumption both to be conservative and to better isolate the discrete impact of the BMC AMI DevX tools on the composite organization.

Risks. This benefit may vary from organization to organization based on:

• Mainframe velocity prior to adopting BMC AMI DevX tools.

• Annual revenue supported by the mainframe codebase.

Results. To account for these risks, Forrester adjusted this benefit downward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $7.1 million.

Reduced Release Risks

Evidence and data. Interviewees reported an increase in overall mainframe software quality with the BMC AMI DevX tooling. They described several ways in which the tools helped their teams:

• Tools including Total Test, Code Debug, and Code Insights created efficiencies when testing and debugging. This time savings for developers plus automation features throughout the BMC AMI DevX suite reduced the total number of issues.

• When issues did escape testing (as inevitably happens in all software development settings), Code Pipeline and other BMC AMI DevX tools made it easy to quickly roll back to the most recent stable releases. Fast rollbacks minimized the fallout from any issues.

The result of these benefits — fewer issues combined with fast rollbacks — was that the interviewees’ organizations experienced fewer mainframe issues even as their development velocities increased. On the whole, the BMC AMI DevX tools reduced the risks associated with changing legacy mainframe codebases.

The interviewees described multiple ways in which the BMC AMI DevX tools helped improve mainframe software quality at their organizations:

• Toolchain automation reduced manual errors. The mainframe DevOps lead in financial services explained: “In the past, we didn’t control our JCLs [(job control languages)]; they were just copied from one environment to another. And in a lot of cases, we ended up with JCL errors. … You know, under pressure, when you’re trying to resolve something very complicated, [it’s easy to make mistakes]. In a lot of cases, we actually missed components, causing issues. But when we adopted pipelines [using Code Pipeline], we started including our JCLs as well. And just that on its own improved quality — just by removing those manual interventions.”
  
  The CIO in the healthcare industry described similar benefits from automation with BMC AMI DevX File-AID: “We have to generate realistic datasets for testing. … But really, by improving our testing, we’re ensuring privacy and compliance [with industry regulations]. … [With File-AID,] we’ve automated the data handling process, [and since we’ve] integrated with other [BMC AMI DevX] tools … the manual interventions [aren’t there]. … [Our old tool] was very powerful, but pretty complex. … The fact that [File-AID] handles all [our] data types and has good compliance features [has helped us] reduce manual data handling by up to 30% and [increase] data compliance [with regulations] by up to 20%.”

• Powerful features increased developer productivity when testing and debugging. The CIO in healthcare explained: “[Our old tool] is a pretty commonly used tool, but it has a high learning curve. It doesn’t have a great user interface. [In comparison, Code Debug] works so well and is easy to learn and has all of these features built in. When we switched from [our old tool] to [Code Debug], the speed of the whole [debugging] process improved. We’ve noticed a 20% improvement [in debugging times] compared to what we were doing before.”
  
  The mainframe systems engineer in financial services also described how BMC AMI DevX tooling made debugging easier: “[Before, with our old tools,] if you had a problem or a program that failed, you just knew the source code [that had failed], but you didn’t know what the change was. Now, [we] have the [change] history, so you can compare everything and [pinpoint] the problem. [You can see] that the problem is in a [certain] line [of code] because that line changed, not the 100 other lines. So it’s very easy [for developers to debug].”

• Rapid rollbacks minimized the impact of any issues. The mainframe systems engineer in financial services estimated that the typical time to restore service after a failure (i.e., the mean time to repair or MTTR) went from one to two days down to minutes with the BMC AMI DevX tools. They also estimated that such failures occurred around 10 to 20 times per year, meaning that the BMC AMI DevX tools were significantly improving uptime at the organization.
  
  The mainframe systems engineer explained: “Before we had Code Pipeline, if you had something in production that failed and you wanted to fall back to the previous version, then you needed to ask someone on the storage team to do a restore, and you just hoped that the last backup they could change could take the load. … You’d lose time because you needed [to coordinate with] someone else. … So it’d take maybe one or two days before you could restore the service.”
  
  They continued: “Now, if you want to restore a service because you want to fall back, it takes 2 minutes. … You just right-click, [and] it’s done. You just fall back, and that’s it. I think that’s very important. … Actually, I can say that 80% [of the process] is automatic, and I think that in one or two months, we’ll have a fully automatic chain.”

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• Any mainframe software issues that escape to production disrupt business operations and result in downtime.

• Before the mainframe team adopts BMC AMI DevX tooling (i.e., with its legacy toolchain):

  • The release failure rate was 25% (that is, one in every four releases had an issue).
  • The average downtime after a failed release was 48 hours (i.e., it took around two days to restore service).

• After the mainframe team adopts BMC AMI DevX tooling:

  • The release failure rate falls by 33% (i.e., there are 33% fewer issues in production), from 25% to 17%. Automation reduces manual errors, and the BMC AMI DevX tools improve testing and debugging so that fewer issues escape.
  • The average downtime after a failed release also falls by 98%, from two days to about 1 hour. Rapid rollbacks enable the team to quickly restore service after an issue.
  • In aggregate, these two improvements result in the composite organization experiencing 99% less downtime with its mainframe applications.

• With the mainframe team’s legacy tooling, codebase changes were relatively risky because there was a significant chance of issues with each release. However, by helping to reduce both the number of issues as well as the fallout from those issues, the BMC AMI DevX tools reduce the risks associated with changing critical code. The benefit to the composite organization may also be thought of as an improvement in quality since there are fewer issues overall as well as greater system stability.

• To calculate the business value of reduced release risks, Forrester assumes that any issue that escapes to production disrupts 75% of business operations supported by the mainframe. Forrester also assumes that the composite organization conducts business 24 hours per day, 365 days per year.

  • This means that for every hour of mainframe downtime, the composite organization loses (or misses out on) around $85,616 in revenue. In other words, for the composite organization, the cost of 1 hour of mainframe downtime is $85,616.
  • The benefit to the composite organization of reduced release risks is the difference in revenue lost before and after BMC AMI DevX tooling.
  • To calculate the net-new business value of this benefit, Forrester applies an operating margin of 12%.

Risks. This benefit may vary from organization to organization based on:

• Frequency and severity of mainframe issues (both minor bugs as well as major issues that disrupt the business) before adopting BMC AMI DevX tools.

• Annual revenue supported by the mainframe codebase.

• Adoption of BMC AMI DevX suite. Like the interviewees’ organizations, the composite organization accrues benefits primarily from toolchain and pipeline automation (i.e., with Code Pipeline, etc.). However, organizations may realize greater benefits by leveraging more BMC AMI DevX tools (e.g., Total Test, Strobe, etc.) and by making more extensive investments in testing automation and performance than the interviewees’ organizations did.

Results. To account for these risks, Forrester adjusted this benefit downward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $1.3 million.

Workforce Development

Evidence and data. The interviewees said that the BMC AMI DevX tools made it easier to recruit and retain new mainframe developers. They praised the usability of the tools, saying that the BMC AMI DevX tools were significantly easier to learn than other products. For the interviewees, this was critical because their teams were disproportionately senior developers nearing retirement. By making mainframe development more accessible, the BMC AMI DevX tools helped the interviewees recruit the next cohort of developers and thereby support the longevity of the platform at their organizations.

• The project lead for change management at a financial services organization explained why the BMC AMI DevX tools appealed to new mainframe developers: “Eighty percent of junior developers worked with Eclipse [(an open-source integrated development environment [IDE] supported by BMC AMI DevX Workbench)] because in a lot of schools and universities, Java is the first programming language that they learned. All developers know Eclipse [as their] graphical interface. … I remember that when I started, I saw the green screen, and I thought, ‘What’s that?’ I think it’s the same for junior developers who start in our company. It’s easier to have a graphical interface. They have a modern platform. You can do [things] with clicks and not with commands.”
  
  The PDM at another financial services organization echoed those comments: “They [junior developers] went to college and learned Git [(an open-source software version control system)] and Eclipse. So when I stick [Workbench in] Eclipse in front of their faces, they say, ‘Okay, I know what this is.’ So yes, it’s infinitely easier [to recruit them].”

• The mainframe systems engineer in financial services reported retention benefits: “Retention is very good; I haven’t seen a lot of young [developers] leave. Before [the BMC AMI DevX tools], retention was [harder]. Developers coming out of school wanted to work on fancy [non-mainframe] stuff. Honestly, now [with the BMC AMI DevX tools], they can work in the same ways; they have the same tools and the same processes. They just check out [code], compile, build, and so on. It’s the same for them, and our goal is to simplify [even further], so that they don’t have [to deal with] any technical complexities from the system, and [the system] just works like any other technology.”

• The platform engineer in financial services said that the BMC AMI DevX tools appealed to new and experienced developers alike: “The newer [developers] prefer to work in Workbench because of its capability to visualize programs. They can do better analysis and have faster turnaround times. … But there’s a demand for training from [developers] who have traditionally used green screen [because they work with developers who use Workbench]. The interest is there; they’re slowly coming.”

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• Before it adopts the BMC AMI DevX tooling, the composite organization has 300 mainframe developers.

  • Senior developers comprise 60%. These developers have decades of experience, and many are approaching retirement. At the composite organization, the average base salary of a senior developer is $175,000 (or $236,250 fully burdened). (A fully burdened salary is a base salary plus the cost of benefits. Forrester assumes that fully burdened salaries are 135% higher than base salaries.)
  • Midcareer developers comprise 30%. These developers have years of work both behind and ahead of them. The average base salary of a midcareer developer is $125,000 (or $168,750 fully burdened).
  • Junior developers comprise 10%. These developers are just beginning their careers, and many were hired right out of university. Junior developers have at most a few years of experience on mainframe. The average base salary of a junior developer is $75,000 (or $101,250 fully burdened).

• Ten percent of senior developers retire per year. (In other words, all senior developers would be expected to retire within 10 years.)

• As senior developers retire, the composite organization hires new junior developers to maintain and even grow its mainframe development capacity.

  • The BMC AMI DevX tools and the modern software development practices enabled by those tools make the mainframe an attractive and accessible platform for new developers, and the composite organization recruits and retains more junior developers than before.
  • With BMC AMI DevX tooling, junior developers are on average 75% as productive as senior developers. Thus, for every senior developer who retires, the composite organization hires more than one junior developer to maintain the team’s overall development capacity.
  • By the end of Year 3, the number of senior developers at the composite organization has fallen from 180 to 126; the number of junior developers has risen from 30 to 102; and the mainframe team size has increased to 318 developers.

• To calculate the business value of the workforce demographic shift enabled by BMC AMI DevX tools, Forrester compares the composite organization’s aggregate spending on mainframe developer salaries before adopting the new tooling to its spending afterward. Over the three-year study period, the total spending on mainframe developer salaries falls, yielding savings for the composite organization.

Risks. This benefit may vary from organization to organization based on:

• Workforce demographics (i.e., percentages of developers who are senior and nearing retirement versus midcareer, junior, etc.) prior to adopting BMC AMI DevX tooling.

• Average developer salaries and the differences between them. As the difference between senior developer and junior developer salaries narrows, this benefit decreases.

• Senior developer retirement rate. If senior developers retire faster, this benefit increases.

• Average junior developer productivity relative to senior developer productivity. Where senior developers are radically more productive than junior developers (e.g., twice as productive or more), organizations must hire far more junior developers to maintain total development capacity as senior developers retire. This decreases the magnitude of this benefit.

• Organization-specific efforts to recruit new mainframe developers and the success of those efforts.

This benefit reflects the composite organization’s success hiring new mainframe developers as it confronts the pressing challenge of an aging workforce. This benefit does not suggest that organizations should replace senior team members with junior ones. Senior developers often possess irreplaceable skills and knowledge, which is why Forrester models them as more productive than junior developers. Organizations not expecting substantial numbers of retirements within the next few years can still realize success recruiting new mainframe talent, but they may not find this benefit applicable.

Results. To account for these risks, Forrester adjusted this benefit downward by 10%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $7.9 million.

Faster Onboarding

Evidence and data. Interviewees from five of the six organizations reported that the onboarding periods for new developers were shorter with the BMC AMI DevX tools than with their prior toolsets. That is, when new developers joined the mainframe team, they ramped up to full productivity faster. The interviewees attributed this both to the tools’ ease of use and the improved processes that the tools enabled.

• The CIO in the healthcare industry said that after adopting the BMC AMI DevX tools, onboarding times fell by 25%, from 12 months down to nine months. They explained: “Coming over here, it’s a different process in the government sector, so there are things [developers have to learn] that just inevitably go with that. [But] in general, we’re looking at a 25% to 30% reduction in the amount of time for [new developers] to learn [BMC AMI DevX tools] versus our [old] platform. In my opinion, that’s a significant time saving. Why? The BMC tools are easier to use. The intuitive interface, the modern [features], etc. But really, what does that do for us in the long term? It reduces our overall operating costs for managing and maintaining these systems.”

• The mainframe systems engineer in financial services reported that typical onboarding times had fallen by 50% to 67%, from six to nine months down to three months. They said that new developers were usually only 50% productive during this ramp-up period, so reducing it had both increased velocity and resulted in savings. The mainframe systems engineer explained: “Before, [as a new developer,] you needed six to nine months to start working productively on [our old] tooling because you needed to know TSO, you needed to know the commands — you needed to know everything. Now, in [around] three months, you can be completely free and can start working productively because you have all [the BMC AMI DevX] tooling and because you don’t need to [learn a] new interface.”

• The mainframe DevOps lead at another financial services firm reported that new developers needed 50% less training than before — only 2 hours instead of 4. They said: “We [used] to run training sessions for 4 hours. [But now,] someone on the team needs to spend 2 hours with a new recruit, and they’re up and running.”

• The project lead for change management at third financial services organization also believed the BMC AMI DevX tools reduced onboarding times. They said: “It helps because they know the interface. [The BMC AMI DevX tools] are very easy to use when you know Eclipse. [Onboarding times] are a little bit individual, [but we have had] people who are very interested [(willing to learn)] and could do [their first] production changes in six months.”

• The PDM in financial services said that onboarding times were faster because — in contrast to the organization’s prior solution, a homegrown toolset — the BMC AMI DevX tools were standard in the industry and had ample learning resources available. The PDM said: “I can say [to a new developer], ‘Here’s a bunch of training videos. Here’s a bunch of industry standard [resources]. Here’s a bunch of BMC [resources]. Go learn.’ So I spend less time training.”
  
  The PDM added that BMC AMI DevX’s support for both traditional green screen and GUIs also reduced onboarding times. The PDM continued: “There’s less [time spent] getting [new developers] comfortable with the tooling. [I can say to a new developer,] ‘Oh, what interface do you like? Okay, use that interface.’ I didn’t have that in my homegrown [tooling]. [We’re] avoiding eight, 10, 12 weeks of training time [due to the support for] different user interfaces.”

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• Before the composite organization adopts BMC AMI DevX tools, new developer hires take nine months to ramp up to full productivity.

• During this onboarding period, new developer hires are only 50% as productive as they could be. This is because they must spend time learning the mainframe team’s systems, processes, tools, etc. (In the software industry, onboarding periods for new hires are normal.)

• With the BMC AMI DevX tooling, new developer hires ramp up to full productivity 50% faster.

• To calculate the business value of this benefit, Forrester compares the total annual productive capacity of a new developer hire before and after BMC AMI DevX tooling.

  • Before BMC AMI DevX tools, new developers are 50% productive for nine months and 100% productive for three months. New developers therefore contribute 7.5 months of output during their first year.
  • After BMC AMI DevX tools, new developers are 50% productive for 4.5 months and 100% productive for 7.5 months. New developers therefore deliver 9.75 months of output during their first year.
  • Thus, with BMC AMI DevX tools, the composite organization gains around 2.25 months of incremental output for each new developer hire. For the composite organization, this additional developer output is about the same as adding 5 developer FTEs per year who use the organization’s legacy tools.

• To model this benefit for the composite organization, Forrester assumes that the only new developers hired are the junior developers recruited in Benefit D: Workforce Development.

• To be conservative, Forrester applies a productivity recapture rate of 75%. Although the new developers gain 2.25 months of additional productivity, they may not use all that time on work that adds value for the composite organization. (E.g., they may spend some of the time on coffee breaks, in meetings, etc.)

Risks. This benefit may vary from organization to organization based on how much time new developer hires need to ramp up to full productivity with an organization’s legacy solution.

Results. To account for this risk, Forrester adjusted this benefit downward by 5%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $825,000.

Developer Productivity

Evidence and data. The interviewees reported that the BMC AMI DevX tools helped their developers save time on everyday tasks such as writing code. According to the interviewees, this was an additional benefit beyond the team-level benefits captured in Benefit B: Increased Release Frequency and Benefit C: Reduced Release Risks. The interviews said that the BMC AMI DevX tools helped developers work faster and more efficiently, and they provided anecdotal evidence that improving the developer experience this way had improved morale.

• The mainframe systems engineer in financial services estimated that developers completed code changes in one-third less time than they did with the organization’s legacy tooling: “For the productivity part, I think they [the developers] have also cut a third of the time they spend on modifications because now they have more control. They know what they’re changing, and they can also [roll] back more quickly if they have some trouble. So that’s a third of their time saved.”

• The mainframe DevOps lead in financial services mentioned, “I think [the BMC AMI DevX platform] has also changed development morale. I mean, if I just think about the amount of time [developers] had to spend creating documents, talking to gatekeepers, [and performing] complex copybook impact analyses. …”

• The product engineer in financial services also explained: “I’ll give you an example: [Take] something like the program analysis function. One developer said it [used to] take days to map out a program and follow through the processes in a program. But now [with the BMC AMI DevX tools], she does it in minutes, and on top of that, she’s able to share with her team members her documentation in terms of when she submits her changes. It’s more understandable; it’s standard; it’s easier. So when she’s working with her analysts and they do impact analysis, it’s now literally at the click of a button, whereas before it would take them days. … It’s much quicker; it’s much cleaner. So [now] she doesn’t even mind when the business wants more changes because she knows the information she’s getting from the program analysis and impact analysis in Code Pipeline is accurate.”

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• Before the composite organization adopts BMC AMI DevX tools, developers spend around 24% of their time — or 500 hours per year — coding.⁵

• With the BMC AMI DevX tools, developers spend 33% more time coding and less time performing less-productive activities like administrative work.

  • With the BMC AMI DevX tools, each developer contributes around 165 more hours per year of coding output. For the composite organization, this additional development output is about the same as adding 25 developer FTEs per year who use the organization’s legacy tools, an approximately 8% increase in effective team size.

• To calculate this benefit for the composite organization, Forrester values the productivity gains based on the average mainframe developer salary after the organization adopts BMC AMI DevX. This average developer salary is lower than before BMC AMI DevX because the composite organization hires new junior developers (as described in Benefit D: Workforce Development).

• To be conservative, Forrester also applies a productivity recapture rate of 75%.

Risks. This benefit may vary from organization to organization based on:

• Productivity gains with BMC AMI DevX tools. Productivity gains may be greater or smaller than modeled for the composite organization, and productivity gains may vary over time based on adoption of the BMC AMI DevX platform.

  • Productivity gains are especially likely to be greater than modeled for organizations that — unlike the composite organization — have no solutions in place before adopting BMC AMI DevX tools. The modeling for the composite organization reflects the experiences of the interviewees’ organizations, which all realized developer productivity gains after switching from alternative platforms.

• Adoption of BMC AMI DevX tools. Interviewees from two of the six customer organizations reported that all their developers were using the BMC AMI DevX tools (i.e., 100% adoption). Interviewees from the other organizations reported high but not full adoption (e.g., 75% to 90%). If developers do not use the BMC AMI DevX tools, then the total team productivity gains will be lower.

Results. To account for these risks, Forrester adjusted this benefit downward by 5%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $8.1 million.

Reduced Admin Effort Due To Tool Consolidation

Evidence and data. On the whole, the interviewees’ organizations spent about half as much time on toolchain administration for the BMC AMI DevX tools than they had spent with their legacy solutions. For some of the interviewees — especially those at organizations with homegrown tooling — the most important benefit they realized from switching to the BMC AMI DevX platform came from retiring expensive legacy tools and consolidating their toolchains around a single trusted vendor.

While interviewees at all the organizations reported that the BMC AMI DevX required less administrative effort than their prior solutions, the interviewees at organizations with homegrown tooling reported the greatest savings. Often these homegrown tools were decades old and maintained by developers who wanted to retire, meaning that switching to the BMC AMI DevX platform was urgent and necessary to support mainframe infrastructure going forward.

• The product engineer in financial services explained: “One of the greatest successes for [Code Pipeline] is that we managed to replace one of the biggest, key, main dependencies and one of the biggest risks for the bank when it comes our source code management and deployment processes.”
  
  The organization’s prior source code management tool (SCM) was 30 years old. It had originally been built by four developers, but for the past 20 years had been maintained by just one developer who worked on the tool full-time. The platform lead for mainframe DevOps at the same financial services organization said: “He’s a superstar, [but] he’s retiring this year. So [Code Pipeline] is our gift to him so that he can retire peacefully and not stress.”
  
  The platform lead continued: “The consolidation of the SCM means that we get to reduce some of the tool stacks and the footprint from multiple vendors. … It’s a consolidation of tools; therefore, it’s cost savings, and there’s an improvement of processes, which will reduce the number of infrastructures we need.”
  
  The platform lead described not only cost savings but also benefits from modernizing with the BMC AMI DevX platform: “From a strategic perspective, one of the top three benefits is risk mitigation. It’s improving our risk posture because we’re centralizing our tools, we’re consolidating from five to one — that kind of thing. We’re modernizing and re-architecting. … [The BMC AMI DevX tools] are creating the flexibility to integrate with modern technology and capabilities — not just BMC toolsets, but also things like SonarQube [(an open-source code analysis tool)]. … [This] is a modernization of our current processes and our current tool stack and technology, [and that reduces] risk.”
  
  The platform lead also mentioned ancillary system optimization benefits: “We’ve identified batch jobs and stuff like that are running and supposed to be decommissioned. [That was] enabled that with the [BMC] tool set.”

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• Before switching to the BMC AMI DevX tools, the composite organization has one developer FTE dedicated to maintaining and supporting its legacy tooling full-time.

• Once the composite organization switches to the BMC AMI DevX tools, it retires its legacy solutions, and the prior administration effort goes away entirely.

• The BMC AMI DevX tools require ongoing administration effort of their own. This work is captured and modeled for the composite organization in Cost J: Ongoing Administration Effort.

• The one developer FTE who maintains the legacy solutions is reallocated to other value-adding projects. Forrester applies a conservative 75% productivity recapture rate.

Risks. This benefit may vary from organization to organization based on how much effort an organization spends supporting its legacy tooling.

In addition, this benefit total could be higher for organizations that also realize licensing cost savings by switching platforms. Third-party toolchains have two costs: the licensing costs charged by the vendor as well as the in-house effort to administer the tools (e.g., manage updates, support users, oversee integrations, etc.). For several reasons, Forrester calculates the benefit of toolchain consolidation for the composite organization based on administrative effort savings alone. First, platform licensing costs can be highly variable in the market (e.g., due to discounting, etc.). Second, the interviewees did not provide robust data on licensing cost savings. This was because for them, any licensing cost savings were far less significant than the administrative effort savings. Third, Forrester’s approach is conservative; organizations that also realize licensing cost savings by switching to BMC AMI DevX can simply expect greater benefits. For more information on the BMC AMI DevX licensing costs for the composite organization, see Cost H: Licensing Costs.

Results. To account for these risks, Forrester adjusted this benefit downward by 5%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $357,000.

Unquantified Benefits

Interviewees mentioned the following additional benefits that their organizations experienced but were not able to quantify:

• Superior enterprise support. The interviewees appreciated their partnerships with BMC, and they said they had stronger relationships with BMC than with other solution providers.

  • The mainframe DevOps lead in financial services emphasized that BMC was at the forefront of mainframe technology: “It’s quite ironic, just thinking about it. I went to the DevOps Enterprise Summit a couple of years ago, and on the stage, [our prior vendor] was talking about how they were only [recently] really embracing DevOps themselves. And that’s funny because the person giving the talk was the same person whom we reached out to six years ago [with our problems].”

• • The project lead for change management in financial services said: “The support by BMC is great. [The support representatives] are great support people — they’re the best. They’re very fast and very motivated. For example, when I open a new case, I’ll get a response in 2 hours, [and it’s from someone] competent. You really feel that they have knowledge in this area and that you’re talking with [experienced] support. And that’s great because then you can spend time on other things. [Our prior vendor] was not so professional. [With BMC,] it’s very comfortable. I have a good connection with our key account manager. I can call them and discuss [my challenges,] and they’re very helpful.”

  • The PDM in financial services appreciated BMC’s product expertise: “The bottom line is they eat their own dog food or drink their own champagne — however you’d like to phrase it.”

Flexibility

The value of flexibility is unique to each customer. There are multiple scenarios in which a customer might implement BMC AMI DevX and later realize additional uses and business opportunities, including:

• Continuous improvement with zAdviser. Although not all of the organizations were using zAdviser, the interviewees from the organizations that were using it said that insights from its data helped improve their mainframe processes. In many cases, zAdviser provided data that the interviewees had never had before.

  • The mainframe DevOps lead in financial services explained: “The fact that we now have that level of data at our fingertips [is phenomenal]. We can track checkouts on specific programs; we can see how often [programs are] changing; we can see how many assignments we deployed on a daily basis over two years. So the insights from that data are invaluable, and [zAdviser] just gave us such a better grasp on the way we act. We now actually have a clear view from a mainframe perspective what our delivery guidance is. We can see clearly how much time we spend in each of the different stages.”
  • The CIO in healthcare also said that using zAdviser reduced data collection and reporting times by 15%. They elaborated: “It’s an excellent tool. [It] provides actionable insight into your workflow [and] benchmarking against industry standards. [You can] look how you allocate your resources [and] how you can boost productivity overall. ... I’d say [it’s] a differentiator.”

• Upcoming artificial intelligence (AI) features may deliver even greater benefits. Several interviewees were excited about the AI enhancements for BMC AMI DevX tools that BMC was working on. At the time of the interviews, these features were not yet generally available. However, one of the interviewees had trialed some of the capabilities in a limited capacity, and a few of the other interviewees were aware of BMC’s roadmap. The PDM in financial services said: “I want to do Code Explain with AI. I want to be a beta tester for that bad boy. I want to get it in my shop and start using it. I’m there! Why? I’m leaving. When I walk out the door, 200 years of knowledge walks out the door with me. I want Code Explain in place. It has got to be there.”

Flexibility would also be quantified when evaluated as part of a specific project (described in more detail in Total Economic Impact Approach).