SAP Change Impact Analysis By Tricentis: Financial Gains From Reduced Testing Scope And Hypercare Costs

Evidence and data. SAP Change Impact Analysis by Tricentis enables decision-makers to analyze the effect of potential software changes, and allow them to focus testing on the business and technical risks posed by those changes. With this focus, interviewees noted that their organizations were able to reduce the costs associated with both testing and hypercare. Before deployment, interviewees described testing processes that were operating blindly regarding risk. Their organizations had no clear understanding of the risks posed by potential software changes, which led them to utilize a high number of test conditions per application tested, hoping to reduce the statistical likelihood of production errors. Of course, such practices led to high testing costs.

After implementing SAP Change Impact Analysis by Tricentis, the interviewees noted their organizations were able to move away from testing a high number of random conditions and begin targeting conditions that carried risk. For example, the data analytics transformation lead from the telecommunications organization shared that they were able to reduce test conditions from 20,000 to 1,000, and then further down to 800 test conditions per release without increasing the number of defects that made it into production. This added focus on risk also enabled their organization to save on hypercare. For those organizations that conducted hypercare, their representatives shared that of the total 90% reduction in hypercare costs from SAP enterprise application QA solution by Tricentis as a whole, SAP Change Impact Analysis by Tricentis contributed 50% (net 45% contribution).

Interviewees also noted that SAP Change Impact Analysis by Tricentis was vital to getting the full benefit of automation and more frequent releases associated with their investment in SAP Enterprise Continuous Testing by Tricentis. Without utilizing the former, interviewees felt the conservative nature of their organizations might prevent the approval of their now-faster release cycles due to fears of an increase in production errors.

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• A total of 20 testing professionals engages in testing work, and another 20 engages in hypercare.

• The prior number of annual application releases was two.

• It needs six days to test 25,000 test conditions per release cycle.

• SAP Change Impact Analysis by Tricentis reduces test conditions by 84%.

• The fully burdened hourly rate of a testing professional is $53.

• Hypercare previously took eight full weeks per annum.

• SAP Change Impact Analysis by Tricentis reduces the length of hypercare by 45%.

Risks. The value gained from SAP Change Impact Analysis by Tricentis will vary with:

• The number of testing professionals.

• The current release rate for application tests.

• The time cost of testing and number of conditions tested per release.

• The current number of professionals and time dedicated to hypercare.

• The fully burdened rate of pay of a testing professional.

Results. To account for these risks, Forrester adjusted this benefit downward by 10%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $533,000.

SAP Enterprise Continuous Testing By Tricentis: Cost Savings From Process Automation, Fewer Production Errors, And Decommissioned Technology

Evidence and data. SAP Enterprise Continuous Testing by Tricentis enables organizations to implement no-code, model-based, easy-to-maintain, end-to-end automation of application testing. By improving the automation of testing and quality assurance practices, interviewees shared that their organizations reduced the cost of test execution, accelerated the validation of changes, and reduced the risk of defects impacting their businesses.

The interviewees noted improved application testing automation rates after deploying SAP Enterprise Continuous Testing by Tricentis. For example, the test manager at the IT services business also reported a large increase in automation, moving from only manual testing with 10% to 20% test case coverage to 80% test case coverage with automation. The agile coach at the insurance organization also noted a sizable improvement to automation, raising average SAP automation rates from 10% to 50%. However, some teams achieved automation rates of 80% to 90%. Interviewees attributed these improvements to the solution’s no-code, model-based automation, which negates the need for technical skills to directly create automation, its single interface for automation across multiple platforms and systems, and its tight integration with SAP.

With the introduction of model-based, no-code automated testing, the interviewees said their organizations reduced their need for skilled developers to focus on coding and programing testing automation scripts. They then began to reallocate their skilled developer resources from testing teams to higher value and higher impact development work. The agile coach at the insurance organization noted, “We’ll reduce resources by 30%, but it will be much higher for some divisions.” Reducing the number of more expensive resources on the testing team also reduced overall testing team costs.

Automation also enabled the interviewees’ organizations to minimize errors or incidents that made it into production. Interviewees shared that they were able to limit bugs that made it into production by between 70% and 93%, depending on the type of bug and amount of automation. The data and analytics transformation lead at the telecommunications organization noted: “SAP Enterprise Continuous Testing by Tricentis allows us to catch bugs within days instead of weeks, reducing the added, higher costs of bug fixing once in production. … We modeled a bug and found that, for 10,000 customers, that billing bug would cost $350,000.” The SAP test manager from the energy company shared: “Finding a defect in code could delay a release. If we didn’t have automated regression test conditions, not much would get done to ensure bugs don’t go into production.” The senior test manager from the CPG organization reported, “We detect defects every 10 minutes.”

Interviewees who had invested in an automated application testing solution prior to SAP Enterprise Continuous Testing by Tricentis were able to decommission these solutions. In doing so, these firms saved an approximate $80,000 annually on licensing costs and reallocated the resource responsible for managing that solution to SAP enterprise application QA Solutions by Tricentis as a whole, incurring no net additional ongoing management costs.

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• SAP Enterprise Continuous Testing by Tricentis reduces testing work by 33% via automation.

• Automation benefits accrue at a rate of 25% in Year 1, 50% in Year 2, and 100% by Year 3.

• On average, production errors cause a delay of eight hours and impact a conservative 200 employees.

• Employees recapture an estimated 25% of their productivity from a reduction in production errors.

• Every year, 10,000 customers are impacted by production errors with a cost to remediate of $30 per customer.

• Automation reduces production errors by 93%.

• Automation benefits accrue at a rate of 25% in Year 1, 50% in Year 2, and 100% by Year 3.

• The composite decommissions a legacy automated testing tool, eliminating the annual licensing cost of $80,000.

• Management costs of the decommissioned tool were equivalent to one tester annually at the time, which was $100,000.

Risks. The cost savings from SAP Enterprise Continuous Testing by Tricentis will vary with:

• The number of testing professionals and the rate at which automation enables their reallocation to higher value work.

• The number of employees impacted by production errors and their associated time costs.

• The rates at which these employees’ productivity can be recaptured by reducing production errors.

• The number of customers impacted by production errors and the associated remediation costs.

• The licensing and internal labor costs of any current testing solutions decommissioned in favor of SAP Enterprise Continuous Testing by Tricentis.

Results. To account for these risks, Forrester adjusted this benefit downward by 10%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $1.7 million.

SAP Enterprise Continuous Testing By Tricentis: Improved Business Value From Reduced Time-To-Release And Increased Application Release Rate

Evidence and data. The increase in automated testing from SAP Enterprise Continuous Testing by Tricentis not only reduced costs associated with manual testing and less efficient automated testing solutions, but also enabled the organizations to increase their rates of application releases. When testing professionals saw reduced testing times while simultaneously reducing production incidents, they made the case internally to move toward continuous integration and continuous deployment (CI/CD) practices, improving rates of release and increasing the business value of testing. The test manager of the IT services firm shared that they ran 15,000 test cases in 20 hours, while the senior director of engineering at the retail firm shared that they executed critical process testing daily, and regular testing weekly. Similarly, the agile coach at the insurance firm noted that they were releasing applications every two weeks, while the data and analytics transformation lead at the telecommunications firm noted that they were now updating their applications monthly instead of quarterly.

In terms of regression testing, the senior test manager from the CPG firm noted that their regression cycles were becoming more frequent, from weekly to daily. Meanwhile, the agile coach at the insurance firm noted their regression cycles moved from biweekly to daily. Similarly, the data and analytics transformation lead at the telecommunications firm noted a reduction from six weeks to less than two days in their regression cycles.

By improving their release rates, the interviewees were able to deliver necessary functionality updates to end users faster, improving the time-to-value of each release. The data and analytics transformation lead at the telecommunications organization explained: “More frequent updates mean the software becomes more reliable. New features can be accessed, old features can be accessed faster, features are enhanced to provide more value, etc. All of these increase our employee productivity, not to mention our internal Net Promoter Score℠ (NPS).²”

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• SAP Enterprise Continuous Testing by Tricentis reduces the time-to-release of applications by 83%, from six months to one month.

• Releases now occur five months faster.

• A conservative average of 800 employees is impacted by each release.

• Each release improves efficiency of these end users by a conservative 2.5%.

• The average fully burdened hourly rate for each impacted employee is $40.

• These employees are able to recapture 50% of the time efficiency productively.

Risks. The business value gained from SAP Enterprise Continuous Testing by Tricentis will vary with:

• The current application release cycle.

• The average efficiency improvement gained from each release.

• The number of employees impacted by application release wait times.

• The rates at which these employees’ productivity can be recaptured.

Results. To account for these risks, Forrester adjusted this benefit downward by 25%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $7.8 million.

SAP Enterprise Performance Testing By Tricentis: Cost Savings From Reduced Performance Testing Cycles And Hypercare Costs

Evidence and data. SAP Enterprise Performance Testing by Tricentis enables organizations to ensure that their applications will still meet performance, capacity, and usage requirements under heavy user loads. Interviewees noted that leveraging the solution enabled their organizations to save on costs associated with performance test maintenance, increase the frequency of performance test case execution, improve coverage of systems tested, and further reduce the costs of hypercare.

Interviewees shared that performance test automation enabled their organizations to reduce the time and cost of performance test maintenance. Interviewees noted that before adopting SAP Enterprise Performance Testing by Tricentis, they had to spend approximately 20 person-days on test maintenance for each quarterly test set execution. With the solution, the time-to-execute test maintenance duration was reduced to between 30 minutes and one hour, depending on the load, achieving a total reduction of between 99.4% and 99.7%.

As test set maintenance time decreased, organizations were able to execute their performance test cases more frequently. Before, organizations were executing performance tests once every quarter. After, these organizations were able to execute performance tests once or twice a week, increasing the rate of performance testing between 12.5 times and 25 times. Organizations improved their performance testing frequency while also improving the number of systems tested from one to seven.

By improving their test maintenance time and test frequency, organizations further reduced their costs of hypercare. In fact, the test manager from the IT services industry reported that the organization was able to eliminate hypercare entirely.

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• Prior to adopting SAP Enterprise Performance Testing by Tricentis, four performance tests were completed annually.

• A total of 17 person-days were needed to run each performance test before.

• The rate of performance testing increases by 12.5 times, enabling 50 performance tests to be completed annually.

• The time-to-test for performance reduces by 99%.

• The need for hypercare for performance-related application updates is reduced, avoiding an additional 45% of annual hypercare costs.

• The composite reduces a total of 90% of its prior hypercare costs after deploying both SAP Change Impact Analysis by Tricentis and SAP Enterprise Performance Testing by Tricentis.

Risks. The value gained from SAP Enterprise Performance Testing by Tricentis will vary with:

• The current rate of performance testing.

• The current time cost of performance tests.

• The time cost of hypercare.

• The fully burdened rate of professional testers.

Results. To account for these risks, Forrester adjusted this benefit downward by 10%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $567,000.

Unquantified Benefits

Interviewees mentioned the following additional benefits that their organizations experienced but were not able to quantify:

• Added value of AI. SAP enterprise application QA solutions by Tricentis leverage AI in unique ways to improve testing processes. The Vision AI feature enables the automation of test cases that are focused on UI changes that would normally not be prone to automation and usually require manual testing. AI-based self-healing capabilities automatically update tests when applications change — reducing test maintenance and preventing false positives. The adoption and usage of Vision AI and self-healing could further increase the benefits reported by the interviewees in relation to test automation rates, time to release, and test maintenance costs. The data analytics transformation lead from the telecommunications firm noted, “Self-healing test automation has reduced test maintenance and is keeping our suites stable.”

• Increased sales. While most interviewees noted that automated testing improved the efficiency of testing teams and the productivity of end users, the test manager from the IT services firm noted: “SAP Enterprise Performance Testing by Tricentis has made our client’s insurance software faster. Where their salespeople used to generate a contract in 10 minutes on average, the increase in performance testing has made the software more efficient. They now generate contracts in three minutes or less.”

• Enhanced tester satisfaction. The interviewees reported that employee satisfaction improved among testing teams after the introduction of SAP enterprise application quality assurance solutions by Tricentis. For example, the agile coach from the insurance industry shared: “We already see a lot of improvements from both a quality perspective and a satisfaction perspective. Our agile teams feel more in control of the end product now.” The data and analytics transformation lead from the telecommunications firm said: “Engineers get bored of manual tasks. Automated testing allows them to focus their energy and skills on more complex activities. It’s a big win for everyone.”

Flexibility

The value of flexibility is unique to each customer. There are multiple scenarios in which a customer might implement SAP enterprise application QA solutions by Tricentis and later realize additional uses and business opportunities, including:

• SAP S/4HANA migration. The interviewees noted particular benefits around leveraging SAP enterprise application QA solutions by Tricentis for their migration to SAP S/4HANA. While several interviewees noted that they planned to use the automated testing solutions for their organization’s transition so as to shorten lead times, the SAP test manager from the energy company reported: “We have loads of test cases that are SAP-specific — in order to cut down on testing all of those for the transition to SAP S/4HANA, we ran SAP Change Impact Analysis by Tricentis to compare the differences between the prior SAP GUI and the current GUI. We then only made the required changes that would support the SAP upgrade, rationalizing the amount of testing needed.”

• Organizational agility. The interviewees also noted that the improvement to release rates improved the rate at which new features and innovation could be deployed across the organization, increasing the rate of innovation adoption and making their organizations more agile. The data analytics transformation lead at the telecommunications organization said: “Faster releases have allowed us to respond more quickly to market shifts and regulatory changes. Teams are delivering updates and features to internal users and customers more frequently, improving agility and satisfaction.”

Flexibility would also be quantified when evaluated as part of a specific project (described in more detail in Total Economic Impact Approach).