Teams responsible for the quality of digital applications and workflows are under increasing pressure as cloud upgrades are more recurrent and the frequency of releases increases. Eggplant Test automates the testing of end-to-end user journeys, supporting the need for increased technology iteration. As a result, bugs are reduced, application and workflow users’ productivity improve, and testers are more productive. The quality of external applications also increases, resulting in better customer experiences.
Eggplant Test is a continuous testing automation tool that uses a model-based approach to combine linear directed test automation with automated exploratory testing. It enables teams working to ensure high-quality digital experiences to scale their testing and therefore also supports a higher frequency of iterations and releases.
Keysight commissioned Forrester Consulting to conduct a Total Economic Impact™ (TEI) study and examine the potential return on investment (ROI) enterprises may realize by deploying Eggplant Test.1 The purpose of this study is to provide readers with a framework to evaluate the potential financial impact of Eggplant Test on their organizations.
To better understand the benefits, costs, and risks associated with this investment, Forrester interviewed two representatives with experience using Keysight’s Eggplant automation platform. For the purposes of this study, Forrester aggregated the interviewees’ experiences and combined the results into a single composite organization that is a services organization with 15,000 employees and multiple billions of dollars in revenue.
Prior to using Eggplant Test, these interviewees noted how their organizations were struggling to cover the increasing demands of application testing. Adding more manual testers was not viable or cost-efficient, while the legacy tools they were using were not sufficiently adopted or did not provide the coverage to capture many bugs and defects.
After the investment in Eggplant Test, the interviewees were able to increase the scale of testing significantly. Key results from the investment include improved productivity of application users, reduced bug remediation effort and manual tester efficiencies.
Key Findings
Quantified benefits. Three-year, risk-adjusted present value (PV) quantified benefits for the composite organization include:
- Application user productivity improvement amounting to $4.2M. After the Eggplant investment, there are fewer defects and issues with applications; furthermore, the number of releases increased, so more new capabilities, fixes and functionality can be built into them. As a result, the productivity of those using applications increases; it is assumed this is equivalent to 2 hours annual time saving per user.
- Cost savings from remediations avoided come to $2.5M. The number of bugs released into the post-production environment is reduced by 80%. This saves application developers thousands of hours of bug remediation time.
- Manual tester productivity improvements amount to $737K. Manual testers are able to significantly increase their productivity, typically by 30-40%. Execution programs not only complete test cases in less time, but they also work 24 hours a day.
- Legacy tool cost avoidance of $167K. The interviewees shared that they were using ineffective tools prior to their Eggplant Test investment. By decommissioning these tools, they were able to avoid these license costs.
Unquantified benefits. Benefits that provide value for the composite organization but are not quantified in this study include:
- Improved customer experience. Many of the applications and end-to-end user journeys that Eggplant is used for are customer facing. By improving their quality and capabilities, customer experiences improve. Some of this impact is captured in the application user productivity benefit because interviewees were not able to estimate the indirect impact on customer metrics such as loyalty, conversion and order value.
- Improved employee experience. Testers and others in quality control and application development no longer need to undertake repetitive and uninteresting tasks. Improved employee experiences can result in improvements in related metrics such as higher productivity, reduced absenteeism and increased retention.
Costs. Three-year, risk-adjusted PV costs for the composite organization include:
- Licensing fees amounted to $783K. Eggplant’s pricing is structured around the number of execution and development licenses.
- Planning and implementation costs amount to $1.6M. This includes the preparation of the applications environments and end-to-end user journeys and the scripting of the test cases.
- Maintenance and incremental hardware costs of $167K. This covers the additional servers to run the execution programs and time needed for the maintenance of the test cases.
The representative interviews and financial analysis found that a composite organization experiences benefits of $7.58M over three years versus costs of $2.89M, adding up to a net present value (NPV) of $4.69M and an ROI of 162%.
-
Return on investment (ROI):
162%
-
Benefits PV:
$7.58M
-
Net present value (NPV):
$4.69M
-
Payback:
9 months
Benefits (Three-Year)
Improved application user productivity Cost savings from avoided remediation Increased manual tester productivity Avoided alternative tool costTEI Framework And Methodology
From the information provided in the interviews, Forrester constructed a Total Economic Impact™ framework for those organizations considering an investment in Eggplant Test.
The objective of the framework is to identify the cost, benefit, flexibility, and risk factors that affect the investment decision. Forrester took a multistep approach to evaluate the impact that Automation Platform can have on an organization.
-
Due Diligence
Interviewed Keysight stakeholders and Forrester analysts to gather data relative to Automation Platform.
-
Interviews
Interviewed two representatives at organizations using Eggplant Test to obtain data with respect to costs, benefits, and risks.
-
Composite Organization
Designed a composite organization based on characteristics of the interviewees’ organizations.
-
Financial Model Framework
Constructed a financial model representative of the interviews using the TEI methodology and risk-adjusted the financial model based on issues and concerns of the interviewees.
-
Case Study
Employed four fundamental elements of TEI in modeling the investment impact: benefits, costs, flexibility, and risks. Given the increasing sophistication of ROI analyses related to IT investments, Forrester’s TEI methodology provides a complete picture of the total economic impact of purchase decisions. Please see Appendix A for additional information on the TEI methodology.
Disclosures
Readers should be aware of the following:
This study is commissioned by Keysight and delivered by Forrester Consulting. It is not meant to be used as a competitive analysis. Forrester makes no assumptions as to the potential ROI that other organizations will receive.
Forrester strongly advises that readers use their own estimates within the framework provided in the study to determine the appropriateness of an investment in Automation Platform.
Keysight reviewed and provided feedback to Forrester, but Forrester maintains editorial control over the study and its findings and does not accept changes to the study that contradict Forrester’s findings or obscure the meaning of the study.
Keysight provided the customer names for the interviews but did not participate in the interviews.
Consulting Team: Jan Sythoff
Drivers leading to the Eggplant Test investment
Interviews
| Role | Industry | Region | Number of employees |
|---|---|---|---|
| IT manager | Retail | North America | 270,000 |
| Chief development officer | Financial services | North America | 2,300 |
Key Challenges
The interviewees shared that there was growing pressure on their quality assurance and testing teams. The need to iterate faster required more and more testing time, while budgets were also under pressure.
The interviewees noted how their organizations struggled with common challenges, including:
- Pressure to increase the number of releases. With more applications sitting in the cloud and enterprises depending more on digital technologies, they are improved and upgraded more and more frequently. But in order to keep up with more releases, more testing is also required. Not only was the option of adding more human testing resources not possible because of budgetary pressure, but it was also not always viable as human testers cannot work fast or long enough.
- Difficulties in maintaining and improving application and workflow quality. Interviewees shared that they were struggling to maintain the level of testing because of the increasing number of releases. The CDO in retail said: “We had a lot of times when it was just too much, and we were having to skip some releases.” If bugs and defects get through to post-production environments, not only does this impact the applications users (both internal and external), but it also takes more time and effort to fix them than in a pre-production environment.
Composite Organization
Based on the interviews, Forrester constructed a TEI framework, a composite company, and an ROI analysis that illustrates the areas financially affected. The composite organization is representative of the two interviewees, and it is used to present the aggregate financial analysis in the next section. The composite organization has the following characteristics:
Description of composite. A large, global, multibillion dollar enterprise in the services industry with a total of 15,000 employees. It has 10 key applications which require testing when they are updated, prior to being released to ensure high quality and bug detection and elimination. As more of the applications have moved to the cloud, and are updated more regularly, there was increasing pressure on the testing team.
Deployment characteristics. It had 12 manual testers covering these applications, but they were no longer able to test across all releases. In the initial period, all 10 application environments were prepared for automation testing. Within each application, there are an average of 150 end-to-end user journeys that need to be tested, and scripts were written for 85% of these at this time. For each end-to-end user journey, there are an average of 10 test cases. Therefore initially, 1,275 test cases were prepared, which increases to 1,350 at the end of year 1 (equivalent to 90% of applications and end-to-end user journeys), and 1,425 in year 2 (equivalent to 95% of applications and end-to-end user journeys).
Furthermore, the number of releases increased from 5 prior to the investment, which increased to 20 in year 1, 24 in year 2 and 30 in year 3.
Quantified benefit data as applied to the composite
Total Benefits
| Ref. | Benefit | Year 1 | Year 2 | Year 3 | Total | Present Value |
|---|---|---|---|---|---|---|
| Atr | Improved application user productivity | $1,606,500 | $1,701,000 | $1,795,500 | $5,103,000 | $4,215,225 |
| Btr | Cost savings from avoided remediation | $765,000 | $972,000 | $1,282,500 | $3,019,500 | $2,462,322 |
| Ctr | Increased manual tester productivity | $256,500 | $299,250 | $342,000 | $897,750 | $737,446 |
| Dtr | Avoided alternative tool cost | $67,500 | $67,500 | $67,500 | $202,500 | $167,863 |
| Total benefits (risk-adjusted) | $2,695,500 | $3,039,750 | $3,487,500 | $9,222,750 | $7,582,856 |
Improved Application User Productivity
Evidence and data. The largest benefit the interviewees identified was the improved quality of the applications. This is not only because the number of bugs and defects was significantly reduced (see next benefit), but also because the number of releases increased significantly, so new features, functions and capabilities could be added.
- The CDO in financial services told us: “We catch bugs earlier in the cycle so that we don't have to catch them in final QA. And so, the number of defects that still remain in the system by the time it gets to actual people is much, much smaller.”
- Both interviewees emphasized the improvement in quality enabled by Keysight Eggplant.
- The CDO in financial services also said that they are now able to increase the number of releases they perform: “We were struggling to complete our tests every 4 weeks of the release cycle. Now we are confidently able to do that every 2 weeks.”
Modeling and assumptions. The impact of this benefit has been quantified by estimating the time savings delivered by the improved quality of the applications used by employees. These quality improvements also pertain to customer facing applications, but for the sake of simplicity and the difficulty for interviewees in monitoring such impacts, the customer impact has been incorporated into this one benefit.
- The composite is able to automate the testing of 8.5 applications in the first year, growing to 9 in year 2 and 9.5 in year 3.
- On average, every application is used by a third of employees.
- For each application covered, it is assumed that an average user saves 2 hours per year. This is because there are less bugs and because improvements and new features can be added that otherwise would not have been.
- The average employee has a fully loaded salary of $75,000, equivalent to $42 (rounded) on an hourly basis.
Risks. It is possible that the impact of this benefit could be lower, for an organization like the composite, if the productivity impact is lower, if the average salary level were lower or if the average number of users per applications were lower.
Results. To account for these risks, Forrester adjusted this benefit downward by 10%, yielding a three-year, risk-adjusted total PV of $4,215,225.
Improved Application User Productivity
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| A1 | Total number of applications covered by Eggplant | Composite | 8.5 | 9.0 | 9.5 | |
| A2 | Average number of users per application (assume 1/3 employees) | Assumption | 5,000 | 5,000 | 5,000 | |
| A3 | Productivity benefit per application per user (hours) | TEI Assumption | 2.0 | 2.0 | 2.0 | |
| A4 | Total number of hours saved | A1*A2*A3 | 85,000 | 90,000 | 95,000 | |
| A5 | Average hourly user salary rate (based on $75,000 fully loaded average) | Composite | $42 | $42 | $42 | |
| A6 | Productivity conversion rate | TEI standard | 50% | 50% | 50% | |
| At | Improved application user productivity | A4*A5*A6 | $1,785,000 | $1,890,000 | $1,995,000 | |
| Risk adjustment | ↓10% | |||||
| Atr | Improved application user productivity (risk-adjusted) | $1,606,500 | $1,701,000 | $1,795,500 | ||
| Three-year total: $5,103,000 | Three-year present value: $4,215,225 | |||||
Cost Savings From Avoided Remediation
Evidence and data. The avoidance of remediation on bugs and defects is an important benefit. Interviewees shared that because they were able to cover much more in terms of testing, with Eggplant, they caught bugs and defects at a much earlier stage and so the number of bugs reaching post-production was much reduced.
- The CDO in financial services said: “We know that the vast majority of bugs have already probably been got caught by the exploratory automation.”
Modeling and assumptions. In order to quantify this benefit, the following assumptions were made:
- For every application, there are a total of 150 end-to-end user journeys, although this will vary from one application to another.
- The baseline number of bugs was 2 per end-to-end user journey per release.
- The average bug-fixing time is 15 minutes. The time savings in bug remediation are particularly important in post-production environments, where it is more time consuming and costly compared to pre-production.
- The number of bugs was reduced by 80%.
- The fully-loaded salary of a developer is $150,000.
Risks. The remediation time avoided could be lower for an enterprise like the composite if:
- The baseline number of bugs (both pre- and post-production) was initially lower.
- The bug fix-time (both pre- and post-production) was already lower prior to the Eggplant investment.
- The developer salary rates were lower.
Results. To account for these risks, Forrester adjusted this benefit downward by 10%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $2,462,322.
Cost Savings From Avoided Remediation
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| B1 | Number of applications covered | Composite | 8.5 | 9.0 | 9.5 | |
| B2 | Number of end-to-end user journeys covered by Eggplant | B1*150 | 1,275 | 1,350 | 1,425 | |
| B3 | Number of releases | Interviews | 20 | 24 | 30 | |
| B4 | Baseline number of bugs per end-to-end user journey per release | Assumption | 2 | 2 | 2 | |
| B5 | Percentage of bugs avoided | Interviews | 80% | 80% | 80% | |
| B6 | Reduction in number of bugs | B2*B3*B4*B5 | 40,800 | 51,840 | 68,400 | |
| B7 | Average E2E user journey bug-fix time (hours) | Assume 15 minutes | 0.25 | 0.25 | 0.25 | |
| B8 | Total amount of time saved in fixing bugs (hours) | B6*B7 | 10,200 | 12,960 | 17,100 | |
| B9 | Hourly rate of application developers ($150,000 fully loaded salary) | TEI standard | $83 | $83 | $83 | |
| Bt | Cost savings from avoided remediation | B8*B9 | $850,000 | $1,080,000 | $1,425,000 | |
| Risk adjustment | ↓10% | |||||
| Btr | Cost savings from avoided remediation (risk-adjusted) | $765,000 | $972,000 | $1,282,500 | ||
| Three-year total: $3,019,500 | Three-year present value: $2,462,322 | |||||
Increased Manual Tester Productivity
Evidence and data. Both the interviewees shared that manual tester time was freed up following the Eggplant implementation. The extent of this benefit depends on how much additional testing could and should be done versus the cost reductions that could be an alternative priority.
- The IT manager in retail said: “what previously would have taken 3 weeks, we are now able to complete in less than a week.”
- The IT manager in retail also told us: “On average, manual testing took 20 minutes per test case. This is only 12 minutes through automation, a 40% reduction.”
Modeling and assumptions. The following assumptions were made to quantify this benefit:
- The composite is assumed to have 12 manual testers initially.
- The number of releases increases to 20 in Year 1, 24 in Year 2 and 30 in Year 3. Other organizations can choose to increase the manual tester time savings, but not increase the release frequency so much.
- The manual tester effort reduces by 30% in Year 1, 35% in Year 2 and 40% in Year 3.
- The average manual tester fully loaded salary is $100,000.
- A 75% productivity conversion rate is applied, given that not all time freed up is necessarily put back into manual testing tasks.
Risks. It is possible that the impact of this benefit could be lower in an organization like the composite if the manual testers were already very efficient and productive. Manual tester salaries may also be lower, or the productivity capture rate could be lower. Also note that the manual tester productivity needs to be balanced with the number of releases.
- The manual testers were already very efficient and productive.
- Manual tester salaries were lower.
- The productivity capture rate was lower. Note that the manual tester productivity needs to be balanced with the number of releases.
Results. To account for this risk, Forrester adjusted this benefit downward by 5%, yielding a three-year, risk-adjusted total PV of $737,446.
Increased Manual Tester Productivity
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| C1 | Baseline number of manual testers | Composite | 12 | 12 | 12 | |
| C2 | Reduction in manual tester effort required | Composite | 30% | 35% | 40% | |
| C3 | Reduced manual tester effort required | C1*C2 | 3.6 | 4.2 | 4.8 | |
| C4 | Manual tester salary (fully loaded) | $100,000 | $100,000 | $100,000 | $100,000 | |
| C5 | Productivity capture rate | TEI standard | 75% | 75% | 75% | |
| Ct | Increased manual tester productivity | C3*C4*C5 | $270,000 | $315,000 | $360,000 | |
| Risk adjustment | ↓5% | |||||
| Ctr | Increased manual tester productivity (risk-adjusted) | $256,500 | $299,250 | $342,000 | ||
| Three-year total: $897,750 | Three-year present value: $737,446 | |||||
Avoided Alternative Tool Cost
Evidence and data. Both the interviewees mentioned that they were using other tools prior to investing into Keysight Eggplant, but they were not effective:
- The IT Manager in retail said that: “the previous tool was not widely used because it did not have the capabilities we needed.”
- The CDO in financial services told Forrester that: “The previous tools didn't cover nearly enough and so that would basically allow more defects to sneak in, which we would then ultimately catch later on.”
Modeling and assumptions. The following assumptions were made to quantify this benefit:
- It is assumed that the composite was using a single tool previously, which it was able to fully decommission.
- The annual cost of the tool was $75,000.
Risks. It is possible that a tool of lower cost was used previously by an enterprise like the composite.
Results. To account for these risks, Forrester adjusted this benefit downward by 10%, yielding a three-year, risk-adjusted total PV of $167,863.
Avoided Alternative Tool Cost
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| D1 | Licensing cost of alternative tool | $75,000 | $75,000 | $75,000 | ||
| Dt | Avoided alternative tool cost | D1 | $75,000 | $75,000 | $75,000 | |
| Risk adjustment | ↓10% | |||||
| Dtr | Avoided alternative tool cost (risk-adjusted) | $67,500 | $67,500 | $67,500 | ||
| Three-year total: $202,500 | Three-year present value: $167,863 | |||||
Unquantified Benefits
Interviewees mentioned the following additional benefits that their organizations experienced but were not able to quantify:
- Improved customer experience. Some of the applications and workflows for which Eggplant was used were customer facing. With release cycles increasing and quality control coverage improved, the customer experience also improved, not only because there were fewer bugs and issues in the applications, but also because new features and functions could be added sooner. Customer related benefits have been captured in the first benefit - improved productivity of application users - but in this case the users are the customers, and the benefits would have been customer metrics such as higher conversion, improved loyalty and increased spend. However, it was not possible for the interviewees to quantify these metrics.
- Improved employee experience. The automation of manual testing processes has a direct positive impact on those in the quality control and testing team. These tasks are often very repetitive and uninteresting, and by reducing them, makes the experience of employees working on them much better. This can result in improvements in related metrics such as higher productivity, reduced absenteeism and increased retention.
Flexibility
The value of flexibility is unique to each customer. There are multiple scenarios in which a customer might implement Automation Platform and later realize additional uses and business opportunities, including:
- Multiple workflows supported. One of the benefits highlighted by the interviews was the flexibility of the Eggplant platform, which accommodates workflows where scanners, self-service kiosks and cash registers and other hardware need to be accommodated.
Quantified cost data as applied to the composite
Total Costs
| Ref. | Cost | Initial | Year 1 | Year 2 | Year 3 | Total | Present Value |
|---|---|---|---|---|---|---|---|
| Etr | Licensing fees | $0 | $294,000 | $315,000 | $341,250 | $950,250 | $783,989 |
| Ftr | Planning and implementation costs | $1,508,122 | $46,970 | $46,970 | $0 | $1,602,062 | $1,589,640 |
| Gtr | Maintenance and incremental hardware costs | $0 | $205,905 | $204,750 | $218,505 | $629,160 | $520,567 |
| Total costs (risk-adjusted) | $1,508,122 | $546,875 | $566,720 | $559,755 | $3,181,472 | $2,894,196 |
Licensing Fees
Evidence and data. Eggplant fees are broken down into development and execution licenses. The per license costs adjust depending on the total size of the contract. The interviewees shared that they automated the bulk of the test cases upfront, with the help of Eggplant to accelerate this process. However, additional automations were built every year.
Modeling and assumptions. For an organization like the composite, in the first year, the licensing fees come to $280,000, growing to $300,000 in year 2 and further to $325,000 in year 3. The composite continues to automate additional test cases, requiring more execution licenses year on year.
- The licensing fees come to $280,000 in Year 1, grows to $300,000 in Year 2, and further to $325,000 in Year 3.
- The composite continues to automate additional test cases, requiring more execution licenses year on year.
Risks. There is a risk that the licensing fees could be higher for an organization the size of the composite, given that prices can change over time.
Results. To account for these risks, Forrester adjusted this cost upward by 5%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $783,989.
Licensing Fees
| Ref. | Metric | Source | Initial | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|---|
| E1 | Licensing fees | Composite | $0 | $280,000 | $300,000 | $325,000 | |
| Et | Licensing fees | E1 | $0 | $280,000 | $300,000 | $325,000 | |
| Risk adjustment | ↑5% | ||||||
| Etr | Licensing fees (risk-adjusted) | $0 | $294,000 | $315,000 | $341,250 | ||
| Three-year total: $950,250 | Three-year present value: $783,989 | ||||||
Planning And Implementation Costs
Evidence and data. The interviewees shared that there were a number of steps taken in order to plan and implement Keysight Eggplant. The first was to prepare the applications environments; the second was to prepare the end-to-end user journeys within each of the applications and thirdly to write the scripts for each of the test cases.
Modeling and assumptions. In the case of the composite, the majority of the planning and implementation effort takes place during the initial period, with incremental test cases built in the following years. To quantify the costs, it was assumed that:
- There are ten application environments that need to be prepared; each takes twelve FTEs twelve full days (96 hours).
- There are, on average, 150 end-to-end user journeys per application, and of those covered by Eggplant, each requires 10 minutes to prepare.
- Each test case takes, on average, an hour to create and there are 10 test cases per end-to-end user journey on average.
- The FTEs preparing the application environments, user journeys and test cases are on a fully loaded salary of $100,000.
Risks. For an organization like the composite, it is possible that the planning and implementation time takes longer, if, for instance, these is a different mix of applications and/ or the end-to-end user journeys are longer or more complicated.
Results. To account for these risks, Forrester adjusted this cost upward by 10%, yielding a three-year, risk-adjusted total PV of $1,589,640.
Planning And Implementation Costs
| Ref. | Metric | Source | Initial | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|---|
| F1 | Number of applications | Composite | 10 | ||||
| F2 | Number of FTEs | Composite | 12 | ||||
| F3 | Build, deploy and training time per application (hours per FTE) | Interviews | 96 | ||||
| F4 | Total application environment preparation time (hours) | F1*F2*F3 | 11,520 | ||||
| F5 | Total number of user journeys (assume 150 per application) | F1*150 | 1,500 | 1,500 | 1,500 | 1,500 | |
| F6 | Portion of user journeys covered by Eggplant | Composite | 85% | 90% | 95% | 95% | |
| F7 | Number of user journeys covered by Eggplant | F5*F6 | 1,275 | 1,350 | 1,425 | 1,425 | |
| F8 | Number of new user journeys covered by Eggplant | F7(yrn-1)-(yrn) | 1,275 | 75 | 75 | 0 | |
| F9 | Time spent building E2E user journey test scripts (hours) | F8*(10/60) | 213 | 13 | 13 | 0 | |
| F10 | Number of new test cases (average 10 per user journey) | 10*F8 | 12,750 | 750 | 750 | 0 | |
| F11 | Total time spent preparing test cases (1 hour per test case) | F10 | 12,750 | 750 | 750 | 0 | |
| F12 | Total planning, implementation and maintenance time | F4+F9+F11 | 24,483 | 763 | 763 | 0 | |
| F13 | FTE hourly rate (based on fully loaded salary of $100,000) | TEI Standard | $56 | $56 | $56 | $56 | |
| Ft | Planning and implementation costs | F12*F13 | $1,371,020 | $42,700 | $42,700 | $0 | |
| Risk adjustment | ↑10% | ||||||
| Ftr | Planning and implementation costs (risk-adjusted) | $1,508,122 | $46,970 | $46,970 | $0 | ||
| Three-year total: $1,602,062 | Three-year present value: $1,589,640 | ||||||
Maintenance And Incremental Hardware Costs
Evidence and data. There were two ongoing costs that interviewees described: the additional servers required to run the robots (which also had to be maintained) and the maintenance of the test cases.
Modeling and assumptions. For the amount of test cases automated, the composite needs 8 robots (or execution licenses) in year 1, another 2 in year 2 and another 3 in year 3. Each requires a single server, each of which is assumed to cost $2,000; the maintenance of the servers is assumed to be 10% of this hardware cost.
- For the amount of test cases that were automated, the composite needs eight robots (or execution licenses) in Year 1, another two in Year 2, and another three in Year 3.
- Each robot requires a single server, and each is assumed to cost $2,000.
- The maintenance of the servers is assumed to be 10% of this hardware cost.
- For the test case upkeep, it is assumed that each requires 15 minutes of maintenance annually.
Risks. There is a small risk that the hardware and maintenance costs could be higher if the server acquisition cost is higher or the average time per test case maintenance is longer.
Results. To account for these risks, Forrester adjusted this cost upward by 5%, yielding a three-year, risk-adjusted total PV of $520,567.
Maintenance And Incremental Hardware Costs
| Ref. | Metric | Source | Initial | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|---|
| G1 | Number of robots | Composite | 0 | 8 | 10 | 13 | |
| G2 | Number of new servers (1 per robot) | 1 server per robot | 0 | 8 | 2 | 3 | |
| G3 | Server acquisition cost ($2,000 each) | G2*$2,000 | $0 | $16,000 | $4,000 | $6,000 | |
| G4 | Server maintenance costs | 10% of cumulative hardware cost | $0 | $1,600 | $2,000 | $2,600 | |
| G5 | Total incremental hardware costs | G3+G4 | $0 | $17,600 | $6,000 | $8,600 | |
| G6 | Number of test cases covered by Eggplant | F10 cumulative | 0 | 12,750 | 13,500 | 14,250 | |
| G7 | Total test case maintenance time (average 15 minutes per test case) | G6*15/60 | 0 | 3,188 | 3,375 | 3,563 | |
| G8 | FTE hourly rate (based on fully loaded salary of $100,000) | TEI standard | $0 | $56 | $56 | $56 | |
| G9 | Total test case maintenance costs | G7*G8 | $0 | $178,500 | $189,000 | $199,500 | |
| Gt | Maintenance and incremental hardware costs | G5+G9 | $0 | $196,100 | $195,000 | $208,100 | |
| Risk adjustment | ↑5% | ||||||
| Gtr | Maintenance and incremental hardware costs (risk-adjusted) | $0 | $205,905 | $204,750 | $218,505 | ||
| Three-year total: $629,160 | Three-year present value: $520,567 | ||||||
Consolidated Three-Year Risk-Adjusted Metrics
The financial results calculated in the Benefits and Costs sections can be used to determine the ROI, NPV, and payback period for the composite organization’s investment. Forrester assumes a yearly discount rate of 10% for this analysis.
-
These risk-adjusted ROI, NPV, and payback period values are determined by applying risk-adjustment factors to the unadjusted results in each Benefit and Cost section.
Cash Flow Chart
Total costs Total benefits Cumulative net benefitsCash Flow Analysis (Risk-Adjusted Estimates)
| Initial | Year 1 | Year 2 | Year 3 | Total | Present Value | |
|---|---|---|---|---|---|---|
| Total costs | ($1,508,122) | ($546,875) | ($566,720) | ($559,755) | ($3,181,472) | ($2,894,196) |
| Total benefits | $0 | $2,695,500 | $3,039,750 | $3,487,500 | $9,222,750 | $7,582,856 |
| Net benefits | ($1,508,122) | $2,148,625 | $2,473,030 | $2,927,745 | $6,041,278 | $4,688,660 |
| ROI | 162% | |||||
| Payback period (months) | 9.0 |
Appendix A: Total Economic Impact
Total Economic Impact is a methodology developed by Forrester Research that enhances a company’s technology decision-making processes and assists vendors in communicating the value proposition of their products and services to clients. The TEI methodology helps companies demonstrate, justify, and realize the tangible value of IT initiatives to both senior management and other key business stakeholders.
Total Economic Impact Approach
-
Benefits represent the value delivered to the business by the product. The TEI methodology places equal weight on the measure of benefits and the measure of costs, allowing for a full examination of the effect of the technology on the entire organization.
-
Costs consider all expenses necessary to deliver the proposed value, or benefits, of the product. The cost category within TEI captures incremental costs over the existing environment for ongoing costs associated with the solution.
-
Flexibility represents the strategic value that can be obtained for some future additional investment building on top of the initial investment already made. Having the ability to capture that benefit has a PV that can be estimated.
-
Risks measure the uncertainty of benefit and cost estimates given: 1) the likelihood that estimates will meet original projections and 2) the likelihood that estimates will be tracked over time. TEI risk factors are based on “triangular distribution.”
The initial investment column contains costs incurred at “time 0” or at the beginning of Year 1 that are not discounted. All other cash flows are discounted using the discount rate at the end of the year. PV calculations are calculated for each total cost and benefit estimate. NPV calculations in the summary tables are the sum of the initial investment and the discounted cash flows in each year. Sums and present value calculations of the Total Benefits, Total Costs, and Cash Flow tables may not exactly add up, as some rounding may occur.
Appendix B: Supplemental Material
Related Forrester Research
“The 12 Must-Dos For Achieving Continuous Software Testing,” Forrester Research, Inc., June 28, 2023
Appendix C: Endnotes
1 Total Economic Impact is a methodology developed by Forrester Research that enhances a company’s technology decision-making processes and assists vendors in communicating the value proposition of their products and services to clients. The TEI methodology helps companies demonstrate, justify, and realize the tangible value of IT initiatives to both senior management and other key business stakeholders.
PDF