The Total Economic Impact™ Of Xray
Cost Savings And Business Benefits Enabled By Xray
A Forrester Total Economic Impact™ Study Commissioned By Xray, March 2025
Organizations using agile development tools need equally powerful solutions to manage and optimize their software testing processes. Although these tools excel at issue tracking and project management, they often lack built-in features to handle complex testing scenarios, automate repetitive tests, and ensure thorough test coverage. A comprehensive testing tool that seamlessly integrates with a development platform can cover the entire testing lifecycle from planning to reporting with full traceability.
Xray is natively designed for Jira and provides a structured approach to managing test cases, executing tests, and tracking defects. It features full traceability and reporting essential for maintaining testing process agility and flexibility. Xray’s use of Jira-native issue types ensures a streamlined workflow with a familiar interface that aligns with agile development methodologies and fosters collaboration between cross-functional teams. Xray supports integrations with automation frameworks and continuous integration (CI) tools and facilitates automated testing within CI and continuous delivery (CD) pipelines. Features like exploratory testing, traceability between requirements and tests, and customizable reporting enhance quality assurance and collaboration across development and QA teams.
Xray commissioned Forrester Consulting to conduct a Total Economic Impact™ (TEI) study and examine the potential return on investment (ROI) enterprises may realize by deploying Xray.1 The purpose of this study is to provide readers with a framework to evaluate the potential financial impact of Xray on their organizations.
Return on investment (ROI)
226%
Net present value (NPV)
$4.87M
To better understand the benefits, costs, and risks associated with this investment, Forrester interviewed ten decision-makers in five organizations with experience using Xray. For the purposes of this study, Forrester aggregated the interviewees’ experiences and combined the results into a single composite organization that is a B2B company in a regulated industry with $2.5 billion in annual revenue and 10,000 employees in locations around the world. The organization recently adopted an agile development approach and holds 2,000 Jira licenses for project management.
Interviewees said that prior to using Xray, their organizations relied on outdated, unintegrated, on-premises test management tools that lacked features and were difficult to maintain. These legacy tools were poorly supported and had insufficient documentation, making them hard for new users to learn. High licensing costs led organizations to limit access to testing assets, leading to siloed teams and poor communication. The absence of integration between these tools and Jira also created synchronization problems, complicating and slowing workflows for agile teams.
After adopting and implementing Xray, the interviewees’ organizations consolidated all test management processes in Jira, enabling an agile yet disciplined development approach. Xray’s full traceability capabilities enabled testers to connect and relate issues more easily and effectively, shortening the time to repair production bugs. Xray’s intuitive dashboards and reports improved communication and streamlined processes, leading to better coordination and productivity across agile teams. The organizations also saved time and expenses after retiring inefficient legacy test management solutions.
Key Findings
Quantified benefits. Three-year, risk-adjusted present value (PV) quantified benefits for the composite organization include:
- Gaining $3.9 million in improved test management productivity over three years. Xray’s intuitive user interface and workflows streamline the composite organization’s testing processes, lighten administrative tasks, and improve communication for faster, more efficient issue resolution. Consequently, the composite organization can increase the number of test cases it manages without augmenting resources.
- Gaining $1.5 million in software development productivity over three years. Integrating Xray into Jira facilitates a more disciplined development approach, improves communication, and streamlines processes, leading to better coordination and productivity across agile teams and enabling the composite organization to scale development projects.
- Saving $1.3 million by reducing the mean time to repair production bugs. Xray makes it easier for the composite organization to identify and resolve incidents, reducing rework for developers and time to validate rework for testers.
- Saving $216,000 after retiring legacy test management solutions. After transitioning to Xray, the composite organization phases out its legacy test management solutions, saving on licensing costs and system upkeep.
Unquantified benefits. Benefits that provide value for the composite organization but are not quantified for this study include:
- Native integration into Jira. Xray’s seamless integration with Jira eliminates the need to switch tools, streamlines processes, and provides a complete view of the software development lifecycle within Jira.
- Comprehensive service and support. The composite organization makes good use of Xray’s service and product teams. Users especially appreciate the prompt issue resolution, ability to influence feature development, and documentation and training resources.
- Enhanced flexibility. The composite organization finds Xray easy to modify, and it can tailor implementation to its own standards.
- Easy integration. Integrating Xray with automation tools through the API layer is relatively easy, making it a powerful tool for managing manual and automated tests within a unified platform.
- Transparent reporting. The composite organization benefits from the full traceability provided by Xray, which links requirements to test executions and defects in a single platform. This traceability improves test coverage, standardizes acceptance criteria, and leads to better quality and easier compliance audits. Xray’s data extraction and report-building capabilities help the composite organization’s business users quickly know a project’s status.
- Faster development cycles. Streamlined communication between developers and testers contributes to faster development cycles, as do workflow changes to conduct testing at earlier stages in the development process.
- Ease of adoption. Because Xray works as a seamless extension of Jira, most of the composite organization’s users find it easy to understand and navigate. The organization also takes advantage of training and onboarding support provided by Xray Academy.
- Improved user experience. Users at the composite organization agree that the overall experience of using Xray is much better than what they had before.
Costs. Three-year, risk-adjusted PV costs for the composite organization include:
- Xray licensing. The composite organization starts with standard Xray Cloud licenses for its Jira users. It later migrates to Xray Enterprise Cloud to access additional features and premium support. These costs total $97,000 over three years.
- Planning, implementation, and ongoing management. Planning and implementation — including migrating existing projects — takes the composite organization three months to complete and involves four data architects working part time during that period. After the initial implementation, one Java administrator/project manager spends between 10% and 15% of their time maintaining Xray in the Jira environment, including onboarding new users, setting up new projects, and adding integrations. These costs total $123,000 over three years for the composite organization.
- Training and change management. The composite organization takes the opportunity to overhaul and update its testing processes. To instill the changes, testers and their managers receive 20 hours of initial training on Xray, supplemented in subsequent years with informal quarterly sessions to review updates and best practices. In the third year, they receive another 20 hours of training on Xray Enterprise’s advanced features and functionality. Software developers involved in testing receive 8 hours of initial training and participate in informal semiannual sessions focused on best practices. These activities cost the composite organization a total of $1.9 million over three years.
The representative interviews and financial analysis found that a composite organization experiences benefits of $7.02 million over three years versus costs of $2.15 million, adding up to a net present value (NPV) of $4.87 million and an ROI of 226%.
Key Statistics
-
Return on investment (ROI)
226%
-
Benefits PV
$7.02M
-
Net present value (NPV)
$4.87M
-
Payback
13 months
Benefits (Three-Year)
Improved test management productivity Improved software development productivity Value of faster mean time to repair Cost savings from retiring legacy test management solutions
TEI Framework And Methodology
From the information provided in the interviews, Forrester constructed a Total Economic Impact™ framework for those organizations considering an investment in Xray.
The objective of the framework is to identify the cost, benefit, flexibility, and risk factors that affect the investment decision. Forrester took a multistep approach to evaluate the impact that Xray can have on an organization.
-
Due Diligence
Interviewed Xray stakeholders and Forrester analysts to gather data relative to Xray.
-
Interviews
Interviewed ten people at five organizations using Xray to obtain data about costs, benefits, and risks.
-
Composite Organization
Designed a composite organization based on characteristics of the interviewees’ organizations.
-
Financial Model Framework
Constructed a financial model representative of the interviews using the TEI methodology and risk-adjusted the financial model based on issues and concerns of the interviewees.
-
Case Study
Employed four fundamental elements of TEI in modeling the investment impact: benefits, costs, flexibility, and risks. Given the increasing sophistication of ROI analyses related to IT investments, Forrester’s TEI methodology provides a complete picture of the total economic impact of purchase decisions. Please see Appendix A for additional information on the TEI methodology.
Disclosures
Readers should be aware of the following:
This study is commissioned by Xray and delivered by Forrester Consulting. It is not meant to be used as a competitive analysis.
Forrester makes no assumptions as to the potential ROI that other organizations will receive. Forrester strongly advises that readers use their own estimates within the framework provided in the study to determine the appropriateness of an investment in Xray.
Xray reviewed and provided feedback to Forrester, but Forrester maintains editorial control over the study and its findings and does not accept changes to the study that contradict Forrester’s findings or obscure the meaning of the study.
Xray provided customer names for the interviews but did not participate in the interviews.
Consulting Team:
Caro Giordano
Drivers leading to the Xray investment
Interviews
| Role | Industry | Region | Revenue | Users |
|---|---|---|---|---|
| Test guild lead | Financial services | Domestic, headquartered in Europe | <$1 billion | 2,000 Jira licenses; 20% use Xray |
| Release and
quality
assurance manager Release and quality assurance manager |
Energy |
Multinational, headquartered in Europe |
$1 billion to $25 billion |
2,000 Jira licenses; 25% use Xray |
| Chief control officer | Financial services | Multinational, headquartered in United Kingdom | $25 billion to $50 billion | 2,000+ Jira licenses; not all use Xray |
| Director of
technical product management Product manager of agile planning Product manager of agile test management |
Pharmaceutical |
Multinational, headquartered in United States |
$50 billion to $100 billion |
25,000 Jira licenses; not all use Xray |
| Test
manager Test manager |
Automotive | Multinational, headquartered in Europe | >$100 billion | 100,000 Jira licenses; 25% use Xray |
Key Challenges
Prior to implementing Xray, most of the interviewees’ organizations relied on outdated, unintegrated, on-premises test management tools that lacked features and were difficult to maintain. The interviewees noted how their organizations struggled with common challenges, including:
- High costs and poor support. Legacy tools were expensive and often had inadequate support due to a reliance on third-party partners. According to a release and quality assurance manager at an energy company: “[The legacy testing tool] was expensive, and the support was not that good because our contract with them went through [a third-party vendor]. Every time we had to open a ticket, [they] typically didn’t know how to answer, so issues took a long time to resolve.”
-
Limited access to testing assets. The high licensing costs of legacy tools restricted testing asset
access, preventing broader team involvement. QA and development teams worked
in isolation, leading to poor communication and a lack of transparency. A
release and quality assurance manager at an energy company explained: “Only
a limited number of people had access to testing assets due to the high cost
of licenses, which inhibited expanding access to all teams. QA and
development were working in silos, with no transparency between teams and
difficult communication. Additionally, many tests and executions were kept
in spreadsheets and documents, making it difficult to control test
assets.”
A test manager at an automotive company added, “Having [the legacy tool] separate from Jira meant already onboarded Jira users would need to gain new approval, access, and extensive training for a completely different tool, making it cost prohibitive to extend this tool to the entire organization; hence, only testers would have access to it, leaving them in a silo.”
The test guild lead at a financial services firm told Forrester that some of their development teams had already moved to Jira, leaving the QA team in isolation: “We started to feel that we were left out because testing was somewhere else. We were reporting from two different tools, so we decided to switch to Jira.” -
Integration, performance, and compatibility
problems. Legacy testing tools were not integrated
with Jira, creating synchronization problems and complicating the workflow
for agile teams. According to a test manager at an automotive company: “We
had a combination of nonintegrated, standalone tools for managing
requirements, tests, and defects. Maintaining synchronization between [the
legacy tools] and Jira was complex and consumed a lot of IT resources.
Having two separate ecosystems made requirements traceability challenging
and lacking. There were no centralized processes or workflows, no
centralized reporting or risk management. Tests were not related to
requirements.”
The director of technical product management at a pharmaceutical company told Forrester: “[The legacy tool] was very outdated; it had performance and compatibility issues and many other problems. This impacted both development cycle times and coordination between developers and testers. [The legacy tool] was used mainly by testers and approvers. Developers didn’t even want to connect into the system. They would, for example, need to install virtual machines just to access [the legacy tool]. It was difficult for them.”
In one interviewee’s case, migrating data centers risked losing data due to the old server’s infrastructure. The test guild lead at a financial services firm explained that they were told “the [legacy tool] was running on a very old server, and if we wanted to migrate it, it might break down and we’d lose all our data.” - Cumbersome UI and lack of flexibility. The interviewees noted that legacy tools’ user interfaces were difficult to navigate, causing frustration among daily users. These tools also lacked the necessary flexibility for effective data analysis, reporting, and agile project support. According to a release and quality assurance manager at an energy company: “[Our legacy tool] was difficult to use. It was not a user-friendly tool, especially with the adoption of agile work. It was not integrated with Jira, so the management capabilities were somewhat difficult for us, and it didn’t provide the flexibility I have today to look at data, get reports, measure what is going on, and set improvement actions.”
- Insufficient documentation. A release and quality assurance manager at an energy company noted their legacy tool lacked detailed documentation. Specifically, they said that extra information was unavailable online, which made it harder for new users to learn the legacy tool as quickly as needed for an agile environment.
Solution Requirements
The interviewees’ organizations recognized the need for a more sophisticated solution and searched for a test management tool that delivered:
-
Jira-native integration capabilities and
user-friendliness. Many of the interviewees’
organizations were already using or migrating to Jira and wanted a solution
that would seamlessly integrate with it. According to the director of
technical product management at a pharmaceutical company: “Jira was already
the backbone of the company in terms of development, so one of the criteria
was to have a solution that was very closely compatible with Jira.
Integration capabilities were important; not only integration with Jira, but
also various test automation tools.”
This interviewee added that for regulated industries such as pharmaceuticals, integration with compliance tools was critical. The interviewee explained: “All our testing tools and test management tools must be GxP validated. It’s a pretty lengthy process on our side, with 10 to 15 people involved.”
Interviewees liked that Xray was intuitive for Jira users, reducing the need for extensive training and making it easier to onboard new projects. According to the chief control officer at a financial services firm: “Integration, usability, and intuitiveness of the tool to the teams were key selection criteria. Because we don’t have a one-size-fits-all business, we had to make sure any tool we picked could be used by business analysts and technologists as well as business users. … Some of our teams have business analysts who are dedicated to the testing function and discipline. In other places, we have developers doing the testing. In some places, we recruit users or subject matter experts (SMEs) from the business to do most of the testing. When we tested with the SMEs and the business users, they found Xray the most intuitive to use. That’s where Xray excelled and tipped the scale.” - Strong support, commitment to collaboration, and cost effectiveness. Xray’s strong support and commitment to continuous improvement were important to the interviewees’ organizations, as was cost effectiveness. According to a test manager at an automotive company: “Xray was selected based on usability, seamless integration in Jira, and a commitment from Xray to collaborate on missing features. The investment was further justified by its low cost compared to the existing test management solution, although the advantage of Jira integration was valued higher than cost considerations.”
- Comprehensive features. Xray’s capabilities addressed organizations’ quality concerns and bridged gaps in their testing capabilities, which were crucial for maintaining production availability. According to the chief control officer at a financial services firm: “When we changed methodologies from waterfall to agile a few years ago, we built a very sophisticated home-grown tool for release management and approval of our testing capabilities, which impacted quality. There was a period of time when we had several defects that caused availability problems with key processes in the organization. Because there wasn’t a centralized tool, and the tool we had developed to manage the agile methodology didn’t address quality, we needed something to bridge that gap.”
-
Agile transformation. To
stay competitive, the interviewees’ organizations had adopted agile
methodologies, which drove the need to integrate testing and development
within the same ecosystem. According to the test guild lead at a financial
services firm: “One of the main requirements of the IT management team was
to centralize processes, ideally into a single, multifunctional tool.
Developers and stakeholders were using or migrating to Jira and wanted to
have one system for everything instead of a separate tool for
QA.”
A release and quality assurance manager at an energy company agreed: “Having both development and test management tools fully integrated, with all tests linked to requirements, was an absolute requirement. Without this, it was impossible to have a complete understanding of the software development lifecycle (SDLC) and have full traceability of the testing process.”
The interviewees’ organizations used a thorough evaluation process for a handful of tools and ran one or more pilots as a proof of concept before making their selection and rolling it out.
-
The chief control officer at a financial services firm said:
“We started several pilot projects in different areas of the organization to
make sure [the solution] would fit. We had several of our more advanced
teams — where testing was a core competency — evaluate it, build some test
cases, and run productivity analysis, including how much time it took to
utilize the tool and some of the benefits. It took nine to 11 months to go
through that analysis.”
The interviewee continued: “We looked at three other Jira plug-ins and other Jira-compatible tools. They were quite compelling. We started to do a proof of concept on all three, but we followed the proof of concept through to the end with Xray. We got to a point with the other two where we stopped because there was a lot of feedback from the teams about usability and integration capabilities. Xray’s API integrated the best with [our current automation servers and developer platforms]. There was also some strategy around creating CI/CD pipelines, and some of the other tools weren’t as committed as Jira. With Jira, we wanted something that would integrate to the future state.” - A test manager at an automotive company told Forrester: “Because Jira was new for most of the organization, we decided to do a six-month pilot phase with selected teams. The feedback from all of the pilots was positive about using Xray, so we offered it as an option for teams with smaller projects. After a year, we released it for all initiatives and teams, irrespective of size. Gradually, adoption grew such that by mid-2023, we announced that [the legacy tool] would be discontinued. For the last half year, Xray is the only test management tool used. Gradually moving off gave people time to migrate and spread out the training and onboarding process.”
- According to a release and quality assurance manager at an energy company: “The pilot phase provided an opportunity to clean up our processes — all the forms and fields. We had about 100 fields on defects in [the legacy tool]. Now we have 30 and only some of them are really mandatory. We had time to try the tool out in a testing environment. There were some differences from [the legacy tool], but we saw that this would be the tool for us.”
Composite Organization
Based on the interviews, Forrester constructed a TEI framework, a composite company, and an ROI analysis that illustrates the areas financially affected. The composite organization is representative of the interviewees’ organizations, and it is used to present the aggregate financial analysis in the next section. The composite organization has the following characteristics:
Description of composite. The composite organization is a B2B company in a regulated industry with $2.5 billion in annual revenue and 10,000 employees in locations around the world. The organization recently adopted an agile development approach and holds 2,000 Jira licenses for project management, including 400 testers and 1,200 developers who provide testing and quality status for all Jira users. Prior to implementing Xray, the organization relied on an outdated, unintegrated, on-premises test management tool that lacked features and was difficult to maintain.
Deployment characteristics. After piloting Xray, the organization takes three months to deploy the solution, migrate existing test cases, and train users on test management processes leveraging Xray.
Quantified benefit data as applied to the composite
Total Benefits
| Ref. | Benefit | Year 1 | Year 2 | Year 3 | Total | Present Value |
|---|---|---|---|---|---|---|
| Atr | Improved test management productivity | $716,040 | $1,790,100 | $2,386,800 | $4,892,940 | $3,923,605 |
| Btr | Improved software development productivity | $335,644 | $716,040 | $895,050 | $1,946,734 | $1,569,364 |
| Ctr | Value of faster mean time to repair | $231,660 | $494,208 | $926,640 | $1,652,508 | $1,315,235 |
| Dtr | Cost savings from retiring legacy test management solutions | $61,695 | $92,543 | $111,051 | $265,289 | $216,003 |
| Total benefits (risk-adjusted) | $1,345,039 | $3,092,891 | $4,319,541 | $8,757,471 | $7,024,207 | |
Improved Test Management Productivity
Evidence and data. Integrating Xray into Jira boosted test management productivity, enabling the interviewees’ organizations to increase the number of test cases managed without augmenting resources. Xray’s user-friendly interface streamlined testing processes, minimized administrative tasks, and enhanced communication for efficient issue resolution. Direct defect entry and requirement linking improved transparency, while simplified time tracking and multiproject capabilities enhanced efficiency. Insights from Jira Query Language functions and dashboards ensured the organizations could maintain high standards in test case creation and compliance. Overall, these improvements led to cleaner, more streamlined processes, saving testers time and increasing their productivity.
- The chief control officer at a financial services firm explained: “The time that it saved was all that going back and forth with different teams trying to identify the root cause and who caused the problem. When you have a constantly changing environment, everybody has a different perception of what may have caused the issue. Everybody knows their environment to good extent, but they don’t really know the integration between the data flows.”
-
A test manager at an automotive company told Forrester: “Before
the introduction of Xray, we had different tools for managing tests and
defects. Jira was also used for managing defects and requirements, so the
teams that worked with [the legacy tool] needed to synchronize defects
between two tools. That made it harder to have requirements coverage with
tests.”
The interviewee added: “Test coverage status of requirements is the single most important feature that we have with Xray. Before, we spent a lot of time maintaining synchronization between [the legacy tool] and Jira. It took a lot of resources because of the old setup. Now, users like the transparency they have. The overview and connection to requirements makes it easier to design test cases. Basically, everyone works with Jira and therefore are working the backlog in Jira and Xray for tests. It is almost seamless in that respect. It is more efficient for testers.”
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:
- The number of software testers and test managers at the composite organization remains constant as their work shifts to Xray following a three-month deployment period.
- These software testers and test managers spend 65% of their time on test management tasks.
- Xray immediately improves their productivity by 8%, which saves the composite organization 31,980 hours in Year 1.
- Productivity improves by 15% in Year 2 and 20% in Year 3, which saves the composite organization 186,550 hours over that time.
- Of the time saved, 50% is recaptured for work on other tasks and initiatives.
- Averaged together, the fully burdened annual salary for a software tester and test manager is $108,000.
Risks. Organizational differences that may impact test management productivity gains include the number and compensation of software testers and test managers and their prior state of efficiency. These time savings can be even greater for testers who leverage Xray features and functionality more proficiently.
Results. To account for these risks, Forrester adjusted this benefit downward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $3.9 million.
Improved Test Management Productivity
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| A1 | Software testers/managers | Composite | 400 | 400 | 400 | |
| A2 | Percentage of time spent on test management tasks | Composite | 65% | 65% | 65% | |
| A3 | Months Xray is deployed | Interviews | 9 | 12 | 12 | |
| A4 | Percentage of time saved attributable to Xray | Interviews | 8% | 15% | 20% | |
| A5 | Percentage of time recaptured | Forrester standard | 50% | 50% | 50% | |
| A6 | Fully burdened annual salary for a software tester/manager (blended) | Forrester standard | $108,000 | $108,000 | $108,000 | |
| At | Improved test management productivity | A1*A2*(A3/12)*A4*A5*A6 | $842,400 | $2,106,000 | $2,808,000 | |
| Risk adjustment | ↓15% | |||||
| Atr | Improved test management productivity (risk-adjusted) | $716,040 | $1,790,100 | $2,386,800 | ||
| Three-year total: $4,892,940 | Three-year present value: $3,923,605 | |||||
Improved Software Development Productivity
Evidence and data. The integration of Xray into Jira boosted software development productivity and enabled the interviewees’ organizations to scale their number of development projects. Centralized test management coupled with a comprehensive quality framework fostered continuous improvement and operational efficiency. Enhanced collaboration between testers and developers made defect management and communication more efficient. Overall, Xray integrating with Jira facilitated a more disciplined development approach, improved communication, and streamlined processes, leading to better coordination and productivity across agile teams.
- The chief control officer at a financial services firm summed up the benefits to their software development organization: “It enables a more disciplined approach to software development. It allows for better communication and better collaboration among teams and with the business. If somebody’s out, and you need to talk to your business partner about where things are, you can go and look rather than try to locate the version of the spreadsheet they were working on. You’re able to get on the same page with the business. Same thing with cross/multifunctional teams. Collaboration is easier because they can pass test plans back and forth. When they’re locked in, the other team can review and understand exactly what they’re testing to make sure that their downstream testing compliments what’s being tested upstream. We have very extensive testing for some of our applications.”
- The test guild lead at a financial services firm told Forrester: “It is much easier for developers to take a look at defects, for example. The administration takes some time but other than that, it is easy for them. It’s easier for them to add commands and prepare batches and communicate with testers.”
- The director of technical product management at a pharmaceutical company explained: “Previously, most developers didn’t even get access to [the legacy tool]. They just didn’t care. It was a black box for them.” Collaboration was a key benefit for developers according to this interviewee, who noted: “Most of all, they can participate. We are trying to promote collaboration. Previously, testing and test data were isolated from the rest of the project. They hired a group of testers who did the testing, but developers didn’t care in those cases. We’re trying to bring these two worlds together for them to really cooperate on the deliverables.”
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:
- The number of software developers at the composite organization remains constant as Xray becomes available to them following a three-month deployment period.
- These developers spend 65% of their time on development activities, 25% of which is spent on testing activities.
- Xray immediately improves these developers’ test-related productivity by 5%, which saves 14,991 hours in Year 1.
- Productivity improves by 8% in Year 2 and 10% in Year 3, which saves the composite organization 71,955 hours over that time.
- Of the time saved, 50% is recaptured for work on other tasks and initiatives.
- The average fully burdened annual salary for a software developer is $108,000.
Risks. Organizational differences that may impact software developer productivity gains include the number and compensation of software developers, the nature of the development environment, and their prior state of efficiency. These time savings can be even greater for developers who leverage Xray features and functionality more proficiently.
Results. To account for these risks, Forrester adjusted this benefit downward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $1.6 million.
Improved Software Development Productivity
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| B1 | Software developers involved in testing | Composite | 1,200 | 1,200 | 1,200 | |
| B2 | Percentage of time spent on software development | Composite | 65% | 65% | 65% | |
| B3 | Percentage of software development time spent on testing activities | Composite | 25% | 25% | 25% | |
| B4 | Months Xray is deployed | Interviews | 9 | 12 | 12 | |
| B5 | Percentage of time saved attributable to Xray | Interviews | 5% | 8% | 10% | |
| B6 | Percentage of time recaptured | Forrester standard | 50% | 50% | 50% | |
| B7 | Fully burdened annual salary for a software developer | Forrester standard | $108,000 | $108,000 | $108,000 | |
| Bt | Improved software development productivity | B1*B2*B3*(B4/12)*B5*B6*B7 | $394,875 | $842,400 | $1,053,000 | |
| Risk adjustment | ↓15% | |||||
| Btr | Improved software development productivity (risk-adjusted) | $335,644 | $716,040 | $895,050 | ||
| Three-year total: $1,946,734 | Three-year present value: $1,569,364 | |||||
Value Of Faster Mean Time To Repair
Evidence and data. Integrating Xray into Jira reduced the mean time to repair (MTTR) by simplifying incident identification and relation. Although the impact varied by team and application, the new processes decreased bug incidents overall. One interviewee also experienced enhanced deliverable quality crucial in regulatory environments. Broadly, Xray streamlined issue management and resolution, leading to reduced rework for developers and reduced time to validate rework for testers.
- According to the test guild lead at a financial services firm: “Before, it was more complicated to find out what an incident was connected to. Now it’s much easier to relate, and that helps us raise the quality of the result and the quality of the process. We have around 300 different applications, and there are specific issues with each of them. With our internal development, sometimes we are able to develop and put things into production fast, but in other cases, we’ll only update four times a year. It really depends on the application, but I would say that due to the transformation, the new processes, and also due to Jira, the number of bug incidents has decreased 20%.”
- The chief control officer at another financial services firm noted: “It didn’t make meeting deadlines any faster, but it made the quality coming up at that point in time better. Sometimes you’re dealing with new regulations where they are still confusing and regulators are giving inconsistent answers. You’re trying to develop capabilities based on changing requirements and you have to implement them by a certain date. It’s not always perfect, but we are more perfect than we would have been without it.”
- This interviewee continued: “The productivity gained was more about getting a fact-based resolution of what actually was the root cause. On a given incident or issue, that saves 5% to 10% on issue resolution time, because now you can go and quantify it and demonstrate exactly who did what, rather than having a perception of what they thought was done. That kind of discussion always gets convoluted because nobody knows exactly who did something that could have caused the issue. [Test Case Versioning] eliminated all of that and provided a more disciplined approach to testing. End to end, I think it saves us 4% to 5% on average.”
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:
- The number of software developers, testers, and test managers at the composite organization remains constant as Xray becomes available to them following a three-month deployment period.
- The developers spend 65% of their time on development activities, 15% of which is spent on rework related to production bug fixes.
- The testers and test managers spend 65% of their time on test management tasks, 10% of which is spent verifying production bug repairs.
- Xray reduces the time spent on rework by 5% in Year 1, which saves 14,658 hours.
- Time spent on rework is reduced by 8% in Year 2 and 15% in Year 3, which saves the composite organization an additional 67,425 hours over that time.
- Of the time saved, 50% is recaptured for work on other tasks and activities.
- The average fully burdened annual salary for a software developer is $108,000.
- Averaged together, the fully burdened annual salary for software testers and test managers is $108,000.
Risks. Organizational differences that may impact the value of faster MTTR include the number and compensation of software developers, testers, and test managers; the number of production bugs experienced; and their prior state of efficiency addressing them. The value of this benefit also does not include the avoided costs of reputational harm to the organization, which could be substantial for widely used, mission-critical applications.
Results. To account for these risks, Forrester adjusted this benefit downward by 20%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $1.3 million.
Value Of Faster Mean Time To Repair
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| C1 | Software developers involved in testing | Composite | 1,200 | 1,200 | 1,200 | |
| C2 | Percentage of time spent on software development | Composite | 65% | 65% | 65% | |
| C3 | Percentage of time spent on rework related to production bug fixes | Composite | 15% | 15% | 15% | |
| C4 | Fully burdened annual salary for a software developer | Forrester standard | $108,000 | $108,000 | $108,000 | |
| C5 | Subtotal: Developer cost of rework related to production bug fixes | C1*C2*C3*C4 | $12,636,000 | $12,636,000 | $12,636,000 | |
| C6 | Software testers/managers | Composite | 400 | 400 | 400 | |
| C7 | Percentage of time spent on test management tasks | Composite | 65% | 65% | 65% | |
| C8 | Percentage of time spent verifying production bug repairs | Composite | 10% | 10% | 10% | |
| C9 | Fully burdened annual salary for a software tester/manager (blended) | Forrester standard | $108,000 | $108,000 | $108,000 | |
| C10 | Subtotal: Tester cost of verifying production bug repairs | C6*C7*C8*C9 | $2,808,000 | $2,808,000 | $2,808,000 | |
| C11 | Months Xray is deployed | Interviews | 9 | 12 | 12 | |
| C12 | Percentage of time saved attributable to Xray | Interviews | 5% | 8% | 15% | |
| C13 | Percentage of time recaptured | Forrester standard | 50% | 50% | 50% | |
| Ct | Value of faster mean time to repair | (C5+C10)*(C11/12)*C12*C13 | $289,575 | $617,760 | $1,158,300 | |
| Risk adjustment | ↓20% | |||||
| Ctr | Value of faster mean time to repair (risk-adjusted) | $231,660 | $494,208 | $926,640 | ||
| Three-year total: $1,652,508 | Three-year present value: $1,315,235 | |||||
Cost Savings From Retiring Legacy Test Management Solutions
Evidence and data. Retiring legacy test management solutions and transitioning to Xray often resulted in significant cost savings for interviewees’ organizations. With Xray, the license cost per user was a fraction of what they paid for legacy tools, enabling them to increase the number of active users. Because it is a Jira application, interviewees’ organizations found Xray to be less expensive to maintain compared to standalone tools.
- The test guild lead at a financial services firm shared that their Xray license cost one-third of the price of their legacy tool. According to this interviewee, the legacy vendor wanted their organization to update to a newer version, which they declined due to the high license price. Consequently, the organization did not invest in upgrades or support the product, which ultimately led them to look for other solutions.
- A test manager at an automotive company explained: “The cost of [the legacy tool] is totally different in how it’s calculated because it has a licensed system with concurrent users, so only parallel sessions are counted. The cost of a single license was quite expensive initially, but after that you pay only a maintenance fee on those licenses. … Conversely, our Xray license cost $16,000 and we don’t have any additional operations effort in terms of infrastructure. It is part of Jira. Of course, we have an internal head count supporting Xray — one FTE — but from that aspect, we save around half a million dollars on licensing per year. Operationally it is an add-on, which is cheaper than operating standalone tools.”
- The director of technical product management at a pharmaceutical company told Forrester that Xray was significantly less expensive than their previous tool but could not divulge specific financial data due to company policy.
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:
- The composite organization spends $135,000 on licensing costs for its legacy test management solution. It also employs one FTE who spends an average of 30% of their time supporting the solution.
- As the organization migrates testing work to Xray, it reduces these costs by 50% in Year 1. The savings grow over time as it shifts more testing work from the legacy solution to Xray.
- Of the support time saved, 50% is recaptured for work on other tasks and initiatives.
- The average fully burdened annual salary for FTEs involved in maintaining the legacy solution is $128,250.
Risks. Organizational differences that may impact the cost savings from retiring or reducing the use of legacy test management solutions include the licensing and pricing models of those solutions, the costs associated with their upkeep, and the pace at which the organization migrates away from those solutions.
Results. To account for these risks, Forrester adjusted this benefit downward by 20%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $216,000.
Cost Savings From Retiring Legacy Test Management Solutions
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| D1 | Legacy test management solutions licensing costs | Interviews | $135,000 | $135,000 | $135,000 | |
| D2 | FTEs involved in maintaining legacy solutions | Interviews | 1 | 1 | 1 | |
| D3 | Percentage of time spent supporting legacy solutions | Interviews | 30% | 30% | 30% | |
| D4 | Fully burdened annual salary for FTEs involved in maintaining legacy solutions | Forrester standard | $128,250 | $128,250 | $128,250 | |
| D5 | Percentage of time recaptured | Forrester standard | 50% | 50% | 50% | |
| D6 | Cost of maintaining legacy solutions | D1+(D2*D3*D4*D5 ) | $154,238 | $154,238 | $154,238 | |
| D7 | Reduction in use of legacy solutions | Composite | 50% | 75% | 90% | |
| Dt | Cost savings from retiring legacy test management solutions | D6*D7 | $77,119 | $115,679 | $138,814 | |
| Risk adjustment | ↓20% | |||||
| Dtr | Cost savings from retiring legacy test management solutions (risk-adjusted) | $61,695 | $92,543 | $111,051 | ||
| Three-year total: $265,289 | Three-year present value: $216,003 | |||||
Unquantified Benefits
Interviewees mentioned the following additional benefits that their organizations experienced but were not able to quantify:
-
Native integration in Jira. Xray’s seamless integration in Jira eliminated the need to switch
tools, streamlining processes and providing a complete view of the software
development lifecycle within Jira. Testing artifacts are managed as issues
in the same way as all other Jira issue types.
When asked what they considered to be the biggest benefits of Xray, the product manager of agile planning at a pharmaceutical company answered: “The deep integration in Jira. From the point of view of the end user, it is just one of your extra buttons in Jira. That’s a great benefit because it allows a complete trail from the story through the code, through the test, and through the back. Everything is seamlessly connected and automatically linked. The visibility into the software delivery life cycle is the biggest benefit I see in Xray.”
A release and quality assurance manager at an energy company confirmed this benefit and said: “It’s not an add-on that users need to access differently. That helps me a lot when working with project managers and business analysts. We have a complete view of the software development lifecycle, jobs for developers, as well as information about which tests passed or failed for each user story, all on the same platform. For me, that full integration is the biggest asset of Xray.” -
Xray service and support. The interviewees praised Xray’s service and product teams for
their responsiveness and accessibility. They especially appreciated the
prompt issue resolution, ability to influence feature development, and
documentation and training resources. Some of the organizations used Xray’s
Enterprise support, which was particularly beneficial for addressing urgent
needs and collaborating on best practices. Overall, Xray’s support
significantly enhanced user experience and satisfaction.
According to the product manager of agile planning at a pharmaceutical company: “The documentation and the support Xray provides is some of the best that I’ve seen. Compared to other vendors, it’s much better. One thing I especially appreciate is that we have the ability to directly communicate with the product management. That’s something that very few vendors offer.”
The interviewee added: “Both their first-line support and product teams are very responsive. It’s easy to get them on a call and they are very eager to understand our situation and our use case and find the best solution. … They are very receptive to feedback, very open about their plans, and usually share if something is coming soon. … Compared to other vendors, the visibility and communication is very good.”
A release and quality assurance manager at an energy company emphasized the importance of the support provided by Xray: “Our experience has been very positive. Whenever we encounter an issue, communication with the Xray team is straightforward. Additionally, we are able to influence the implementation of new features, which is not something we often see with other tools.”
The interviewee also noted: “From a communications perspective, it helps that Xray is a main sponsor of a leading testing conference. This sponsorship reinforces to our testers and other stakeholders that we are using a highly recognized and top-tier tool. The industry recognition that Xray has garnered is a plus for us.” -
Flexibility. Xray was
easy to modify so users could take different approaches and use different
methodologies with the same artifact. However, although there was more
freedom in selecting fields and creating widgets and dashboards, one
interviewee stressed the importance of standardization for a more stable and
manageable environment and more effective collaboration among
teams.
According to the test guild lead at a financial services firm: “There is much more freedom in selecting things or creating widgets and dashboards. There are many ways to do it. At first people try to ask us, what is the correct way? But there is not one correct way. You are free to try it differently. Now they are pretty much used to it. There are some people who, when they are selecting or filtering issues, ask around about how to do it or ask for hints, but a lot of people are using it at a very high level, and they have no problem.”
A release and quality assurance manager at an energy company agreed: “The tool is flexible. It’s easy to modify, so we can use different approaches and methodologies with the same artifact. We don’t restrict our teams in the business to one function. This flexibility makes it easy for all of our testers and users to adapt to it.”
A test manager at an automotive company recognized Xray’s flexibility but noted their organization took a structured approach to foster better collaboration: “Having had the experience of the previous tool, we decided to reduce complexity with the introduction of Jira and the new tool chain. Now we have 100% standardization, including Xray. Everything is native to Jira but is 100% standardized on this platform in order to allow the teams to collaborate across projects, to simplify communication, to reduce complexity in the operations, and so on. There are lots of reasons why we have this standardization, and in the initial phase, this required the feedback from multiple teams to fine-tune the setup, which needed to be very stable. Xray is flexible, but in this case, we wanted to use a standard.” -
Easy integration.
Interviewees’ organizations found that integrating Xray with automation
tools through the API layer was relatively easy due to its native
integrations and flexibility, making it a powerful tool for managing manual
and automated tests in a unified platform. Xray’s comprehensive support also
helped some interviewees’ organizations overcome challenges with needing
specific configurations and customizations to fit unique project
requirements.
A release and quality assurance manager at an energy company said: “Because Xray integrates with Jira, you have the same kind of technology. If you want to build on it, you can. This is particularly useful if you need to integrate with test automation tools. We run our test automations and get the results automatically in Xray. If you have two different tools doing this, this work would be much more complicated than it is today. The way we have things implemented now simplifies our process. … So far we haven’t had any issues or problems. Maybe it’s because we are not using a lot of add-ons or different tools.”
A test manager at an automotive company told Forrester: “Xray is fully integrated into our DevOps toolchain. This centralized toolchain is used by all development teams. The integration is of high importance.” -
Improved reporting.
Interviewees considered Xray an easily legible, modern tool with a friendly
user interface. They especially appreciated that it provided full
traceability from requirements to test executions and defects all in one
platform. This transparency improved test coverage and standardized
acceptance criteria, leading to improved quality and compliance audits.
Moreover, Xray’s data extraction and report-building capabilities helped
business users such as project managers and release managers quickly find a
project’s status.
According to the test guild lead at a financial services firm: “It’s helped simplify our management audits because the reporting is easier by having data all in one place. Now when auditors ask to see changes to an application, we can show that delivery, and it contains everything. It’s all there and interconnected.”
The chief control officer at another financial services firm told Forrester: “The reports and dashboards are useful. The dashboards work at the application testing level quite well for projects that are for a specific release or releases within the business.”
The interviewee continued: “A lot of the processes we test with Xray are complex, so when there is an issue in production or in regression testing or in our performance testing environments, we can analyze exactly how we passed from user acceptance testing to the next stage in testing. And we find, for example, a test developed with data from an upstream system that changed in latter stages of the test cycle causing a problem with unexpected values. It gives very clear traceability not only within the test case but from version to version of the application.”
The interviewee continued: “[Test Case Versioning] gave us a more consistent, fact-based position. I was able to demonstrate it by providing the lineage and the evidence to make the story stronger. Now we have a tool-based story rather than spreadsheets and manually created documents. When you’re putting the stories together manually, you can’t always find the files. You’re going through backup files and piecing things together. All that time is now saved and it’s less dependent on expertise. People who were less involved can traverse the system and get the evidence. It’s a significant savings.”
The interviewee also added that business end users found Xray reporting helpful: “The business feels more engaged and invested in the process. They understand what the technology is doing for testing their applications. They’re able to participate and have more confidence in the testing process. They can see into that black box a bit more than they could before.” -
Faster development cycles. Some interviewees found that streamlined communication between
developers and testers contributed to shortened development cycles for their
organizations, as did workflow changes to conduct testing at earlier stages
in the development process. Other interviewees were still evaluating Xray’s
impact on development cycles.
A release and quality assurance manager at an energy company told Forrester: “It’s enabled us to move faster — not just the testing, but because everything is integrated in Jira. Communication between developers and testers is so much faster and doing project management is much faster. Everyone wins with this.”
The test guild lead at a financial services firm agreed development cycles were generally faster, although results varied by team: “For many teams, we are able to shift left quite a lot, unite test coverage and some automation and so on, but in other teams, they are still on the way. Before we changed to Jira, we had quite distinct test phases. Professional testers and system testers tested first, then they asked business testers to test. Now business is testing from the earliest stages. So yes, there was definitely a shift left.” -
Ease of adoption. Xray
works as a seamless extension of Jira, and interviewees’ organizations found
that users with Jira experience could understand and use Xray easily. If
they had questions, interviewees noted that users could find answers by
reading the manual or watching a training video accessible through Xray
Academy. This accessibility and onboarding support fostered widespread Xray
adoption, enhancing communication and collaboration between agile teams of
testers, developers, analysts, and business
stakeholders.
A test manager at an automotive company noted: “Many people at our company have experience with Jira. It was much easier to adopt Xray because the issues are the same way of working compared to [the legacy tool], which was a standalone tool with a different graphical user interface and methods to manage things. It’s a plus that Xray seamlessly integrated into Jira, which makes it easier for people to adopt.”
A release and quality assurance manager at an energy company said: “It is easy for everyone to learn about Xray because you have Xray Academy. A lot of forum threads [about Xray] exist on the internet because it’s used worldwide, so for every problem that users encounter, they can do a search and find a solution.”
The interviewee continued: “It helps that Xray is integrated with Jira. Many people don’t realize it is a separate tool because it’s in Jira, so it’s very smooth. This is why we have these high numbers of active users, and it’s not only testers. It was so important to be able to extend this to our suppliers using our Jira and our Xray. For example, one of our business units has more than 10 suppliers. The most important ones we use for our release certifications use our Jira Xray workflows to follow up on the defects and everything that we are working on for the certification process, so there are no emails from one side to the other to figure out what is going on.”
The product manager of agile test management at a pharmaceutical company liked that Xray is “browser-based, doesn’t require any installation on end users’ computers, and has a fast response. … It is very strong, a major improvement compared to historical tools.”
The test guild lead at a financial services firm said: “Now it’s much more interconnected. Every time there is a delivery going live, I know which test covered this delivery, which bugs were found in this delivery. Activities and release requirements are there, so everybody knows that the delivery is ready because they passed all the checks. It’s much easier to keep eyes on things.”
This interviewee noted their organization was still migrating business users to Jira and Xray: “All internal and external testers must use Jira and Xray as a testing tool. They need to record defects in Jira and their testing activities in Xray. As for business users, we are in the process of migrating their activities into Jira.” -
Improved user experience. Interviewees agreed that although some users had to adjust to
using Xray and Jira, the experience was better than what they had before.
According to the test guild lead at a financial services firm: “At first, it
was too much freedom, but now, everybody found their own degree of freedom.
They’re able to follow the process, create and execute, and report. They’re
able to raise defects. [Some users] like creating their own dashboard or
doing a bit of optimization.”
A release and quality assurance manager at an energy company agreed and said: “The user experience has significantly improved. While we still need to monitor users to ensure they create test cases to support their stories, feedback regarding Xray’s user-friendliness has been largely positive.”
Flexibility
The value of flexibility is unique to each customer. There are multiple scenarios in which a customer might implement Xray and later realize additional uses and business opportunities, including:
-
Expanding test automation. Several interviewees told Forrester their organizations were in the
process of growing test automation through Xray. The test guild lead at a
financial services firm noted: “We are in the process of starting to execute
automated tests from Jira. We needed to focus on our agile transformation
first, so we put automation on the back burner. Now we are in a fully agile
setup.”
A test manager at an automative company told Forrester that adoption of Xray at their organization was already high but would continue to grow as test automation rose: “We expect to increase adoption of test automation and parameterized automated tests, which reports results back to Xray. We have an end-to-end solution which is quite commonly used. Because Xray offers the best API, we have added other test automation tools in certain areas, which also report results to Xray. … We’re continuing to add things and integrate test automation tools. That’s worked well with the Xray APIs.”
A release and quality assurance manager at an energy company also said their organization planned to expand test automation with Xray: “[The legacy system provided a tool for automation], but it was difficult to use and very technical. Today, our numbers are very different, from 200 to more than 6,000 applications automated. At the moment, 30% of our regression test cases are automated. This expansion is also due to the fact that users have easier access to test cases and can identify what should be automated in regression test plans themselves. Because of the integration, everything is tied together. All the information is at the distance of a click.” -
Experimenting with generative AI. Generating test cases with AI was of keen interest to the
interviewees, and they hoped Xray would release AI capabilities to help them
with the faster speed of code development.
According to a test manager at an automotive company: “Generative AI has accelerated application output. Specific support for AI-generated tests could save our efforts and help us cope with the faster speed we get with the AI generated code we have. We need to take advantage of these new methodologies to create test cases. We’ve looked at other solutions, but if this could be integrated into Xray, it would be very beneficial.”
At the time of this study, Xray is actively developing AI solutions to complement and strengthen its core testing activities like test case creation and test case designer. To learn more about its status, please contact the Xray team.
Flexibility would also be quantified when evaluated as part of a specific project (described in more detail in Appendix A).
Quantified cost data as applied to the composite
Total Costs
| Ref. | Cost | Initial | Year 1 | Year 2 | Year 3 | Total | Present Value |
|---|---|---|---|---|---|---|---|
| Etr | Xray licensing | $0 | $28,980 | $28,980 | $61,793 | $119,753 | $96,721 |
| Ftr | Planning, implementation, and ongoing management | $73,747 | $22,124 | $14,749 | $22,124 | $132,744 | $122,671 |
| Gtr | Training and change management | $1,052,480 | $239,200 | $239,200 | $621,920 | $2,152,800 | $1,934,878 |
| Total costs (risk-adjusted) | $1,126,227 | $290,304 | $282,929 | $705,837 | $2,405,297 | $2,154,270 | |
Xray Licensing
Evidence and data. Xray licensing costs are based on the number of Jira users, with pricing tiers depending on whether Jira Cloud or Jira Data Center is used.
- The standard module of Xray includes features that enable test case creation and management, requirements traceability that links tests to requirements, built-in reports and customizable dashboards, and integration with CI/CD tools and automation frameworks.
- Xray Enterprise includes advanced features like Test Case Versioning, Test Case Design, Dynamic Test Plans, and 24/7 support.
The interviewees’ organizations used a mix of the two license types, typically starting with the standard module and later adopting Xray Enterprise.
- According to a test manager at an automotive company: “When we first adopted Xray, the earlier standard version had an add-on premium support package that provided 24/7 support. Since then, they grouped the premium support package with some advanced features and called it Xray Enterprise. Standard Xray no longer has the premium support option, so we switched over to Xray Enterprise to get that enhanced support.”
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:
- The composite organization starts with standard Xray Cloud licenses and migrates to Xray Enterprise Cloud in Year 3.
- Pricing may vary. Contact Xray for additional details.
Risks. Organizational differences that may impact the costs associated with Xray licensing include the size and scale of Jira deployment and the license type deployed.
Results. To account for these risks, Forrester adjusted this cost upward by 5%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $97,000.
Xray Licensing
| Ref. | Metric | Source | Initial | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|---|
| E1 | Xray Cloud licensing costs | Provided by Xray | $27,600 | $27,600 | |||
| E2 | Xray Enterprise Cloud licensing costs | Provided by Xray | $58,850 | ||||
| Et | Xray licensing | E1+E2 | $27,600 | $27,600 | $58,850 | ||
| Risk adjustment | ↑5% | ||||||
| Etr | Xray licensing (risk-adjusted) | $0 | $28,980 | $28,980 | $61,793 | ||
| Three-year total: $119,753 | Three-year present value: $96,721 | ||||||
Planning, Implementation, And Ongoing Management
Evidence and data. Interviewees reported planning and implementation time ranging from one to six months depending on the size and scope of their organization’s testing operations. Two organizations used Xray partners to help with the transition, but others managed it themselves. All of the interviewees’ organizations ran pilot projects and took a phased adoption approach, working with different test and development teams to facilitate change management. Once implemented, Xray required low maintenance, primarily consisting of Jira-related configuration adjustments, update deployments, and tool integrations.
- The product manager of agile planning at a pharmaceutical company found Xray’s implementation to be relatively simple, even in a highly regulated industry that requires rigorous internal documentation: “When it comes to Xray, the technical implementation was very easy. I was actually surprised how fast it was for us to set up. We had to customize the default configurations, but the documentation is very straightforward, and everything is really user-friendly. In a matter of a month, it was ready for large scale adoption.”
- A test guild lead at a financial services firm noted: “There was a core team of six people: two testers, one technology tester, one application tester, plus two test managers. It took about 30% of their time over three months. We started with bugs because it was much easier. We didn’t need a lot of customization; we just changed some fields and created a workflow. After that, we went to test preparation, execution, and reporting.”
- The chief control officer at another financial services firm ran three pilot projects over several weeks while managing other priorities: “Because the proof of concept was only three projects, we just did a baseline implementation configuration. We were more focused on the end users and whether the tool could manage the end-to-end process correctly rather than the technical aspects of managing the platform, so it took us probably a couple weeks to get it set up.”
- A test manager at an automotive company told Forrester: “After a six-month pilot phase with 10 teams, Xray was released as an alternative to the existing test management solution for small projects. After 12 months, it was released to all as an option. Finally, it was set as the preferred tool and later the retirement process of the old tool was announced.”
- According to the product manager of agile planning at a pharmaceutical company: “Xray itself requires a rather low amount of maintenance effort. I would say 10% to 20% of an FTE on average. It’s commercial, off the shelf. We update once every few months when there is a new version that we want to deploy. There are some ongoing configuration changes, small tweaks, since it’s part of Jira. That is the great advantage for many, many points. It doesn’t require any specific maintenance itself. It just uses the Jira instance and set up.”
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:
- Planning and implementation, including migrating existing projects, takes place over three months.
- Four data architects dedicate 50% of their time to the planning and implementation process.
- After implementation, one Jira administrator/project manager spends between 10% and 15% of their time maintaining Xray in the Jira environment, including onboarding new users, setting up new projects, and adding integrations.
- The fully burdened annual salary for a data architect and a Jira administrator/project manager is $128,250.
Risks. Organizational differences that may impact costs associated with planning, implementation, and ongoing management include the volume, organization, and state of test data inside the company prior to deploying Xray; the number and scope of pilot tests run; the number of integrations made to Xray; the efficiency of the teams involved in deploying and maintaining Xray; and local compensation rates.
Results. To account for these risks, Forrester adjusted this cost upward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $123,000.
Planning, Implementation, And Ongoing Management
| Ref. | Metric | Source | Initial | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|---|
| F1 | Initial planning and implementation, including existing project migration (months) | Interviews | 3 | ||||
| F2 | Data architects involved in initial planning and implementation | Interviews | 4 | ||||
| F3 | Percentage of time dedicated to planning and implementation | Interviews | 50% | ||||
| F4 | Fully burdened monthly salary for a data architect | Forrester standard | $10,688 | ||||
| F5 | Subtotal: Planning and implementation | F1*F2*F3*F4 | $64,128 | ||||
| F6 | Jira administrators/project managers involved in ongoing management | Interviews | 1 | 1 | 1 | ||
| F7 | Percentage of time dedicated to ongoing management and integrations | Interviews | 15% | 10% | 15% | ||
| F8 | Fully burdened annual salary for a Jira administrator/project manager | Forrester standard | $128,250 | $128,250 | $128,250 | ||
| F9 | Subtotal: Ongoing management | F6*F7*F8 | $0 | $19,238 | $12,825 | $19,238 | |
| Ft | Planning, implementation, and ongoing management | F5+F9 | $64,128 | $19,238 | $12,825 | $19,238 | |
| Risk adjustment | ↑15% | ||||||
| Ftr | Planning, implementation, and ongoing management (risk-adjusted) | $73,747 | $22,124 | $14,749 | $22,124 | ||
| Three-year total: $132,744 | Three-year present value: $122,671 | ||||||
Training And Change Management
Evidence and data. Interviewees said their organizations often held internal trainings and customized Xray Academy-provided training materials to suit their team structures and processes. For teams familiar with Jira, Xray was easy to adopt; however, business testers with no exposure needed additional support. Change management was a significant training component to instill new test management processes and ways of working enabled by Xray.
- The chief control officer at a financial services firm explained: “We take advantage of Xray Academy and Xray’s documentation. We also created our own internal testing training based on Xray, Jira, and other integrations. There are different trainings that we give to developers, to testing teams, and to business users. We also do things like podcasts and on-demand training that people can listen to, or short videos so they don’t have to sit through an hour-long class. There are formal training classes that point out best practices and tips and tricks. Some teams do lunch and learn sessions.”
- The test guild lead at a financial services firm said: “There were two or three teams that were very excited to change to the tool so we started with them, and they gave us feedback on our training process. Once that was done, we had several sessions with groups of people from those teams along with test managers and tried to spread it from there. The mindset shift was a concern because we had people who had been with the company a long time and they were used to [the legacy tool]. We got them to buy into the change by promising things would be easier. They wouldn’t have to switch to different tools when they wanted to report something or find information.”
- The interviewee ended the interview by observing: “It’s a great opportunity to update your processes. Anything you are not happy about; it’s an opportunity to change. Don’t forget about training. Don’t rush it. People need time to explore the tool and learn about the features and how they will be able to use it, so training is also important. Explore the tool. For me personally, it was the best way to deal with the change. You are doing what every tester would be doing. You are testing what it’s capable of. I would say to clean up your processes, don’t forget training, and explore the tool to find out how things works best for you.”
Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:
- Test administrators receive 20 hours of initial Xray training supplemented with informal quarterly sessions to review updates and best practices. In Year 3, test administrators receive another 20 hours of training on Xray Enterprise’s advanced features and functionality.
- Software developers involved in testing receive 8 hours of initial training and participate in informal semiannual sessions focused on best practices.
- The fully burdened hourly rate for test administrators and software developers is $52.
Risks. Organizational differences that may impact costs associated with training and change management include time needed for up-front planning and training material development, familiarization time needed by users, and local compensation rates.
Results. To account for these risks, Forrester adjusted this cost upward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $1.9 million.
Training And Change Management
| Ref. | Metric | Source | Initial | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|---|
| G1 | Software testers/managers | Composite | 400 | 400 | 400 | 400 | |
| G2 | Time spent on training and change management per software tester/manager (hours) | Interviews | 20 | 4 | 4 | 20 | |
| G3 | Fully burdened hourly salary for a software tester/manager (blended) | Forrester standard | $52 | $52 | $52 | $52 | |
| G4 | Subtotal: Training and change management costs for software testers | G1*G2*G3 | $416,000 | $83,200 | $83,200 | $416,000 | |
| G5 | Software developers involved in testing | Composite | 1,200 | 1,200 | 1,200 | 1,200 | |
| G6 | Time spent on training and change management per software developer (hours) | Interviews | 8 | 2 | 2 | 2 | |
| G7 | Fully burdened hourly salary for a software developer | Forrester standard | $52 | $52 | $52 | $52 | |
| G8 | Subtotal: Training and change management costs for software developers | G5*G6*G7 | $499,200 | $124,800 | $124,800 | $124,800 | |
| Gt | Training and change management | G4+G8 | $915,200 | $208,000 | $208,000 | $540,800 | |
| Risk adjustment | ↑15% | ||||||
| Gtr | Training and change management (risk-adjusted) | $1,052,480 | $239,200 | $239,200 | $621,920 | ||
| Three-year total: $2,152,800 | Three-year present value: $1,934,878 | ||||||
Consolidated Three-Year, Risk-Adjusted Metrics
Cash Flow Chart (Risk-Adjusted)
Total costs Total benefits Cumulative net benefits Initial Year 1 Year 2 Year 3-
The financial results calculated in the Benefits and Costs sections can be used to determine the ROI, NPV, and payback period for the composite organization’s investment. Forrester assumes a yearly discount rate of 10% for this analysis.
These risk-adjusted ROI, NPV, and payback period values are determined by applying risk-adjustment factors to the unadjusted results in each Benefit and Cost section.
Cash Flow Analysis (Risk-Adjusted)
| Initial | Year 1 | Year 2 | Year 3 | Total | Present Value | |
|---|---|---|---|---|---|---|
| Total costs | ($1,126,227) | ($290,304) | ($282,929) | ($705,837) | ($2,405,297) | ($2,154,270) |
| Total benefits | $0 | $1,345,039 | $3,092,891 | $4,319,541 | $8,757,471 | $7,024,207 |
| Net benefits | ($1,126,227) | $1,054,735 | $2,809,962 | $3,613,704 | $6,352,174 | $4,869,937 |
| ROI | 226% | |||||
| Payback | 13 months | |||||
Appendix A: Total Economic Impact
Total Economic Impact is a methodology developed by Forrester Research that enhances a company’s technology decision-making processes and assists solution providers in communicating their value proposition to clients. The TEI methodology helps companies demonstrate, justify, and realize the tangible value of business and technology initiatives to both senior management and other key stakeholders.
Total Economic Impact Approach
-
Benefits represent the value the solution delivers to the business. The TEI methodology places equal weight on the measure of benefits and costs, allowing for a full examination of the solution’s effect on the entire organization.
-
Costs comprise all expenses necessary to deliver the proposed value, or benefits, of the solution. The methodology captures implementation and ongoing costs associated with the solution.
-
Flexibility represents the strategic value that can be obtained for some future additional investment building on top of the initial investment already made. The ability to capture that benefit has a PV that can be estimated.
-
Risks measure the uncertainty of benefit and cost estimates given: 1) the likelihood that estimates will meet original projections and 2) the likelihood that estimates will be tracked over time. TEI risk factors are based on “triangular distribution.”
The initial investment column contains costs incurred at “time 0” or at the beginning of Year 1 that are not discounted. All other cash flows are discounted using the discount rate at the end of the year. PV calculations are calculated for each total cost and benefit estimate. NPV calculations in the summary tables are the sum of the initial investment and the discounted cash flows in each year. Sums and present value calculations of the Total Benefits, Total Costs, and Cash Flow tables may not exactly add up, as some rounding may occur.
Appendix B: Supplemental Material
Related Forrester Research
February 18, 2025, Top Emerging Technologies: TuringBots: Generative AI For SW Development, Webinar.
Diego Lo Giudice, Amid The AI Hype, Agile Still Remains Relevant In 2025, Forrester Blogs.
The State Of Agile Development, 2025: It’s Still Relevant, With Benefits And Challenges, Forrester Research, Inc., January 17, 2025.
The Forrester Wave™: Modern Application Development Services, Q1, 2025, Forrester Research, Inc., January 15, 2025.
Predictions 2025: Software Development, Forrester Research, Inc., October 22, 2024.
Appendix C: Endnotes
1 Total Economic Impact is a methodology developed by Forrester Research that enhances a company’s technology decision-making processes and assists solution providers in communicating their value proposition to clients. The TEI methodology helps companies demonstrate, justify, and realize the tangible value of business and technology initiatives to both senior management and other key stakeholders.
PDF
