As products and services increasingly depend on high-quality continuous data, the importance of reliant and scalable data delivery has never been greater. Enterprise leaders who use Apache Airflow to manage data pipelines often struggle with frequent downtime that threatens core business applications, significant labor to upkeep systems, and difficulty deploying and scaling services. Astronomer’s Astro is a managed service that enables Airflow users to achieve improved data stability and observability, accelerated speed to market, and reduced service and infrastructure management.
Astro is a managed service by Astronomer that offers a unified platform to orchestrate and govern data. Astro reduces the workload required for organizations to use Apache Airflow to run their workflows, allows them to accelerate development, and improves security. This is accomplished with Astro’s visibility and controls, intelligent infrastructure management, and tools for developer productivity. Astro automates much of the process of deploying and scaling Airflow, managing existing Airflow directed acyclic graphs (DAGs), writing and testing new code, and upgrading Airflow to the newest release. Astro also provides greater visibility across environments, providing deeper insight for infrastructure teams and business leaders.
Astronomer commissioned Forrester Consulting to conduct a Total Economic Impact™ (TEI) study and examine the potential return on investment (ROI) enterprises may realize by deploying Astro.1 The purpose of this study is to provide readers with a framework to evaluate the potential financial impact of Astro on their organizations.
To better understand the benefits, costs, and risks associated with this investment, Forrester interviewed four representatives with experience using Astro. For the purposes of this study, Forrester aggregated the interviewees’ experiences and combined the results into a single composite organization. The composite is a B2B SaaS organization with 4,000 employees, leverages Airflow for 15 major service releases each year, and spends $200,000 annually on Airflow infrastructure.
Interviewees said that prior to using Astro, their organizations struggled with Airflow instances crashing and the lack of an audit trail to repair and debug data pipelines. In addition, many key staff members relied on a heavily manual process of deploying and managing Airflow instances, rather than spending their time on higher-value and creative work. However, prior attempts yielded limited success, leaving them with lost revenue from unstable services and dissatisfaction from employees and clients.
After the investment in Astro, the interviewees were able to significantly improve the stability of their services that leverage Airflow and allowed employees to reinvest their time toward higher-value work. Key results from the investment include reduced critical services downtime by 70%, reduced time to resolve Airflow issues in noncritical services by 92%, accelerated service release by seven days, reduced infrastructure costs by 45%, and reduced infrastructure management by 75%.
Key Findings
Quantified benefits. Three-year, risk-adjusted present value (PV) quantified benefits for the composite organization include:
- Accelerated speed to market and time to scale by seven days. Developers in the composite organization benefit from faster feedback loops and the ability to write code at scale, allowing for faster iteration as well as scaling of services. The additional profit enabled by this acceleration is worth $462,000 to the composite organization over a three-year period.
- Saved 4,200 hours of development time. The faster feedback loops and ability to write code at scale allows developers to free up their time for higher-value projects. This saved time benefits the composite organization by $344,000 over a three-year period.
- Improved data reliability and organizational security (business value). Astro reduces Airflow incidents for critical services by 60% due to greater observability as well as development best practices, which Astro automatically incorporates into the composite organization’s environments. In addition, remaining critical issues are solved 25% faster as Astro allows for full data traceability and easier debugging. These two improvements combine for a 70% reduction in Airflow downtime for critical services, worth $532,000 over three years.
- Improved Airflow stability and data visibility (labor efficiency). Astro reduces the number of Airflow incidents for noncritical services by 73% and the mean-time-to-resolution (MTTR) by 67% for any remaining issues. In total, 92% less time is spent resolving these issues, worth $135,000.
- Reduced Airflow infrastructure and management costs. The composite organization uses Astro to reduce its infrastructure costs in two ways. First, cloud computing becomes more efficient as Astro provides greater consistency in the correct types of Airflow instances deployed. Second, as the composite organization moves to Astro Cloud in Years 2 and 3, infrastructure costs reduce even further. In addition, the composite organization reallocates 75% of an infrastructure engineer’s time as management of infrastructure using Airflow is largely automated and no longer requires the same level of manual governance. The combined infrastructure and management cost savings for the composite organization are $200,000 over a three-year period.
Unquantified benefits. Benefits that provide value for the composite organization but are not quantified for this study include:
- Protected reputation among customers, partners, and organizational stakeholders. Since Airflow is a critical underlying technology for many important services, the composite organization can protect its reputation internally and externally as Astro improves the stability, reliability, and speed of these services to meet internal and external expectations and SLAs.
- Enabled improved profit margin while scaling services. Scaling services with Astro is not only faster but can also be done with lower headcount and infrastructure costs, improving the profit margin for the composite organization. This is especially important during early and high-growth phases of new products, when efficiency of scale can be difficult to achieve and there are major risks to delaying rollout.
- Improved the number and quality of artificial intelligence (AI) and machine learning (ML) services. The composite organization benefits as Astro brings together data engineering and machine learning teams on a single platform and enables high-quality training data provided faster and more reliably, in turn allowing for a greater number of artificial intelligence and machine learning services. In addition, freed-up developer and infrastructure engineer time is reallocated to these projects, further increasing the number and quality of services.
- Accelerated access to new Airflow features. Airflow can be updated with significantly less labor and in less time by using Astro than was possible before. This not only allows employees to work on higher-value projects, but it also allows developers and services to benefit from new Airflow features faster than was possible without Astro.
- Provided better employee experience. Freeing employees up for higher-value projects not only provides the composite organization value in terms of business results of those projects but also provides a better employee experience. Developers and engineers can reduce the number of mundane tasks, which are now automated by Astro, and focus instead on creative work.
- Ensured greater protection of personally identifiable information (PII) and compliance with data privacy regulation. The composite organization uses Astro to create a “paper trail” of sensitive data across its data pipelines. This added visibility gives the composite organization more control over PII, ensuring that it is handled and ultimately removed in accordance with all relevant regulations. This also helps the composite organization to avoid substantial fines and loss of customer trust.
Costs. Three-year, risk-adjusted PV costs for the composite organization include:
- Astro usage-based license costs. The composite organization pays Astronomer according to a usage-based model for services that use Astro, as well as for services that run on Astro Cloud. The combined usage-based costs are $286,000 over a three-year period.
- Implementation and ongoing labor. Some change management is required as the composite organization assesses its data ecosystem and prioritizes the highest-value services that should move to Astro first. In addition, a data engineer spends 24 hours each year on ongoing Astro management. This time is mostly low-touch and high-level monitoring for brief periods each week. The combined labor, mostly for the initial change management and internal assessment, costs $25,000 over a three-year period.
The representative interviews and financial analysis found that a composite organization experiences benefits of $1.67 million over three years versus costs of $311,000, adding up to a net present value (NPV) of $1.36 million and an ROI of 438%.
Key Statistics
-
Return on investment (ROI):
438%
-
Benefits PV:
$1.67M
-
Net present value (NPV):
$1.36M
-
Payback:
<6 months
Benefits (Three-Year)
Accelerated speed to market and scalability Saved developer time Improved data reliability and organizational security (business value) Improved Airflow stability and data visibility (labor efficiency) Reduced Airflow infrastructure and management costsTEI Framework And Methodology
From the information provided in the interviews, Forrester constructed a Total Economic Impact™ framework for those organizations considering an investment Astro.
The objective of the framework is to identify the cost, benefit, flexibility, and risk factors that affect the investment decision. Forrester took a multistep approach to evaluate the impact that Astro can have on an organization.
-
Due Diligence
Interviewed Astronomer stakeholders and Forrester analysts to gather data relative to Astro.
-
Interviews
Interviewed four representatives at organizations using Astro to obtain data about costs, benefits, and risks.
-
Composite Organization
Designed a composite organization based on characteristics of the interviewees’ organizations.
-
Financial Model Framework
Constructed a financial model representative of the interviews using the TEI methodology and risk-adjusted the financial model based on issues and concerns of the interviewees.
-
Case Study
Employed four fundamental elements of TEI in modeling the investment impact: benefits, costs, flexibility, and risks. Given the increasing sophistication of ROI analyses related to IT investments, Forrester’s TEI methodology provides a complete picture of the total economic impact of purchase decisions. Please see Appendix A for additional information on the TEI methodology.
Disclosures
Readers should be aware of the following:
This study is commissioned by Astronomer and delivered by Forrester Consulting. It is not meant to be used as a competitive analysis.
Forrester makes no assumptions as to the potential ROI that other organizations will receive. Forrester strongly advises that readers use their own estimates within the framework provided in the study to determine the appropriateness of an investment in Astro.
Astronomer reviewed and provided feedback to Forrester, but Forrester maintains editorial control over the study and its findings and does not accept changes to the study that contradict Forrester’s findings or obscure the meaning of the study.
Astronomer provided the customer names for the interviews but did not participate in the interviews.
Consulting Team:
Chris Layton
Tony Lam
Drivers leading to the Astro investment
Interviews
| Role | Industry | Revenue | DAGs |
|---|---|---|---|
| Data platform lead | Financial services | $500 million+ | 60 |
| Vice president of product and data | SaaS | $100 million+ | 35 |
| Tech lead and data architect | SaaS | $5 billion+ | 500+ |
| Vice president of data solutions and engineering | Media | $1 billion+ | 150 |
Key Challenges
Prior to investing in Astro, some interviewees first attempted to build their own Airflow platform. They struggled with their engineering and DevOps teams being less efficient because deploying, managing, and developing on Airflow took significant amounts of time. In addition, Airflow instances were often set up incorrectly, meaning they could not properly scale and frequently failed, resulting in damage to business operations and reputation.
Other interviewees attempted to use third-party Airflow managed services. These interviewees found that the other platforms neither enabled full use of Airflow’s features nor saved their teams enough time as much of the deployment and management of Airflow was still manual.
Common challenges shared by the interviewees included:
- Instability of Airflow instances and alternate data orchestration platforms, resulting in missed SLAs and lost trust. Interviewees struggled to meet SLAs as data pipelines would frequently fail or not perform at the level required. Depending on the importance of the data pipeline, this hurt ecosystem relationships and limited future business opportunities. The data platform lead at a financial services company said: “When I joined, we were using [an alternate platform] to orchestrate our data pipeline. It was horrendously unstable and would fail about one-third of the time. We would miss our SLA with a capital markets process about once every two weeks, which is unacceptable.”
- Inability to scale data pipelines to meet growing customer needs. Interviewees found that scaling to meet larger customers’ needs was difficult when using alternate Airflow managed services or their own homegrown solutions. Tasks would not scale, systems were not stable, and data was slower. While these issues still surfaced during smaller processes, they became critical when attempting larger workloads. The tech lead and data architect at a SaaS company said: “Airflow doesn’t do as well with faster data requirements, and that’s where Astronomer has certain features that will help you with that. They have top-level lineage debugging for stuck jobs and have monitoring capabilities that were missing with our Airflow deployment.”
- Poor performance and limited functionality of alternate Airflow management platforms. Interviewees who used alternate Airflow managed services struggled with key functionality, resulting in hung tasks that were difficult to resolve. This hurt their organizations’ stability and put SLAs at further risk. It also required that engineers spend their time maintaining the platform rather than developing new features to drive business value. The data platform lead at a financial services organization said: “[An alternative Airflow platform] is missing certain features that are incredibly useful for managing Airflow, such as a REST API. We can get this with Astro with no problem.” The same interviewee later said, “We initially tried to use [an alternative Airflow platform], and we quickly realized it’s a very poor-quality product that necessitated our move to Astro.”
-
Heavy labor and infrastructure burden with homegrown Airflow solutions. Interviewees who attempted to manage their own Airflow platform found that it required excessive labor and took their teams away from core responsibilities. Teams also lacked insight into the underlying processes and infrastructure as metadata would be lost. The tech lead and data architect at a SaaS company said: “Airflow has issues with underlying infrastructure, so jobs get rerun when they shouldn‘t, and metadata on why jobs are being run gets lost. We didn’t have visibility into the logging entries or missing metadata, and it was difficult for us to build that functionality with our existing infrastructure team.”
The vice president of product and data at a SaaS company said: “Once you’ve made the decision to go with Airflow, you have to decide if you want to manage it yourself. In theory, you have more control, but it’s difficult and it’s expensive. Once you decide you want to go with a managed service provider for Airflow, you’re talking about features and price between providers. Astronomer clearly wins on both of those from our perspective.”
The vice president of data solutions and engineering at a media company said: “If you use vanilla Airflow, you end up spending your time managing infrastructure instead of focusing on the right things. It costs you engineering time and productivity. We decided to use Astro to get away from that headache and no longer need to manage or spend the time upgrading Airflow. It’s really much easier for us.”
-
Limited employee effectiveness due to missing Airflow features and control. Interviewees who tried to use alternate Airflow managed services found that they did not have the level of control of Airflow that their stakeholders required. This limited how effective DevOps, data science, and other teams could be. In addition, many core Airflow functionalities were either not fully available or had been completely removed, further inhibiting data teams’ ability to effectively deliver data and end users’ ability to effectively leverage that data in their products and services.
The vice president of product and data at a SaaS company said: “Other services take Airflow and do weird proprietary stuff to it that you have to figure out. They say it’s Airflow, but then you look under the hood, and a bunch of stuff is missing.”
- Difficulty manually upgrading Airflow versions. Interviewees who attempted to manage their own Airflow deployment or used an alternative Airflow managed service often found that upgrading Airflow versions took significant time and resources. Interviewees who did upgrade found it took several days or even weeks until the new Airflow version was ready for production. Other interviewees found the process too difficult and instead would not upgrade Airflow, missing out on new features and security patches. The vice president of product and data at a SaaS company said, “Updating Airflow with other services was so difficult that we often just didn’t do it; the value of the incremental feature being released was not high enough to justify the time to run the update.”
Solution Requirements/Investment Objectives
The interviewees’ organizations searched for a solution that could:
- Provide assurance of Airflow stability and reduce risk to business operations of application and service downtime.
- Increase stakeholder trust in data reliability and freshness.
- Significantly reduce the labor required to leverage Airflow and free up developers and engineers to focus on higher-value work.
- Leverage the full power and flexibility of Airflow, rather than heavily limited environments that could not meet all data platform requirements.
After an RFP and business case process evaluating multiple vendors, the interviewees’ organizations chose Astro and began deployment.
- Interviewees initially tested out Astro in smaller environments for a few weeks and then quickly expanded usage to critical environments once they observed the results of the pilot tests.
-
Most interviewees used Astro across almost all data pipelines; however, some more complex organizations had more fragmented systems and structure using a variety of platforms. In these cases, Astro’s usage was generally expected to increase and grow to become the centralized data platform.
The tech lead and data architect at a SaaS company said: “Astro came out on top when we looked at alternative options to our existing infrastructure build of Airflow. I definitely don’t know how else we could be doing better.”
Composite Organization
Based on the interviews, Forrester constructed a TEI framework, a composite company, and an ROI analysis that illustrates the areas financially affected. The composite organization is representative of the four interviewees, and it is used to present the aggregate financial analysis in the next section. The composite organization has the following characteristics:
Description of composite. The global, billion-dollar business-to-business organization provides custom SaaS solutions for its clients, leveraging large data sets to train its artificial intelligence and machine learning models. The organization has a strong brand, operates globally, releases 15 new services each year that leverage Airflow, and brings in an average of $5 million in new revenue annually. Its existing Airflow environments have 4 9s (99.99%) uptime and cost about $200,000 each year in cloud computing infrastructure.
Deployment characteristics. The composite organization begins in Year 1 by leveraging Astro for its Airflow environments that support the most critical services and part of the SaaS business. In Year 2, this is expanded to most major services, and additional workflows are shifted from the composite organization’s infrastructure to Astro’s cloud infrastructure. In Year 3, all major services leveraging Airflow are run through Astro, and most workflows are run in Astro.
Quantified benefit data as applied to the composite
Total Benefits
| Ref. | Benefit | Year 1 | Year 2 | Year 3 | Total | Present Value |
|---|---|---|---|---|---|---|
| Atr | Accelerated speed to market and scalability | $95,890 | $191,781 | $287,671 | $575,342 | $461,801 |
| Btr | Saved developer time | $71,400 | $142,800 | $214,200 | $428,400 | $343,857 |
| Ctr | Improved data reliability and organizational security (business value) | $184,800 | $231,000 | $231,000 | $646,800 | $532,463 |
| Dtr | Improved Airflow stability and data visibility (labor efficiency) | $46,236 | $55,983 | $61,570 | $163,789 | $134,558 |
| Etr | Reduced Airflow infrastructure and management costs | $35,000 | $86,000 | $129,000 | $250,000 | $199,812 |
| Total benefits (risk-adjusted) | $433,326 | $707,564 | $923,441 | $2,064,331 | $1,672,491 |
Accelerated Speed To Market And Scalability
Evidence and data. Interviewed IT decision-makers who used Astro consistently found that it allowed them to accelerate development time through faster feedback loops, reliable data delivery, and easier development.
- A financial services organization was able to iterate and debug its new services much faster with Astro than using Airflow on its own. The data platform lead attributed this to the Astro’s faster feedback loops, which provided developers with near-real-time feedback, tightened the development cycle, and enabled scaling of solutions. The data platform lead estimated that this improved speed and scalability enabled their organization to get to market three to six months faster than otherwise possible and that this acceleration from Astro was worth hundreds of thousands to low millions of dollars in incremental revenue.
- The vice president of product and data at a SaaS organization similarly found that Astro reduced environment development time as feedback loops were faster. The acceleration and scalability enabled by Astro shortened the environment development time for their developers from several weeks with Airflow to just three days with Astro.
- In addition to shorter development time, the same vice president of product and data found that Astro considerably decreased the time to spin up an environment from one to two months before Airflow to a week with Airflow — and down to just half an hour with Astro managing Airflow. Astro was credited with this improvement through better documentation, proper Airflow management, and shorter test cycles.
- A media organization no longer needed to pause its production for a week twice each year while Airflow was being upgraded. This allowed some new models to be pushed out a week faster than otherwise possible.
- Interviewees also found that Astro shortened development time by allowing for quicker integration of new data sources as their services and environments expanded.
Modeling and assumptions. Forrester leveraged interview data to model the financial impact for the composite organization and assumes the following:
- Each major new service provides an average of $5 million in revenue each year.
- An average of 15 new services are released each year, all of which use Astro by Year 3.
- Developers benefit from faster feedback loops, rapid integration of new data sources, and easier scalability with Astro. This shortens development time by seven days, meaning the service is released and begins providing business value seven days earlier.
- The operating profit margin is 25%.
Risks. Forrester recognizes that these results may not be representative of all experiences and that the impact may vary depending on several factors:
- Many interviewees estimated that the revenue accelerated by Astro was higher than this analysis assumes, but it is also possible that this figure could be lower for other organizations depending on the use of a new service and which parts of the business leverage Airflow.
- While all interviewees found that Astro accelerated their speed to market and scalability, many teams used additional tools that helped them receive additional benefit. This analysis only accounts for the improvement provided by Astro, but readers should consider their own technology stack and capabilities that will benefit from shorter feedback loops and development cycles.
Results. To account for these risks, Forrester adjusted this benefit downward by 20%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $462,000.
Accelerated Speed To Market And Scalability
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| A1 | Average business value of a new service using Astro and Airflow | Composite | $5,000,000 | $5,000,000 | $5,000,000 | |
| A2 | New services using Astro and Airflow released annually | Composite | 5 | 10 | 15 | |
| A3 | Development time saved per new service built with Astro (days) | Interviews | 7 | 7 | 7 | |
| A4 | Incremental revenue due to Astro (rounded) | (A1/365)*A2*A3 | $479,452 | $958,904 | $1,438,356 | |
| A5 | Operating profit margin | Composite | 25% | 25% | 25% | |
| At | Accelerated speed to market and scalability | A4*A5 | $119,863 | $239,726 | $359,589 | |
| Risk adjustment | ↓20% | |||||
| Atr | Accelerated speed to market and scalability (risk-adjusted) | $95,890 | $191,781 | $287,671 | ||
| Three-year total: $575,342 | Three-year present value: $461,801 | |||||
Saved Developer Time
Evidence and data. In addition to accelerated speed to market and scalability providing greater business value, developers and other staff also saved time with faster feedback loops enabled by Astro.
- In each of the examples shared in the previous section, interviewees found that developers’ time was freed up, so they could focus on other high-value work such as improving existing services or beginning development on a new service.
- The data platform lead at a financial services company said, “[Astro] doesn’t just save time in deployment to production, but because it’s automated and easy, everyone is going to fall into deployment best practices without even thinking about it.”
- The vice president of data solutions and engineering at a media company shared that for every week that Airflow no longer had to be paused for updates, a team of six staff (two DevOps and four engineers) regained the entire week and provided more value to the organization as Astro made Airflow upgrades significantly easier and faster.
Modeling and assumptions. Forrester leveraged interview data to model the financial impact for the composite organization and assumes the following:
- Each major new service involves a team of five developers.
- An average of 15 new services are released each year, all of which use Astro by Year 3.
- Developers benefit from faster feedback loops and easier scalability with Astro, and development time is shortened by seven days, meaning the service is released and begins providing business value seven days earlier.
- Development operations engineers have a fully burdened hourly wage of $60.
- Half of the time saved with faster development can be reinvested into higher-value tasks that provide the organization with material benefits.
Risks. Forrester recognizes that these results may not be representative of all experiences and that the impact may vary depending on several factors:
- The value of saving labor depends largely on what employees can do with their saved time. If developers do not have high-value work they can shift to with their saved time, organizations may not see as high a benefit with Astro enabling faster feedback loops.
- Similarly, the value of saved labor will also depend on the types of employees that are impacted. While this analysis focuses on saved time for developers, it is likely that other employees will experience time savings with Astro as well. For instance, frontline workers may save additional time if new features they rely on are released earlier.
Results. To account for these risks, Forrester adjusted this benefit downward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $344,000.
Saved Developer Time
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| B1 | New services using Astro and Airflow released annually | A2 | 5 | 10 | 15 | |
| B2 | Development time saved per new service built with Astro (days) | A3 | 7 | 7 | 7 | |
| B3 | Data engineers involved with development for each new service | Interviews | 5 | 5 | 5 | |
| B4 | Development time saved annually with Astro (hours) | B1*B2*B3*8 | 1,400 | 2,800 | 4,200 | |
| B5 | Development operations engineer fully burdened hourly wage | TEI standard | $60 | $60 | $60 | |
| Bt | Saved developer time | B4*B5 | $84,000 | $168,000 | $252,000 | |
| Risk adjustment | ↓15% | |||||
| Btr | Saved developer time (risk-adjusted) | $71,400 | $142,800 | $214,200 | ||
| Three-year total: $428,400 | Three-year present value: $343,857 | |||||
Improved Data Reliability And Organizational Security (Business Value)
Evidence and data. One of the most significant improvements interviewees found with Astro was in the stability and data reliability of systems leveraging Airflow.
- Interviewees generally found that Astro provided a full order of magnitude of improvement for Airflow services (for example, improving uptime from four 9s or 99.99% to five 9s or 99.999%).
- The data platform lead at a financial services organization said: “We use Airflow to orchestrate that daily reporting process, which means that if that Airflow DAG fails, we cannot operate as a company. We actually have to dig into our capital in order to get by. It’s a huge deal.”
- The vice president of product and data at a SaaS organization shared that if they managed Airflow on their own, that services uptime would be between two to three 9s (99% to 99.9%), as compared to four 9s (99.99%) with Astro. This represents between a 90% and 99% reduction in downtime enabled by Astro.
- A financial services company found that it had one critical issue affecting its Airflow-enabled services each month before Astro, and solving the issue took up to a full day. This time was reduced to just 1 to 2 hours with Astro.
- Interviewees shared that part of the improvement with Astro was that it made best practices for Airflow deployment automatic, so the overall system resiliency improved. This contrasts to when interviewees had leveraged alternative Airflow platforms or had attempted to manage Airflow on their own and found that service reliability suffered.
- The vice president of data solutions and engineering at a media company said: “Our operations center has Astro up on the all the TV screens, so our support team can watch how everything is performing. It’s a beautiful thing and lets us debug faster.”
Modeling and assumptions. Forrester leveraged interview data to model the financial impact for the composite organization and assumes the following:
- The composite organization has an uptime of four 9s for Airflow services before they began using Astro. Downtime is reduced by 60% due to fewer critical issues after Astro manages Airflow environments.
- In addition, critical incidents are resolved 25% faster with Astro due to better telemetry, audit trails, and accessibility of metadata.
- The total reduction in downtime for critical Airflow systems is 70% due to fewer issues and faster resolution enabled by Astro.
- Each hour of downtime for critical systems costs the composite organization $500,000, including lost revenue, legal fees, SLA penalties, and labor.
Risks. Forrester recognizes that these results may not be representative of all experiences and that the impact may vary depending on several factors:
- The reduction in downtime enabled by Astro will depend largely on the type of environment, prior investments, institutional knowledge, and maturity of organizations considering an investment in Astro.
- The cost of system downtime depends on the types of systems leveraging Airflow. Organizations that only use Airflow for minor and noncritical systems may not receive as much benefit from improved stability and data reliability of those systems.
Results. To account for these risks, Forrester adjusted this benefit downward by 25%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $532,000.
Improved Data Reliability And Organizational Security (Business Value)
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| C1 | Airflow uptime before Astro | Interviews | 99.99% | 99.99% | 99.99% | |
| C2 | Critical application downtime from Airflow incidents (hours, rounded) | 365*24*(1-C1) | 0.88 | 0.88 | 0.88 | |
| C3 | Average cost of application downtime of critical services per hour | Composite | $500,000 | $500,000 | $500,000 | |
| C4 | Subtotal: Annual cost of application downtime due to critical Airflow issues | Composite | $440,000 | $440,000 | $440,000 | |
| C5 | Reduction in critical Airflow issues with Astro | Interviews | 45% | 60% | 60% | |
| C6 | Reduction in MTTR of critical Airflow issues with Astro | Interviews | 20% | 25% | 25% | |
| C7 | Total percentage reduction in Airflow downtime of critical services | C5+((1-C5)*C6) | 56% | 70% | 70% | |
| Ct | Improved data reliability and organizational security (business value) | C4*C7 | $246,400 | $308,000 | $308,000 | |
| Risk adjustment | ↓25% | |||||
| Ctr | Improved data reliability and organizational security (business value) (risk-adjusted) | $184,800 | $231,000 | $231,000 | ||
| Three-year total: $646,800 | Three-year present value: $532,463 | |||||
Improved Airflow Stability And Data Visibility (Labor Efficiency)
Evidence and data. Astro also helped organizations avoid downtime for noncritical systems, which helped improve the quality of services and saved time that employees could reallocate to higher-value tasks.
- The vice president of data solutions and engineering at a SaaS organization found that before Astro, Airflow systems would have at least one issue a day — and potentially several. Each issue required staff to interrupt their work and spend time debugging and resolving. After Astro, Airflow systems were considerably more stable, and virtually 100% of these issues were eliminated.
- Similarly, a financial services organization used to spend 2 to 3 hours each week resolving issues with Airflow services. After they began using Astro, this was reduced to less than 1 hour per month, or roughly a 90% to 95% reduction in time spent resolving Airflow issues.
- When asked how Astro improved Airflow stability and data reliability, interviewees shared that Astro enabled best practices to be automatically integrated into their Airflow environments and allowed for the correct types of instances to be deployed, drastically reducing the chance of a failure. They shared that these features provided the greatest value to their teams and saved employees significant time in resolving Airflow issues.
- In addition, audit trails, metadata, operational dashboards, and telemetry included in Astro’s platform allowed teams to get ahead of potential issues before they interrupted services.
Modeling and assumptions. Forrester leveraged interview data to model the financial impact for the composite organization and assumes the following:
- Prior to Astro, the composite organization experiences 276 Airflow incidents to its noncritical systems each year. These issues require resolution, but they do not significantly impact the composite organization as long as they are resolved within a day.
- Prior to Astro, these issues require a team of eight data engineers to spend 1.3 hours on resolution on average.
- By Year 3, the number of incidents reduces by 73% with Astro, and the MTTR decreases by 67%.
- Data engineers have a fully burdened hourly wage of $55.
- Half of the time saved with improved Airflow stability and data reliability can be reinvested into tasks that provide the organization with greater value.
Risks. Forrester recognizes that these results may not be representative of all experiences and that the impact may vary depending on several factors:
- The reduction in downtime enabled by Astro will depend largely on the type of environment, prior investments, institutional knowledge, and maturity of organizations considering an investment in Astro.
- The size of resolution teams varied among interviewees depending on the severity of issues encountered and the organization’s resources. Organizations that do not spend significant time resolving Airflow errors may not receive as large of benefit from Astro.
Results. To account for these risks, Forrester adjusted this benefit downward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $135,000.
Improved Airflow Stability And Data Visibility (Labor Efficiency)
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| D1 | Noncritical Airflow incidents per year without Astro | Interviews | 276 | 276 | 276 | |
| D2 | MTTR of noncritical issues without Astro (hours) | Interviews | 1.3 | 1.3 | 1.3 | |
| D3 | FTEs involved per noncritical Airflow incident | Interviews | 8 | 8 | 8 | |
| D4 | Subtotal: Annual labor to resolve Airflow issues without Astro (hours, rounded) | D1*D2*D3 | 2,870 | 2,870 | 2,870 | |
| D5 | Reduction in the number of low-priority Airflow incidents with Astro | Interviews | 55% | 64% | 73% | |
| D6 | Noncritical Airflow incidents avoided with Astro (rounded) | D1*D5 | 152 | 177 | 202 | |
| D7 | Noncritical Airflow incidents remaining with Astro | D1-D6 | 124 | 99 | 74 | |
| D8 | Reduction in MTTR of noncritical incidents with Astro | Interviews | 34% | 51% | 67% | |
| D9 | Time saved with Astro per noncritical incident (hours, rounded) | D2*D8 | 0 | 0.7 | 0.9 | |
| D10 | Subtotal: Annual labor saved resolving noncritical Airflow incidents with Astro (hours, rounded) | (D2*D3*D6)+(D3*D7*D9) | 1,978 | 2,395 | 2,634 | |
| D11 | Reduction in labor hours to resolve noncritical Airflow incidents with Astro | D10/D4 | 69% | 83% | 92% | |
| D12 | Data engineer fully burdened hourly wage | TEI standard | $55 | $55 | $55 | |
| D13 | Time recaptured | Composite | 50% | 50% | 50% | |
| Dt | Improved Airflow stability and data visibility (labor efficiency) | D10*D12*D13 | $54,395 | $65,863 | $72,435 | |
| Risk adjustment | ↓15% | |||||
| Dtr | Improved Airflow stability and data visibility (labor efficiency) (risk-adjusted) | $46,236 | $55,983 | $61,570 | ||
| Three-year total: $163,789 | Three-year present value: $134,558 | |||||
Reduced Airflow Infrastructure And Management Costs
Evidence and data. Airflow reduced infrastructure costs in two ways. First, as Astro allowed for more appropriate types of Airflow instances to be deployed, the computing costs of these instances were slightly less than prior to Astro, when less-efficient instances were often used. Second, interviewees generally began moving Airflow computing from their own cloud or on-premises infrastructure to Astro’s cloud. In addition, fewer FTEs were required to manage this infrastructure, saving labor costs.
- The vice president of data solutions and engineering at a media company found that their infrastructure costs reduced by 60% after investing in Astro. This was in part because computing took place on Astro Cloud as opposed to the organization’s own infrastructure.
- The vice president of product and data of a SaaS organization found that cloud computing costs fell by 5% to 10% because Astro made best practices of Airflow deployment automatic and more easily allowed the correct type of deployment to be used.
- The tech lead and data architect of a SaaS organization said “You might think it’s easy to maintain Airflow on the cloud, but that’s not always true. The autoscaling could cost you more than you realize, and this is something we’ve been able to manage quite clearly with Astro.”
- The tech lead and data architect found that their infrastructure costs fell by 25% due to greater efficiencies from deploying Airflow through Astro. Data teams reinvested time managing this infrastructure into building new data pipelines, providing further value to the organization.
- Other interviewees found that Astro freed up time for one to two FTEs who no longer needed to spend as long managing Airflow infrastructure.
- The vice president of product and data of a SaaS organization said, “Airflow has very cool features and lots of bells and whistles, but there is a giant cost of managing it on your own.” The same interviewee continued: “Infrastructure costs reduced because Astro lets us control the instances types we assign, and different instances have different prices. We’ve been able to switch the types of deployments based on workloads, whereas before Astro, we had to use a generic instance for everything.”
Modeling and assumptions. Forrester leveraged interview data to model the financial impact for the composite organization and assumes the following:
- The composite organization spends $200,000 on Airflow cloud computing costs each year before Astro manages their Airflow platform.
- Cloud computing costs are reduced by 45% due to gaining efficiencies from and shifting cloud computing to Astro.
- One infrastructure engineer, who is paid $95,000 per year, saves 75% of their time managing Airflow due to efficiencies gained with Astro. This time is fully reinvested in managing non-Airflow infrastructure.
Risks. Forrester recognizes that these results may not be representative of all experiences and that the impact may vary depending on several factors:
- Organizations that did not leverage Astro’s cloud computing did not see as much efficiency as those whose computation still took place in their own infrastructure. However, they still observed some efficiency from automated improvements through Astro.
- The amount of time saved by infrastructure teams will likely depend on the environments and processes prior to an investment in Astro. Readers should consider the costs associated with their Airflow environments today and how time spent managing infrastructure could be reinvested.
Results. To account for these risks, Forrester adjusted this benefit downward by 20%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $200,000.
Reduced Airflow Infrastructure And Management Costs
| Ref. | Metric | Source | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|
| E1 | Airflow infrastructure costs before Astro | Composite | $200,000 | $200,000 | $200,000 | |
| E2 | Percentage reduction in infrastructure costs due to Astro | Interviews | 10% | 30% | 45% | |
| E3 | Reduction in infrastructure costs due to Astro | E1*E2 | $20,000 | $60,000 | $90,000 | |
| E4 | Saved time on ongoing Airflow infrastructure management | Interviews | 25% | 50% | 75% | |
| E5 | Infrastructure engineers required to maintain Airflow without Astro | Composite | 1 | 1 | 1 | |
| E6 | Infrastructure engineer fully burdened annual salary | TEI standard | $95,000 | $95,000 | $95,000 | |
| E7 | Reduction in Airflow management costs due to Astro | E4*E5*E6 | $23,750 | $47,500 | $71,250 | |
| Et | Reduced Airflow infrastructure and management costs | E3+E7 | $43,750 | $107,500 | $161,250 | |
| Risk adjustment | ↓20% | |||||
| Etr | Reduced Airflow infrastructure and management costs (risk-adjusted) | $35,000 | $86,000 | $129,000 | ||
| Three-year total: $250,000 | Three-year present value: $199,812 | |||||
Unquantified Benefits
Interviewees mentioned the following additional benefits that their organizations experienced but were not able to quantify:
- Protected reputation among customers, partners, and organizational stakeholders. Interviewees were under pressure from both internal and external stakeholders to correctly process data, promptly deliver it, and consistently offer support services. Astro helped ensure that each of these expectations was met and that the brand of the organization was protected.
- Enabled improved profit margin while scaling services. Interviewees in high-growth organizations noted that infrastructure costs often increased faster than revenue as services rapidly scaled, meaning operating profit suffered in the short term. Astro enabled these services to scale more efficiently and without interruption to service, and it alleviated large spikes in infrastructure costs to protect operating profit margin.
-
Improved number and quality of AI and ML services. Each interviewee worked with data teams developing new AI and ML capabilities, and Astro helped these teams with this work in two different ways. First, it more quickly and reliably provided high-quality training data, improving the effectiveness of development and allowing new models and features to be integrated faster. Second, Astro freed up employee time on Airflow management and maintenance, meaning more employees could be allocated to AI and ML projects and broadening the scope of the types of initiatives organizations and leadership could conduct.
The vice president of data solutions and engineering at a media company said: “Astro is empowering our ML teams as they build offline interference and subscription propensity models. They can use Astro to orchestrate the batch workflows as the data becomes available.”
- Accelerated access to new Airflow features. While interviewees had previously struggled to update their systems to new Airflow releases, Astro made this process considerably faster and allowed organizations to more quickly and reliably benefit from new features and security patches. This process was often delayed, and typically took several days or even weeks prior to the Astro investment, but interviewees were able to upgrade to the latest version of Airflow within one day with Astro. This not only saved labor as teams no longer needed to be interrupted to perform upgrades, but it also provided tangible business value as teams interacting with data benefited from additional Airflow capabilities and improved security posture with new releases.
- Provided better employee experience. Spending time managing Airflow and fixing issues not only reduced employees’ productivity — they also created frustrating working environments. Astro automated much of the management and improved Airflow stability, freeing employee time for more creative, engaging, and fulfilling work.
-
Ensured greater protection of PII and compliance with data privacy regulations. Interviewees, especially those with global operations, operated under strict and disparate data privacy regulations and faced the potential for large fines if data environments did not comply with all requirements. Astro enabled organizations to track data through their pipelines and help meet the strict requirements in each country of operation. Interviewees particularly valued the audit trails that Astro provided for Airflow processes, which made compliance easier to achieve.
Forrester has found that PII is the most common type of compromised data globally, supporting interviewees’ experience that Astro has helped them to avoid regulatory fines.2
The Forrester Data Privacy Compliance Model
Forrester has long advocated for organizations to invest in robust and thorough systems to ensure protection of their sensitive data, as illustrated in the Forrester Corporate Data Privacy Compliance Model.3 Interviewees found that Astro allowed them to improve many aspects of this model and better protect the personally identifiable data they had been trusted with:
- Data intake. Astro enabled better visibility into data sources and allowed IT groups to more easily map how data entered and moved through pipelines.
- Data storage. Interviewees shared that Astro seamlessly integrated into their existing cloud and on-premises infrastructure, meaning they could maintain existing systems and security protocols.
- Data usage. Centralized visibility of data usage, enabled through Astro, allowed stakeholders to better understand how data was being used and ensured that regulatory requirements were met.
- Data maintenance. The improved observability gained with Astro was used by a variety of teams, including those who were responsible for data maintenance. This not only saved labor but also helped improve data compliance and stakeholder confidence.
- Data disposal. Similarly, data teams could use Astro to ensure that they knew which data was being accessed and used and confirm whether this data met all regulatory requirements. Data that should have been disposed of but was still being accessed could more easily be flagged and promptly removed.
Flexibility
The value of flexibility is unique to each customer. There are multiple scenarios in which a customer might implement Astro and later realize additional uses and business opportunities, including:
- Consolidated tools and technology stack. Interviewees expected to consolidate and avoid fragmentation of their data technology stack in the future as Astro provided, or was expected to soon provide, many features and functionalities that they had previously paid third-party vendors to deliver. While this benefit of Astro had not yet occurred or been quantified by the four interviewees in this case study, it represents additional value that was expected in the future due to the flexible and extensible nature of Astro to meet the needs of data use cases.
- Improved security through deeper telemetry, less downtime, and faster investigations. While this analysis has already investigated the business value of improved Airflow stability through Astro, interviewees also found that their security positions were strengthened through the same improvements and features previously discussed. As data security concerns and threats grow, organizations and data teams will need better telemetry, understanding, and control of their data.4 Each of these improvements was provided by Astro to the interviewees and their organizations.
Flexibility would also be quantified when evaluated as part of a specific project (described in more detail in Appendix A).
Quantified cost data as applied to the composite
Total Costs
| Ref. | Cost | Initial | Year 1 | Year 2 | Year 3 | Total | Present Value |
|---|---|---|---|---|---|---|---|
| Ftr | Astro usage-based license cost | $0 | $57,500 | $133,400 | $164,450 | $355,350 | $286,074 |
| Gtr | Implementation and ongoing labor | $21,120 | $1,584 | $1,584 | $1,584 | $25,872 | $25,059 |
| Total costs (risk-adjusted) | $21,120 | $59,084 | $134,984 | $166,034 | $381,222 | $311,133 |
Astro Usage-Based License Cost
Evidence and data. Users paid for Astro on a usage-based model.
- Interviewees paid Astronomer for Astro on a consumption basis, which ranged between $20,000 and $120,000 per year.
- The data platform lead at a financial services company with 60 DAGs and more than $500 million in revenue said: “We pay about $120,000 per year for Astro. That’s money well spent, and among the vendor products I use in my data platform, Astronomer is the least expensive by a significant amount.”
- The vice president of product and data at a SaaS company said: “The ongoing cost of using Astro is lower than of not using Astro. On a day-by-day basis, using Astro is saving us money.”
Modeling and assumptions. Forrester leveraged interview data to model the financial impact for the composite organization and assumes the following:
- The composite organization begins with Astro managing its critical Airflow services in Year 1, and then expands Astro usage to all Airflow services in Year 2.
- The composite organization shifts from its own infrastructure to Astro Cloud over the three-year period. This cost has been incorporated into the usage-based costs in this analysis.
Risks. Forrester recognizes that these results may not be representative of all experiences and that the impact may vary depending on several factors:
- Usage-based license costs will depend largely on the types of workloads and services supported by Astro. Readers should carefully consider their own environment and likely costs rather than using these exact assumptions in their own analysis.
- The usage-based license costs will grow as the number and size of services using Airflow also grows.
Results. To account for these risks, Forrester adjusted this cost upward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $286,000.
Astro Usage-Based License Cost
| Ref. | Metric | Source | Initial | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|---|
| F1 | Astro license cost (including cloud computing) | Interviews | $0 | $50,000 | $116,000 | $143,000 | |
| Ft | Astro usage-based license cost | F1 | $0 | $50,000 | $116,000 | $143,000 | |
| Risk adjustment | ↑15% | ||||||
| Ftr | Astro usage-based license cost (risk-adjusted) | $0 | $57,500 | $133,400 | $164,450 | ||
| Three-year total: $355,350 | Three-year present value: $286,074 | ||||||
Implementation And Ongoing Labor
Evidence and data. Prior to using Astro, interviewees led discovery and change management in their organization to identify where Astro would provide the greatest impact in the shortest amount of time. A few hours of additional oversight was provided each month thereafter as Astro usage was expanded to additional services throughout the organization.
- The most significant type of labor associated with an investment in Astro was internal change management of systems and processes to ensure that Astro is properly being used. This labor was not spent on integration or any technical aspect of Astro, but rather on mapping processes and coordination across teams. Interviewees estimated this process took one FTE one to two months.
- The data platform lead at a financial services organization found it took less than two hours per month to maintain Astro across the organization. This time was shorter than other technologies as much of the management of Astro is automated and does not require frequent oversight.
- The same interviewee noted that the time spent managing Astro was significantly shorter than the two to three hours Astro saved each week managing Airflow.
Modeling and assumptions. Forrester leveraged interview data to model the financial impact for the composite organization and assumes the following:
- One data engineer initially spends 320 hours (approximately two months) on change management to effectively incorporate Astro into critical Airflow services.
- A data engineer spends an additional two hours each month on ongoing maintenance and expansion of Astro as it begins to manage and support more Airflow services.
Risks. Forrester recognizes that these results may not be representative of all experiences and that the impact may vary depending on several factors:
- Change management can be a lengthy and difficult process for a complex environment where many stakeholders and types of users are involved. Larger organizations with more data complexities and disparate groups may require additional effort to successfully integrate any new type of technology such as Astro. Readers should consider the change management required for similar types of projects in the past and plan accordingly as they build a business case for Astro.
- Some organizations, especially those with disparate groups, may require additional staff to manage Astro than what is assumed in this analysis. While the time to manage Astro is still expected to be minimal, there could be additional complexities coordinating efforts across teams.
Results. To account for these risks, Forrester adjusted this cost upward by 10%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $25,000.
Implementation And Ongoing Labor
| Ref. | Metric | Source | Initial | Year 1 | Year 2 | Year 3 | |
|---|---|---|---|---|---|---|---|
| G1 | Initial implementation and change management (hours) | Interviews | 320 | 0 | 0 | 0 | |
| G2 | Development operations engineer fully burdened hourly wage | TEI standard | $60 | $60 | $60 | $60 | |
| G3 | Subtotal: Labor for Astro implementation and internal change management | G1*G2 | $19,200 | $0 | $0 | $0 | |
| G4 | Annual labor to manage Astro (hours) | Interviews | 0 | 24 | 24 | 24 | |
| G5 | Subtotal: Ongoing Astro management | G2*G4 | $0 | $1,440 | $1,440 | $1,440 | |
| Gt | Implementation and ongoing labor | G3+G5 | $19,200 | $1,440 | $1,440 | $1,440 | |
| Risk adjustment | ↑10% | ||||||
| Gtr | Implementation and ongoing labor (risk-adjusted) | $21,120 | $1,584 | $1,584 | $1,584 | ||
| Three-year total: $25,872 | Three-year present value: $25,059 | ||||||
Consolidated Three-Year Risk-Adjusted Metrics
Cash Flow Chart (Risk-Adjusted)
Total costs Total benefits Cumulative net benefits-
The financial results calculated in the Benefits and Costs sections can be used to determine the ROI, NPV, and payback period for the composite organization’s investment. Forrester assumes a yearly discount rate of 10% for this analysis.
These risk-adjusted ROI, NPV, and payback period values are determined by applying risk-adjustment factors to the unadjusted results in each Benefit and Cost section.
Cash Flow Analysis (Risk-Adjusted Estimates)
| Initial | Year 1 | Year 2 | Year 3 | Total | Present Value | |
|---|---|---|---|---|---|---|
| Total costs | ($21,120) | ($59,084) | ($134,984) | ($166,034) | ($381,222) | ($311,133) |
| Total benefits | $0 | $433,326 | $707,564 | $923,441 | $2,064,331 | $1,672,491 |
| Net benefits | ($21,120) | $374,242 | $572,580 | $757,407 | $1,683,109 | $1,361,358 |
| ROI | 438% | |||||
| Payback | <6 months |
Appendix A: Total Economic Impact
Total Economic Impact is a methodology developed by Forrester Research that enhances a company’s technology decision-making processes and assists vendors in communicating the value proposition of their products and services to clients. The TEI methodology helps companies demonstrate, justify, and realize the tangible value of IT initiatives to both senior management and other key business stakeholders.
Total Economic Impact Approach
-
Benefits represent the value delivered to the business by the product. The TEI methodology places equal weight on the measure of benefits and the measure of costs, allowing for a full examination of the effect of the technology on the entire organization.
-
Costs consider all expenses necessary to deliver the proposed value, or benefits, of the product. The cost category within TEI captures incremental costs over the existing environment for ongoing costs associated with the solution.
-
Flexibility represents the strategic value that can be obtained for some future additional investment building on top of the initial investment already made. Having the ability to capture that benefit has a PV that can be estimated.
-
Risks measure the uncertainty of benefit and cost estimates given: 1) the likelihood that estimates will meet original projections and 2) the likelihood that estimates will be tracked over time. TEI risk factors are based on “triangular distribution.”
The initial investment column contains costs incurred at “time 0” or at the beginning of Year 1 that are not discounted. All other cash flows are discounted using the discount rate at the end of the year. PV calculations are calculated for each total cost and benefit estimate. NPV calculations in the summary tables are the sum of the initial investment and the discounted cash flows in each year. Sums and present value calculations of the Total Benefits, Total Costs, and Cash Flow tables may not exactly add up, as some rounding may occur.
Appendix B: Supplemental Material
Related Forrester Research
“Develop Data Privacy Metrics That Matter To The Business,” Forrester Research, Inc., August 2, 2021
“Seven Essential Capabilities To Enable DataOps for AI Development,” Forrester Research, Inc., February 8, 2023
Appendix C: Endnotes
1 Total Economic Impact is a methodology developed by Forrester Research that enhances a company’s technology decision-making processes and assists vendors in communicating the value proposition of their products and services to clients. The TEI methodology helps companies demonstrate, justify, and realize the tangible value of IT initiatives to both senior management and other key business stakeholders.
2 Source: “The State Of Data Security, 2023,” Forrester Research Inc., August 22, 2023.
3 Source: “The Forrester Data Privacy Compliance Model,” Forrester Research Inc., August 1, 2022.
4 Source: “The State of Data Security,” Forrester Research, Inc., August 22, 2023.
PDF 