By Richard Louden, Head of Technology (Data) at Nimble Approach
This blog explores how and why the modern data tooling ecosystem is consolidating through the lens of the Modern Data Stack (MDS). Once celebrated for its flexibility and innovation, the MDS is now being reshaped by major players building unified, end-to-end platforms. We’ll look at what’s driving this shift, its potential impact, and whether organisations should take steps to manage it.
What is the Modern Data Stack?
The Modern Data Stack (MDS) is a term coined around the early 2010s, through a series of talks and blog posts that explored how the data tooling ecosystem was changing. It relates to the development and use of modular, integratable applications for extracting, transforming, and serving data, rather than large, monolithic analytical systems. While using multiple tools or applications to deliver analytical functionality might seem complex, this approach offered several clear advantages for both developers and organisations:
- Modern Architecture: Makes use of modern approaches that utilise cheaper storage methods or parallel processing.
- Interchangeable: Multiple applications operating in the same space reduces the chance of vendor lock-in.
- Skillset Availability: Utilises modern, open source languages and frameworks, making it easier to find developers.
- Innovation: Competition among providers leads to continued feature development and customer focus.
Alongside the rise of more specialised applications for tasks such as data visualisation, orchestration, and extraction came the development of cloud-native data platforms. These platforms aimed to fill the niche of organisations looking to take advantage of modern approaches without the overhead of managing multiple tools across development teams. They could handle most of the processes needed to turn data into insights, whilst still integrating easily with existing tools at minimal effort or cost.
This combination of simplicity and modern tooling drove lots of organisations to either add a central data platform into their existing MDS, or adopt it as their first move and buy in any additional MDS tooling that was required. However, their growing popularity and aggressive push for market share have raised concerns that they could effectively kill off the MDS, with knock-on effects for vendor lock-in, cost, and innovation.
How and Why is it Consolidating?
Consolidation in the technology space is nothing new, with organisations looking to buy in capability when they grow to a size where they struggle to incubate such ideas themselves. In the MDS space, this has mainly revolved around the two main players, Databricks and Snowflake, acquiring companies to enhance their platforms with new capabilities and features. Databricks alone has acquired at least 15 companies in the last few years, ranging from dashboarding software to AI startups and serverless database providers, in order to continue building its unified platform. The important point here is that these acquisitions clearly expand the breadth of the platform’s capability, as they seek to handle the end-to-end process rather than specialise in a given step. Another notable example is the recent FiveTran–dbt merger, which aims to create a more streamlined platform for data extraction and transformation. Again, this all points towards a move to doing as much as possible, rather than being the best in a given area.
In addition to the acquisition approach, the MDS space is also consolidating through cloud providers releasing their own platforms. The most comprehensive example of this is currently Microsoft Fabric, which utilises a mixture of their storage, compute, automation, and visualisation offerings with claims of low code development and deployment experiences. While this approach doesn’t remove existing capabilities – tools like Power BI and Data Lake storage remain available – it does create subtle barriers to using them independently rather than through Fabric.
Both of these approaches require substantial resources – whether it’s the capital needed to acquire other companies or the time and effort to build and market a new platform. So why do it? Ultimately, it’s about expanding market share, and with it, revenue and profit. To achieve that, providers need platforms that are user-friendly, cost-effective, and feature-rich. This drive has only intensified with the rise of large language models, as vendors leverage the AI boom to attract attention and accelerate growth.
What Does This Mean for End Users?
On the face of it, consolidation may seem a positive step for some – simplifying what could be a rather complex technical tooling landscape for an organisation’s data operations. This perception is reinforced by the fact that many of these consolidated products still use widely available coding languages and maintain open-source components, such as dbt Core. However, consolidation also increases the risk of vendor lock-in, reminiscent of the large database systems of the 1990s and early 2000s. As major platform players add more features, migrating away becomes increasingly difficult, both because organisational processes become tightly coupled with the platform and because the overall number of tools on the market has decreased.
The consequences are threefold:
- Rising costs: With fewer alternatives, organisations have limited options to control expenses beyond reducing usage.
- Slower innovation: As platforms increase their market share, competitive pressure reduces and feature development slows down as a result.
- Reduced flexibility: A less malleable ecosystem to innovate within, forcing developers to create workarounds with what they have instead of integrating new tooling.
Should Organisations Do Anything About It?
Whilst there is clearly a move by the larger platform players to acquire data tooling and build out their platforms, this shouldn’t come as a shock to most or inspire any rash actions. These moves are primarily aimed at adding new features to support existing users and attract new ones, and for now, there’s little evidence of widespread vendor lock-in.
That said, as these providers continue to grow and capture more market share in data, analytics, and AI, organisations can take proactive steps to stay prepared:
- Optimise data processes: Build data processes to be as efficient and lean as possible, in order to counteract any potential future price increases.
- Evaluate feature use carefully: Consider the impact of utilising certain features across the wider organisation. What may look like a quick win at first could become a costly mistake down the line.
- Document alternative options: Examine alternative options for key processes where feasible. Even if these are not pursued, document this process in case there is a desire to migrate in the future.
- Stay informed: Keep up to date with the data tooling ecosystem and feed this into your technology and data strategies, understanding where weaknesses may exist in your architecture and how to remediate them.
Key Takeaways
The Modern Data Stack is consolidating as major platform providers acquire and integrate tools to expand their capabilities, particularly around AI. This shift simplifies management and accelerates innovation in the short term but also raises risks of vendor lock-in, higher costs, and slower progress over time. Organisations shouldn’t react hastily, but plan for flexibility – keeping data processes portable where possible, limiting reliance on proprietary features, and monitoring pricing and interoperability changes. In a tightening ecosystem, awareness and adaptability will be key to maintaining control and resilience.
This approach is often discussed with our clients when trying to understand their data maturity and how to enhance it. We take the view that the consolidation of the MDS has both positives and negatives for an organisation, and these need to be understood and accounted for in any decision making.
