By Richard Louden, Head of Technology (Data) at Nimble Approach
This blog explores why organisations are moving to a lakehouse architecture from a data warehouse, what the future holds for the lakehouse, and how they enable organisations to better adopt AI.
The Historic Data Warehouse
As businesses digitised in the late 20th century, OLTP databases (Online Transactional Processing) became the backbone for the various applications that sprung up to support organisational processes. As you would expect, this type of database is highly optimised for fast, repetitive tasks, such as processing a sales transaction or storing user data.
A consequence of this increased use of technology was improved access to organisational data, which had previously been trapped on paper and required hours of manual effort to utilise. As such, there was now a much lower barrier to entry for organisations looking to better understand key aspects of their operations and provide this analysis to the workforce through automated reporting. However, although OLTP databases made the data available, there were two key barriers to immediately adopting them for analytics. First, the data remained fragmented across the associated applications. Second, OLTP systems are not designed to support analytical workloads efficiently. In fact, attempting to transform and aggregate data within OLTP systems posed risks beyond poor performance, as it could prevent applications from writing data and disrupt day-to-day operations.
This friction forced a fundamental architectural split, leading to the design and creation of OLAP (Online Analytical Processing) systems such as data warehouses. These were created specifically to store, transform and analyse large quantities of data, with various integrations into the source OLTP databases. As organisations implemented this type of technology, either as a stand alone application or as part of larger ERP systems, they were able to create unified reports and better understand their operations.
Given the time in which they were developed, the majority of OLAP systems started as on premise deployments, requiring physical changes to hold more data or add compute. Eventually, these systems were migrated to cloud-based solutions, where scalability was less of a constraint. However, this shifted expenditure from capital investment to operational costs, often increasing the overall cost of ownership. Data warehouses remained in this state for many years, fulfilling the analytical and reporting requirements that they were designed to handle. However, as organisations continued to amass more and more data – partly driven by the desire to adopt data science methods – the underlying cost grew. With few advances in the underlying technologies or architectures, data warehouses began to strain under ever-increasing workloads, and their return on investment gradually diminished.
The Rise of the Lakehouse
As data warehouses continued to be a somewhat unified source of an organisation’s data in the background, new technologies designed to handle their drawbacks started to develop. A foundational element of these emerging solutions was the proliferation of data lakes, which provided significantly lower-cost storage than traditional databases. Combined with new technologies that delivered data warehouse levels of security, consistency, and performance, this gave rise to the lakehouse architecture. With this hybrid technology, organisations now had access to a storage method that was incredibly cheap (pennies per GB) but would maintain core rules around data reliability and consistency. This could be paired with burstable compute sources, which were readily available in the public cloud, to provide a much more cost effective way to store and analyse large amounts of data.
A key aspect of this new architecture was the growth of open-source technologies, driven by large technology companies and data platform providers such as Databricks. The latter had continued impact as they built their platform to take advantage of the lakehouse, and made it a core component to how organisations manage data. Critical to this was the development of governance features, including access control, data masking, and lineage across the entire lakehouse. These features were essential for creating a secure and unified data layer, yet would not have been possible using a data lake alone.
More recently, the platform’s functionally has expanded to also manage transactional workloads via the lakehouse. This allows organisations to manage all of their data in a single place, with minimal integrations and movement. As a result, organisations can now leverage lakehouse platforms to support a wide range of data-centric use cases, benefiting from their cost-effectiveness and comprehensive set of modern capabilities.
Lakehouses are the Foundation to AI Adoption
One key area where lakehouses excel is the organisational adoption of Artificial Intelligence (AI), where they play a key role in providing the underlying context needed to support off the shelf models. As organisations generate both structured and unstructured data – including accounts, orders, documentation, images, and more – any AI implementation requires a foundational platform capable of storing and providing rapid access to both data types.
Lakehouses fulfil this role by combining scalable storage and high-performance access with robust governance over the underlying data. As the use of AI continues to grow, this governance layer will only grow in importance as regulators add increased pressure to ensure auditability of what data organisational AI applications are accessing. Managing this across multiple databases, data warehouses, and data lakes would be highly complex. However, a well-governed lakehouse provides a unified storage layer that significantly reduces this complexity, allowing organisations to focus on developing valuable applications.
In addition to the core capability of the lakehouse, the ongoing development in this space by data platform providers is further enhancing their use for AI use cases. As mentioned in the previous section, there is now the ability to manage both transactional and analytical workloads via a lakehouse, through products such as Databricks’s LakeBase. These advancements allow AI models to rapidly access historic and recent tabular data, alongside context from unstructured sources, to provide the most accurate insights and support automated decision making.
Summary
Over the last 30 years, we at Nimble Approach have seen a shift in the way that organisations manage and use their data, driven by the explosion in the amount of data that is generated and how it can be utilised to support decision making.
More recently, we have seen a significant market shift towards the adoption of data lakehouses, which combine the cost-effective storage capabilities of data lakes with the reliability, governance, and consistency of traditional data warehouses. When combined with the advanced capabilities offered by modern data platforms, this approach enables organisations to better manage, govern, and leverage their rapidly growing volumes of data to support increasingly sophisticated reporting and analytics use cases.
Interestingly, this transition only looks to continue, with recent advancements in this area allowing lakehouses to provide the foundational capability needed for organisations to adopt AI. We believe that organisations embracing this shift will be better positioned to establish a robust data foundation, enabling them to accelerate the journey from AI experimentation to the deployment of production-ready, AI-enabled processes.
