Lakehouse: the meeting point between Data Lake and Data Warehouse

Data play a critical role in organizations nowadays: it enables to understand the customer and deliver better experiences, anticipates unexpected situations in the market, establishes alternative supply chains in case of a disruption, optimizes processes, launches customized products and services, simulates business changes in virtual environments and replicates them in the real world only when it is proven to work.

Indeed, the companies that manage to capture, manage and exploit their data in the best possible way are the ones most likely to compete and lead in this increasingly competitive and changing environment. Until now, there seemed to be two different ways to reach this goal: using a data warehouse as a repository or betting on the data lake model.

Two repository models

Data warehouse had been focused on structured data analysis, SQL and transaction resolution supported by ACID-compliant databases (atomicity, consistency, isolation and durability, i.e., the set of properties that ensure that database transactions are processed reliably). Some of its main benefits? It is easy to use and understand by all types of users, including operational users, and promotes clear data governance.

Data lake, on the other hand, arrived promising greater flexibility: its ability to store structured, semi-structured and unstructured data made it the ideal repository alternative for emerging solutions linked to data science, machine learning or real-time databases. It includes a number of advantages, such as its ability to use data in its native format, with no fixed limits in terms of size or type. It is a fast-growing market: SNS Insider estimates growth of no less than US$12 billion worldwide in 2022 to US$57 billion in 2030.

Each has its strengths, but also its weaknesses: data warehouse showed significant shortcomings in supporting or integrating with advanced data engineering solutions, while data lake «suffers» with data quality, transactional support, governance issues and query performance problems. In fact, many specialists warn that a poorly managed data lake can eventually turn into a «data swamp,» a situation that is difficult to reverse, especially when the volume of data continues to increase at the speed it does today.

The best of both worlds

So why not use the best of both worlds in a new model? Indeed, a lake house gives us the possibility of maintaining huge data warehouses, without any restrictions in terms of formats or data types and with the guarantee that everything will be easy to maintain and operate, even by users with little specific data knowledge.

As its name suggests, a data lake house integrates and unifies a data warehouse and a data lake. It is an architecture that has a platform that manages the input of data to a storage layer based on a data lake. It also has a processing layer, and has flexibility for queries linked to the world of data science and for robust, efficient and simple database transactions with ACID principles, since the query engines are directly connected to the data lake.