Lesson 6 – What is Microsoft Fabric Lakehouse?

Microsoft Fabric Lakehouse is built upon the foundational framework of OneLake. It operates as a comprehensive database that stores both structured data (Data warehouse) and unstructured data (Data Lake) within a unified location. This allows you to effortlessly store, manage, and analyze both types of data, all conveniently centralized in a single location.

For example, an organization may already use a data warehouse for structured information from transactional systems but also need to analyse website logs, application telemetry, documents, or other less structured information. Managing these separately can make analytics more complex.

Microsoft Fabric Lakehouse provides an approach where different types of data can be stored and analysed together. Lakehouse tables use the Delta Lake open table format in OneLake. Delta Lake provides capabilities such as ACID transactions, schema enforcement, and time travel while keeping the underlying data in an open format.

Microsoft Fabric Lakehouse allows you to analyze various data formats (Structured, Unstructured and Sem-Structured) seamlessly.

Microsoft Fabric Lakehouse combines

• The scalable nature of a Data Lake – allows it to effortlessly expand and accommodate growing volumes of data.
• The query and analysis functionality of Data Warehouse – consolidates and organizes various data sources to simplify the process of analysis and reporting.

Explore Fabric Lakehouse

To create a new lakehouse, check out the “Getting Started with Microsoft Fabric Lakehouse lesson- 7” blog for step-by-step guidance.

A Fabric Lakehouse is a modern approach in handling data that blends the strengths of data warehouses and data lakes. It supports various data types, open formats, and economical storage, it also facilitating quick queries, ensuring data quality, and supporting governance.

(Lineage view- workspace page)

(List View – Workspace page)

What is created with a Lakehouse?

When a lakehouse is created in Microsoft Fabric, it consists of two distinct items:

• The lakehouse includes both storage and metadata, serving as the place where you work with files, folders, and table data.
• The SQL analytics endpoint acts as a service layer, during creation, automatically generates a SQL endpoint and a default dataset.

You can interact with the data in the lakehouse

1. Lakehouse mode-With the Lakehouse, you can add and work with tables, files, and folders.
2. SQL analytics endpoint mode – The SQL Endpoint provides the ability to use SQL for querying tables in the lakehouse and managing its relational data model.

The SQL analytics endpoint provides a read-only T-SQL query surface over Lakehouse Delta tables. You can query the data and create SQL objects such as views, functions, and stored procedures, but INSERT, UPDATE, and DELETE operations on the underlying Lakehouse tables aren’t supported through the SQL analytics endpoint. Data changes are typically performed using Spark or other Lakehouse data ingestion and transformation tools.

Approaches for working with the Lakehouse

Upon creating a Lakehouse, the default landing page is the Lakehouse Explorer. You can work with Lakehouse data in several ways depending on your role and preferred tool.

Lakehouse Explorer – The Lakehouse Explorer allows you to work directly with the Tables and Files areas of the Lakehouse. It provides an easy way to browse, manage, and load data.

Apache Spark and Notebooks – Data engineers and data scientists can use Spark notebooks to read, transform, analyse, and write data in the Lakehouse using languages such as Python, SQL, Scala, and R.

SQL analytics endpoint – Data analysts and SQL users can use the SQL analytics endpoint to query Lakehouse Delta tables using familiar T-SQL syntax.

Lakehouse Items

• New Dataflow Gen2- Dataflow Gen 2 facilitates the ingestion and preparation of data for data engineers. You have the ability to create a dataflow to execute additional transformations using Power Query and, if desired, store the transformed data back into the Lakehouse.
• New Copy Job – Copy data from various on-premises or cloud data sources into the Lakehouse with minimal configuration.

• Open notebook – Notebooks offer a platform to write code, enabling them to read, transform, and write data directly to the Lakehouse as tables and/or folders.
• New shortcut – A new shortcut feature allows you to create a shortcut to any file or folder in your Lakehouse. You can use new shortcuts to quickly access your frequently used data sources, pipelines, notebooks, or other items. The shortcut appears within the Lakehouse while the underlying data remains in its original location. This helps reduce unnecessary data duplication.
• New Eventstream – The Event streams feature in the Microsoft Fabric Real-Time Intelligence experience allows you to bring real-time events into Fabric, transform them, and route them into your Lakehouse.

Lakehouse Schemas

Lakehouse schemas provide a way to logically organise tables into business or functional groups.

For example:

Sales.Customers
Sales.Orders
Finance.Transactions
HR.Employees

Schemas are enabled by default when creating new Lakehouses through the Fabric portal and can help improve organisation and access management as the number of Lakehouse tables grows. Every schema-enabled Lakehouse includes the default dbo schema.

OneLake Storage Tiers and Lifecycle Management (Preview)

OneLake now supports Hot (frequently accessed data), Cool (infrequently accessed data), and Cold (rarely accessed data that needs longer-term retention) storage tiers, allowing organizations to optimize storage costs based on how frequently data is accessed. With Lifecycle Management, files can be automatically moved between tiers based on their age or usage, helping reduce storage costs while keeping data available.

Choosing Between Warehouse and Lakehouse

Microsoft Fabric lets you use both Warehouse and Lakehouse in the same workspace. A Lakehouse is ideal for Spark-based data engineering and large-scale analytics, while a Warehouse is best for SQL-based reporting and business intelligence on curated data.

Benefits of Fabric Lakehouse

The following are some of the benefits of Fabric Lakehouse

• Centralized Data Repository – A lakehouse serves as a single, consolidated storage space for both structured and unstructured data, eliminating the need for separate storage systems.
• Scalability- It can effortlessly grow to accommodate increasing volumes of data, ensuring scalability to meet evolving business needs.
• Unified Location – Lakehouse serve as a centralized hub where data engineers, data scientists, and data analysts can collectively access and utilize data.
• Flexibility in Data Types – Lakehouses accommodate diverse data types, allowing for the storage and analysis of structured and unstructured data in the same environment.
• Open Data Format – Lakehouse tables use the open Delta Lae format, making the data accessible across different analytical engines.
• Reduced Data Duplication– Lakehouses prevent duplication by storing and managing diverse data types in a single location using the open and standard Delta Lake format.
• Integration with Power BI – Power BI semantic models can use Direct Lake to query Delta tables in OneLake directly, reducing the need to import another copy of the data and eliminating traditional import refresh cycles for those tables.
• Cost Efficiency – Lakehouses reduce the total cost of ownership by leveraging low-cost object storage and optimizing both data processing and query performance.