Lesson 3 – Azure Databricks Architecture Explained: Workspaces, Compute, Storage and Unity Catalog

Introduction

In the previous lesson, we learned why organizations choose Azure Databricks over traditional data engineering tools. We saw how Databricks helps organizations process massive amounts of data, supports artificial intelligence (AI), and brings data engineers, analysts, and data scientists together on a single collaborative platform.

Now that we understand why Azure Databricks is used, the next logical question is:

How is Azure Databricks built behind the scenes?

Just like a house has different rooms serving different purposes, Azure Databricks is made up of several components that work together to provide a secure, scalable, and high-performance analytics platform.

Before we start creating notebooks or writing Spark code, it’s important to understand these components because you’ll encounter them almost every day while working with Azure Databricks.

In this lesson, we’ll explore:

• Azure Databricks architecture
• Control Plane and Compute Plane
• Workspaces
• Compute
• Storage
• Unity Catalog

By the end of this lesson, you’ll understand how these components work together to process data securely and efficiently.

What is Azure Databricks Architecture?

Architecture simply means how different parts of a system are designed and work together.

Azure Databricks consists of different components, each with a specific purpose.

Some components manage the platform.
Some process data.
Some store data.
Some secure data.
Together, they create a complete analytics platform.

High-Level Azure Databricks Architecture

At a high level, Azure Databricks consists of four major components:

• Workspace
• Compute
• Storage
• Unity Catalog

Behind the scenes, these components are divided into two environments:

• Control Plane
• Compute Plane (also called the Data Plane in classic deployments)

Let’s understand each component one by one.

Understanding the Control Plane

The Control Plane is the management layer of Azure Databricks and this is where u receive the request. It is fully managed by Microsoft and Databricks. You don’t install it, configure it, or maintain it. Whenever you open Azure Databricks in your browser, you’re interacting with the Control Plane.

It is responsible for:

• User authentication
• Workspace management
• Notebook management
• Cluster creation
• Job scheduling
• REST APIs
• Platform administration

Think of the Control Plane as the brain of Azure Databricks. It coordinates everything but doesn’t perform the actual data processing. TheControl Plane manages Azure Databricks but doesn’t process your data.

Understanding the Compute Plane

The Compute Plane is where your data is processed. Unlike the Control Plane, the Compute Plane runs inside your Azure subscription. When you run a notebook or execute a Spark job, Azure Databricks creates compute resources in your Azure environment.

These compute resources:

• Read data
• Transform data
• Execute Spark jobs
• Train machine learning models
• Write data back to storage

Because the Compute Plane runs within your Azure subscription, your organization’s data remains under your control.

Why is this important?

Many organizations work with sensitive information such as:

• Banking transactions
• Healthcare records
• Customer data
• Financial reports

Keeping the data processing inside the customer’s Azure environment improves security and helps meet compliance requirements.

Control Plane vs Compute Plane

Control Plane	Compute Plane
Managed by Microsoft	Runs in your Azure subscription
Hosts the web interface	Runs Spark clusters
Handles authentication	Processes your data
Schedules jobs	Executes notebooks
Manages clusters	Reads and writes data

A simple way to remember this is:

• Control Plane manages.
• Compute Plane processes.

What is a Workspace?

A Workspace is the place where users perform their daily work.

Inside a workspace, you can:

• Create notebooks
• Develop Spark applications
• Build SQL queries
• Schedule jobs
• Manage dashboards
• Track ML experiments
• Connect to Git repositories

A workspace allows multiple users to collaborate on the same project.

We may think,

“Why are the workspaces not inside the metastore?”

The answer is because a metastore is not a container for workspaces. Instead, a metastore is a governance service that can be assigned to one or more workspaces.

Why Do Organizations Create Multiple Workspaces?

Most organizations don’t use just one workspace. Instead, they create separate workspaces for different environments.

For example:

• Development
• Testing
• Production

This helps prevent experimental code from affecting production systems.

Types of Azure Databricks Workspaces

When you create an Azure Databricks workspace, one of the first decisions you need to make is choosing the workspace deployment type.

Azure Databricks currently supports two workspace types:

• Serverless Workspace
• Hybrid Workspace (formerly known as Classic Workspace)

Both provide the same Azure Databricks experience for developing notebooks, running SQL queries, building machine learning models, and managing data engineering workloads. The primary difference lies in how the underlying infrastructure is deployed and managed.

Serverless Workspace

A Serverless Workspace is the newest deployment model in Azure Databricks. It is designed to provide a fully managed experience by reducing the need to manage infrastructure.

Source: Microsoft Learn

When you create a Serverless Workspace:

• Azure Databricks automatically configures serverless compute.
• Workspace storage is managed using the workspace’s default storage.
• Microsoft manages the underlying infrastructure.
• You can still connect the workspace to your own Azure Data Lake Storage Gen2 account to access business data.

Because Azure Databricks manages the infrastructure, you can focus on writing notebooks, analyzing data, and building AI solutions instead of provisioning and maintaining compute resources.

Hybrid Workspace (Classic Workspace)

Source: Microsoft Learn

A Hybrid Workspace is the traditional Azure Databricks deployment model.

Unlike a Serverless Workspace, Azure provisions additional Azure resources within your Azure subscription when the workspace is created.

These resources include:

• Workspace storage account
• Compute resources
• Managed resource group
• Optional networking components

Serverless Workspace	Hybrid (Classic) Workspace
Infrastructure is managed by Azure Databricks.	Infrastructure is provisioned in your Azure subscription.
Uses default workspace storage.	Uses a dedicated workspace storage account.
Serverless compute is automatically managed.	Supports both classic compute and serverless compute.
Best for users who want a simplified, fully managed experience.	Best for organizations requiring greater control over networking, security, and infrastructure.

This deployment model provides greater control over networking, security, and infrastructure configuration.

Feature	Azure Databricks Workspace
What is a Workspace?	A collaborative environment where users create notebooks, run Spark jobs, build SQL queries, develop machine learning models, and manage data engineering workloads.
Where is it created?	Created as an Azure Databricks resource from the Azure Portal.
Can you create it inside another workspace?	No. Each workspace is an independent Azure resource and cannot be created from within an existing workspace.
Can an organization have multiple workspaces?	Yes. Organizations commonly create separate workspaces for Development, Testing, UAT, and Production.
What does a workspace contain?	Notebooks, folders, compute clusters, SQL Warehouses, dashboards, jobs, workflows, Git repositories, MLflow experiments, and workspace settings.
Who uses a workspace?	Data Engineers, Data Analysts, Data Scientists, Machine Learning Engineers, and SQL Analysts collaborate within the same workspace.
How is it managed?	Through the Azure Portal for provisioning and the Databricks UI for day-to-day development and administration.
Can workspaces share data?	Yes. Multiple workspaces can securely access the same Azure Data Lake Storage Gen2 account and can be governed using Unity Catalog.
Can workspaces share metadata?	Yes. Multiple workspaces can be attached to the same Unity Catalog Metastore, enabling centralized governance, permissions, and data lineage.
Why create multiple workspaces?	To isolate Development, Testing, and Production environments, improve security, simplify management, and prevent development changes from affecting production workloads.

Understanding Compute

Compute refers to the processing power used to execute workloads in Azure Databricks. Whenever you run code, Azure Databricks needs computing resources. These resources are called Compute. Azure Databricks provides different compute options depending on the workload.Azure Databricks primarily relies on a single Apache Spark engine for almost every workload. Whether you perform data engineering, machine learning, or analytics, Spark is the core processing engine.

All-Purpose Compute

Designed for interactive development.

Ideal for:

• Learning Spark
• Running notebooks
• Data exploration
• Debugging code

Since it remains available while users are working, it is commonly used by developers.

Job Compute

Job Compute is designed for automated production workloads. Instead of running continuously, it starts only when a scheduled job begins and automatically stops after the job finishes. This helps reduce cloud costs.

SQL Warehouses

SQL Warehouses are optimized specifically for SQL queries and reporting.

Business analysts use SQL Warehouses to:

• Run SQL queries
• Build dashboards
• Connect Power BI
• Generate reports

Serverless Compute

Serverless Compute removes the need to manage clusters. Azure Databricks automatically provisions the required infrastructure behind the scenes.

Benefits include:

• Faster startup times
• Automatic scaling
• Reduced operational overhead

Compute Comparison

Compute Type	Best Used For
All-Purpose Compute	Interactive development
Job Compute	Scheduled production jobs
SQL Warehouse	Reporting and BI
Serverless Compute	Fully managed workloads

Understanding Storage

One of the biggest strengths of Azure Databricks is that it separates processing from storage. Instead of storing business data inside the compute environment, data is typically stored in cloud storage such as Azure Data Lake Storage Gen2 (ADLS Gen2). This separation allows organizations to scale storage and compute independently.

DBFS (Databricks File System)Root

DBFS is a distributed file system provided by Azure Databricks that allows you to store and access files using a familiar file path. Every Databricks workspace includes a DBFS Root. It stores workspace-related files such as:

• Libraries
• Temporary files
• Logs
• Default MLflow artifacts

Although convenient, it is not recommended for storing production business data.

Azure Data Lake Storage Gen2

Production data is typically stored in ADLS Gen2.

Benefits include:

• Highly scalable
• Cost-effective
• Secure
• Supports structured, semi-structured, and unstructured data

Azure Databricks can read and write data directly from ADLS Gen2 without moving the data into the workspace.

Understanding Metastore and Unity Catalog

Before we learn about Unity Catalog, let’s first understand Metastore, because these two terms are closely related and are often confused.

Many beginners ask questions like:

• Is Unity Catalog the same as a Metastore?
• Why do people say, “Attach a Metastore to a Workspace”?
• Why doesn’t Microsoft simply say, “Attach Unity Catalog”?

Step 1 – What is Metadata?

Before understanding a Metastore, we need to understand Metadata.

Metadata simply means:

Step 2 – What is a Metastore?

A Metastore is a repository that stores metadata. It does not store the actual data.

Instead, it stores information such as:

• Catalog names
• Schema names
• Table names
• Column names
• Data types
• Storage locations
• Owners
• Permissions

Think of a metastore as a directory or an index that tells Databricks where the data is located and who can access it.

Step 3 – What is Unity Catalog?

Now that we understand the Metastore, let’s look at Unity Catalog. As organizations grow, managing permissions across multiple workspaces becomes challenging.

It provides a single place to manage:

• Data permissions
• Metadata
• Data lineage
• Auditing
• Data discovery
• Access control

across one or more Azure Databricks workspaces. Instead of managing data separately in every workspace, organizations can manage everything centrally through Unity Catalog. Earlier, each Databricks workspace maintained its own metadata using the Hive Metastore.

Note : Hive Metastore is the legacy metadata management system in Databricks.

This made governance difficult. Unity Catalog solves this by providing a centralized governance solution. Many people think Unity Catalog and Metastore are the same.

They are not. A Metastore only stores metadata. A Unity Catalog is a complete governance solution that uses a metastore and adds enterprise governance features. The Metastore is one component of Unity Catalog.

Why Do People Say, “Attach a Metastore to a Workspace”?

This is one of the most common questions.

Suppose an organization has three workspaces.

• Development Workspace
• Testing Workspace
• Production Workspace

Instead of every workspace maintaining its own metadata, all three workspaces can use the same Unity Catalog Metastore. All three teams need access to the same Sales and Customer data.

Without centralized governance, administrators would have to manage permissions separately in each workspace. This becomes difficult as the number of users, tables, and workspaces grows.

This is the problem that Unity Catalog solves.

With Unity Catalog, organizations can manage:

• Tables
• Views
• Volumes
• Permissions
• Data lineage
• Auditing

across multiple workspaces from a single location.

Can One Workspace Use Multiple Metastores?

No.

A workspace can be attached to only one Unity Catalog metastore at a time. However, one Unity Catalog metastore can be shared by multiple workspaces. This is the most common enterprise deployment model.

Unity Catalog organizes data using a three-level hierarchy:

This hierarchy makes it easier to organize data and apply permissions consistently across the organization.

How Everything Works Together

Let’s see how these components interact during a typical workflow.

1. A user signs in to the Azure Databricks workspace.
2. The Control Plane authenticates the user and manages the workspace.
3. The user opens a notebook.
4. A compute cluster executes the notebook
5. The cluster reads data from Azure Data Lake Storage Gen2.
6. Unity Catalog checks whether the user has permission to access the data.
7. The processed data is saved back to storage or used for reporting, machine learning, or analytics.

When administrators say:

“Attach the workspace to the Unity Catalog Metastore.”

they simply mean:

“Connect this workspace to the centralized metadata repository managed by Unity Catalog.”

The workspace is not inside the metastore, and the metastore is not inside the workspace. The workspace uses the metastore.

Best Practices

• Use separate workspaces for Development, Testing, and Production.
• Store production data in Azure Data Lake Storage Gen2 instead of DBFS Root.
• Use Job Compute for scheduled production workloads.
• Use SQL Warehouses for reporting and business intelligence.
• Implement Unity Catalog for centralized governance and security.
• Enable automatic cluster termination to reduce cloud costs.

Key Takeaways

• Azure Databricks separates platform management from data processing through the Control Plane and Compute Plane.
• A Workspace is where users create notebooks, jobs, dashboards, and collaborate on projects.
• Compute provides the processing power required to execute Spark workloads.
• Azure Data Lake Storage Gen2 is the recommended storage solution for production data.
• Unity Catalog provides centralized governance, permissions, and data discovery across the entire organization.

Conclusion

Azure Databricks is more than just a platform for running Spark code. It is a well-designed architecture that separates management, processing, storage, and governance into distinct components. This separation makes the platform secure, scalable, and suitable for enterprise workloads.

Understanding these architectural building blocks is essential before moving on to Spark programming, because every notebook you create, every job you run, and every dataset you process relies on these components working together.

In the next lesson, we’ll explore the Databricks Lakehouse Architecture and understand how it combines the best features of Data Lakes and Data Warehouses to create a unified platform for modern data engineering, analytics, and AI.

Tags	Azure Databricks
Official Databricks Blog	Unity Catalog Documentation
Microsoft Docs	Azure Databricks High-Level Architecture
Test Your Knowledge	Quiz