Lesson 24 – Migration Path for Azure Data Factory to Microsoft Fabric

As organizations modernize their data integration processes, many are exploring Microsoft Fabric as an alternative to Azure Data Factory (ADF). Microsoft Fabric provides tools that make the migration process easier and more streamlined.

In this blog, we will explain the key steps and benefits of moving from ADF to Microsoft Fabric in a simple and beginner-friendly way.

Important Update

Microsoft has not announced any retirement date or migration deadline for Azure Data Factory.Both Azure Data Factory (ADF) and Fabric Data Factory continue to be fully supported.

Microsoft currently recommends:

• Migrating existing ADF solutions gradually and at your own pace using the available migration tools.
• Using Fabric Data Factory for new projects.

Microsoft Fabric Data Factory

Microsoft Fabric Data Factory serves as the integral data integration component within the unified Microsoft Fabric platform dedicated to data and analytics. Combining the strengths of Azure Data Factory and the flexibility of Power Query Dataflows, it enables users to seamlessly handle data ingestion, transformation, and orchestration from diverse sources.

Whether using a low-code, no-code interface, or employing custom code through scripts or notebooks, Microsoft Fabric Data Factory offers a versatile and comprehensive solution. Its seamless integration with other Fabric services, including Purview, Data Pipelines, and AI Insights, further enhances its capabilities.

Foundations of Microsoft Fabric Data Factory

• Data and Transformation with Dataflow Gen2 – Microsoft Fabric’s Data Factory integrates Power Query and Azure Data Factory, providing a modern data integration experience. Dataflow Gen2, a pivotal component, leverages Fabric compute engines for scalable data processing and transformation, enabling organizations to handle data at any scale.
• Data Orchestration – The familiarity of Azure Data Factory’s data pipelines is seamlessly extended into Microsoft Fabric, offering robust orchestration capabilities. These pipelines support a wide range of activities and empower users to build simple to complex data workflows to meet diverse integration needs.
• Enterprise-ready data movement – Microsoft Fabric’s Data Factory ensures serverless and intelligent data movement, accommodating both small-scale and petabyte-scale scenarios. With support for 200+ connectors, it facilitates data movement across multiclouds, on-premises sources, and virtual networks. Intelligent throughput optimization automatically adapts to the required compute resources for efficient data movement.

Benefits of Migrating to Microsoft Fabric Data Factory

Migrating from Azure Data Factory (ADF) to Fabric is beneficial for data integration and analytics scenarios because Fabric offers a unified, easy-to-use, and powerful data platform that enables you to solve complex data challenges. Some of the benefits of Fabric are

• Fabric provides a unified and powerful data platform, making migration from Azure Data Factory (ADF) advantageous for solving complex data challenges.
• Utilize Data Factory in Fabric to effortlessly craft data pipelines seamlessly integrating with the Fabric Lake house, data warehouse, and other data services.
• Leverage Dataflow Gen2 in Fabric for building and executing data transformations with a user-friendly interface, driven by Power Query and Apache Spark.
• The copy assistant in Fabric facilitates quick and easy data copying from diverse sources to designated destinations, incorporating built-in AI for automating common tasks.
• Explore the Fabric workspace for discovering, sharing, and collaborating on data assets organization-wide, promoting seamless data exchange and reducing integration time and effort.
• Built-in AI capabilities with Copilot to help create and manage pipelines, and a formal, assessment-first migration experience (rather than only manual rebuilding) that lets you evaluate readiness before moving any workloads

Migration Features for Azure Data Factory to Microsoft Fabric

Microsoft Fabric offers a set of features and capabilities that facilitate a smooth and efficient migration path from ADF to Microsoft Fabric. These features include

• Enhanced Data Pipeline Activities – Supports familiar ADF activities in Microsoft Fabric Data Factory. Introduction of new activities for notifications, including Teams and Outlook.
• OneLake/Lakehouse Connector Integration -Seamless integration for ADF customers to connect with Microsoft Fabric’s OneLake. Facilitates the smooth transfer of data from Azure Data Factory to Fabric OneLake.
• Mapping Dataflow Guidance – Comprehensive guide for ADF customers contemplating new data transformations in Microsoft Fabric. Assistance in understanding and implementing Azure Data Factory Mapping Dataflows in Fabric.Dataflow Gen2 for governed, low-code rowset transformations; Fabric Warehouse SQL for set-based ELT/MERGE operations; or Spark notebooks for advanced, large-scale transformations.
• Code Conversion Assistance – The Fabric Customer Advisory Team (Fabric CAT) provides sample code to help you transform your mapping dataflows into Spark code, allowing easy integration with Microsoft Fabric.
• Azure Data Factory (Mount)-Microsoft Fabric now includes the Azure Data Factory (Mount) feature, which lets you connect your existing Azure Data Factory (ADF) instance directly to a Fabric workspace.After mounting your ADF instance, you can view and manage your existing pipelines, dataflows, triggers, and integration runtimes from within Microsoft Fabric.

Key points:

1. Your ADF resources remain in Azure they are not copied or moved.
2. Existing pipelines continue to run on Azure infrastructure.
3. Billing continues through Azure, not Microsoft Fabric capacity.
4. Self-Hosted Integration Runtimes continue to work without changes.
5. You can unmount the ADF instance at any time without affecting your original Azure Data Factory.

• Migrate ADF Pipelines to Microsoft Fabric (Preview)-Microsoft Fabric also provides a Migrate to Fabric (Preview) feature that helps you move your Azure Data Factory pipelines to Fabric in a guided and controlled way.Instead of migrating everything at once, you can review your pipelines, identify any compatibility issues, and migrate only the pipelines you choose.

The migration process typically follows these steps:

1. Assess your ADF pipelines for compatibility with Microsoft Fabric.
2. Mount your Azure Data Factory instance into a Fabric workspace.
3. Select and migrate the pipelines you want to move.
4. Validate the migrated pipelines and manually enable triggers, which are disabled by default for safety.

These features collectively form a robust set of tools and capabilities, facilitating a smooth and efficient migration path from Azure Data Factory to Microsoft Fabric.