Lesson 24 – Migration Path for Azure Data Factory to Microsoft Fabric

As organizations continue to evolve their data integration strategies, the need for seamless migration solutions becomes vital. Microsoft Fabric offers a great solution, especially if you’re moving from Azure Data Factory (ADF). In this blog post, we’ll break down the steps and advantages of shifting from ADF to Microsoft Fabric in a simple way.

Microsoft Fabric Data Factory

Microsoft Fabric Data Factory serves as the integral data integration component within the unified Microsoft Fabric platform dedicated to data and analytics. Combining the strengths of Azure Data Factory and the flexibility of Power Query Dataflows, it enables users to seamlessly handle data ingestion, transformation, and orchestration from diverse sources.

Whether using a low-code, no-code interface, or employing custom code through scripts or notebooks, Microsoft Fabric Data Factory offers a versatile and comprehensive solution. Its seamless integration with other Fabric services, including Purview, Data Pipelines, and AI Insights, further enhances its capabilities.

Foundations of Microsoft Fabric Data Factory

• Data and Transformation with Dataflow Gen2 – Microsoft Fabric’s Data Factory integrates Power Query and Azure Data Factory, providing a modern data integration experience. Dataflow Gen2, a pivotal component, leverages Fabric compute engines for scalable data processing and transformation, enabling organizations to handle data at any scale.
• Data Orchestration – The familiarity of Azure Data Factory’s data pipelines is seamlessly extended into Microsoft Fabric, offering robust orchestration capabilities. These pipelines support a wide range of activities and empower users to build simple to complex data workflows to meet diverse integration needs.
• Enterprise-ready data movement – Microsoft Fabric’s Data Factory ensures serverless and intelligent data movement, accommodating both small-scale and petabyte-scale scenarios. With support for 170+ connectors, it facilitates data movement across multiclouds, on-premises sources, and virtual networks. Intelligent throughput optimization automatically adapts to the required compute resources for efficient data movement.

Benefits of Migrating to Microsoft Fabric Data Factory

Migrating from Azure Data Factory (ADF) to Fabric is beneficial for data integration and analytics scenarios because Fabric offers a unified, easy-to-use, and powerful data platform that enables you to solve complex data challenges. Some of the benefits of Fabric are

• Fabric provides a unified and powerful data platform, making migration from Azure Data Factory (ADF) advantageous for solving complex data challenges.
• Utilize Data Factory in Fabric to effortlessly craft data pipelines seamlessly integrating with the Fabric Lake house, data warehouse, and other data services.
• Leverage Dataflow Gen2 in Fabric for building and executing data transformations with a user-friendly interface, driven by Power Query and Apache Spark.
• The copy assistant in Fabric facilitates quick and easy data copying from diverse sources to designated destinations, incorporating built-in AI for automating common tasks.
• Explore the Fabric workspace for discovering, sharing, and collaborating on data assets organization-wide, promoting seamless data exchange and reducing integration time and effort.

Migration Features for Azure Data Factory to Microsoft Fabric

Microsoft Fabric offers a set of features and capabilities that facilitate a smooth and efficient migration path from ADF to Microsoft Fabric. These features include

• Enhanced Data Pipeline Activities – Supports familiar ADF activities in Microsoft Fabric Data Factory. Introduction of new activities for notifications, including Teams and Outlook.
• OneLake/Lakehouse Connector Integration -Seamless integration for ADF customers to connect with Microsoft Fabric’s OneLake. Facilitates the smooth transfer of data from Azure Data Factory to Fabric OneLake.
• Mapping Dataflow Guidance – Comprehensive guide for ADF customers contemplating new data transformations in Microsoft Fabric. Assistance in understanding and implementing Azure Data Factory Mapping Dataflows in Fabric.
• Code Conversion Assistance – The Fabric Customer Advisory Team (Fabric CAT) provides sample code to help you transform your mapping dataflows into Spark code, allowing easy integration with Microsoft Fabric.
• Mounting of ADF in Fabric (Upcoming Feature) – Expected to be available in Q2 CY2024. Enables customers to mount existing ADF in Microsoft Fabric. All ADF pipelines will continue running on Azure while providing an opportunity to explore Fabric for a comprehensive upgrade plan.
• Upgrade from ADF Pipelines to Fabric (Upcoming Feature) – Soon, you’ll have the ability to test and upgrade your current data pipelines from Azure Data Factory (ADF) to Microsoft Fabric. This feature, currently in development with input from customers and the community, aims to simplify the upgrade process, ensuring a smooth transition from ADF to Microsoft Fabric.

These features collectively form a robust set of tools and capabilities, facilitating a smooth and efficient migration path from Azure Data Factory to Microsoft Fabric.