Skip to content

Latest commit

 

History

History
902 lines (622 loc) · 42.1 KB

File metadata and controls

902 lines (622 loc) · 42.1 KB

Deployment Guide for Medallion Architecture and PBI in Fabric

Deploy the Unified Data Foundation with Fabric solution accelerator using Azure Developer CLI - get a complete data platform with medallion architecture in minutes.


Key Sections

Section Description
Prerequisites Required permissions, tools, and setup
Deployment Overview Overview of deployed resources and architecture
Deployment Options Local, cloud, and CI/CD deployment methods
Deployment Commands One-command deployment instructions
Deployment Results Expected outcomes and verification steps
Advanced Configuration Options Optional customization parameters
Known Limitations Important constraints to review
Environment Cleanup How to remove deployed resources
Additional Resources Support and further reading

Alternative Deployment Methods

This guide focuses on automated deployment using Azure Developer CLI. For manual deployment or existing Fabric capacity integration, refer to the Manual Deployment Guide.


1. Prerequisites

To deploy this solution, ensure you have the following tools and permissions.

Software Requirements

You need these tools installed to run the deployment commands.

Tool Version Purpose Download
Azure Developer CLI Latest Orchestrates deployment Install azd
Azure CLI Latest Authentication Install az
Python 3.9+ Fabric configuration scripts Install Python

💡 Tip: You can skip installing tools by using Azure Cloud Shell or GitHub Codespaces.

Permissions

Your deployment identity (User or Service Principal) requires the following permissions.

🔐 Azure Permissions

🔗 API Permissions

  • Microsoft Graph API - User.Read: Delegated permission to read signed-in user profile information using Microsoft Graph User permissions
  • Microsoft Graph API - openid: Delegated permission for sign in and user profile authentication using OpenID Connect scopes
  • Fabric REST API - Workspace Management: Access to create and manage Fabric workspaces for workspace structure deployment using Fabric workspace APIs
  • Fabric REST API - Item Creation: Access to create lakehouses, notebooks, and reports for Fabric content deployment using Fabric item APIs
  • Fabric REST API - Content Upload: Access to upload files and manage workspace content for sample data and notebook deployment using Fabric REST API scopes
  • Power BI API - Tenant.Read.All: Delegated permission to read organization's Power BI tenant information using Power BI REST API permissions
✅ Quick Check — Verify your tools are ready
# Check Azure CLI
az --version
az account show

# Check Azure Developer CLI
azd version

# Check Python
python --version

2. Deployment Overview

This solution accelerator uses a two-phase deployment approach to provision a complete data platform. The process is fully automated, idempotent, and safe to re-run.

1️⃣ Phase 1: Infrastructure (Azure)

Powered by Bicep & Azure Resource Manager This phase creates the physical resources in your Azure subscription.

  • Resource Group: A container for your resources.
  • Fabric Capacity: The compute engine (F SKU) that powers your data workloads.

2️⃣ Phase 2: Data Platform (Fabric)

Powered by Python & Fabric REST APIs This phase configures the logical architecture inside Microsoft Fabric through 10 deployment steps:

  1. Workspace Setup: Creates or configures the workspace on your Capacity
  2. Workspace Administrators: Adds administrators to the workspace
  3. Folder Structure: Creates organized folder structure (lakehouses, notebooks, reports, environment)
  4. Lakehouses: Deploys the Medallion Architecture (BronzeSilverGold)
  5. Sample Data: Loads sample CSV datasets into the Bronze lakehouse
  6. Notebooks: Deploys 50+ notebooks for data transformation and management
  7. Data Pipelines: Executes sequential data transformation jobs (Bronze→Silver→Gold)
  8. Power BI Reports: Deploys Power BI reports and configures dataset connections
  9. Environment: Creates Fabric Environment with custom Python libraries
  10. Data Agent: Configures AI Data Agent with Lakehouse data source (preview feature)

Deployment Architecture

The Fabric deployment is orchestrated by deploy_udf_solution.py, which coordinates modular helper functions:

Helper Module Purpose Key Functions
udf_workspace.py Workspace creation and capacity assignment setup_workspace()
udf_workspace_admins.py Administrator management with Graph API integration setup_workspace_administrators()
udf_folder.py Folder structure creation setup_folder_structure()
udf_lakehouse.py Lakehouse deployment and data loading setup_lakehouses(), load_csv_data_to_lakehouse()
udf_notebook.py Notebook deployment with lakehouse references deploy_notebooks()
udf_jobs.py Sequential notebook execution schedule_notebook_jobs_sequential()
udf_powerbi.py Power BI report deployment and dataset configuration deploy_powerbi_reports()
udf_environment.py Environment creation with custom libraries setup_environment()
udf_data_agent.py Data Agent configuration setup_data_agent()
utils.py Common utilities Token replacement, file operations, logging

This modular architecture enables:

  • Independent testing of each deployment component
  • Graceful error handling with detailed progress tracking
  • Easy customization by modifying individual helper modules
  • Reusability across different deployment scenarios

🔄 Idempotency & Re-runs

The deployment is designed to be safe to re-run. If you run azd up again:

  • Infrastructure: Only updates settings if they have changed (e.g., resizing Capacity).
  • Workspace: Detects existing workspace and skips creation.
  • Content:
    • Notebooks/Reports: Updated to the latest version (overwrites changes).
    • Data: Preserved (sample data is re-uploaded if missing).
    • Admins: New admins are added; existing ones remain.

The deployment orchestration coordinates both phases, passing deployment parameters and ensuring proper sequencing. See deployment options for different ways to run this deployment based on your preferred environment.


3. Deployment Options

Choose your deployment environment based on your workflow and requirements. All options use the same Deployment commands with environment-specific setup.

Environment Best For Setup Required Notes
Local Machine Full development control Install software requirements Most flexible, requires local setup
Azure Cloud Shell Zero setup Just a web browser Pre-configured tools, session timeouts
GitHub Codespaces Team consistency GitHub account Cloud development environment
Visual Studio Code (WEB) Zero setup Just a web browser Web-based VS Code, session timeouts
Dev Container Standardized tooling Docker Desktop + VS Code Containerized consistency
GitHub Actions Automated CI/CD Service principal setup Production deployments
1. Local Machine

Deploy with full control over your development environment.

Setup requirements: Install the software requirements

Deployment: Use the standard Deployment commands

2. Azure Cloud Shell

Deploy from Azure's browser-based terminal with zero local installation.

Setup: Open Azure Cloud Shell and install Azure Developer CLI:

curl -fsSL https://aka.ms/install-azd.sh | bash && exec bash

Deployment: Run the Deployment commands (Azure CLI pre-authenticated)

3. GitHub Codespaces

Deploy from a cloud development environment with pre-configured tools.

Setup:

  1. Go to the Unified Data Foundation with Fabric solution in GitHub Codespace
  2. Follow the instructions on screen to create a new codespace with default setup.
  3. Wait for the environment to initialize (2-3 minutes)
  4. All tools are pre-installed; proceed to deployment

Deployment: Install azd and run Deployment commands with device authentication:

# Install azd if needed
curl -fsSL https://aka.ms/install-azd.sh | bash && exec bash

# Use device code authentication  
az login --use-device-code
azd auth login --use-device-code

# Continue with deployment commands
4. Visual Studio Code (WEB)

Deploy from VS Code in the browser with zero local installation.

Setup:

  1. Open the following link to launch VS Code Web:

    Open in Visual Studio Code Web

  2. When prompted, sign in using your Microsoft account linked to your Azure subscription. Select the appropriate subscription to continue.

  3. Once the solution opens, the AI Foundry terminal will automatically start running the following command to install the required dependencies:

    sh install.sh

    During this process, you’ll be prompted with the message:

    What would you like to do with these files?
    - Overwrite with versions from template
    - Keep my existing files unchanged
    

    Choose “Overwrite with versions from template” and provide a unique environment name when prompted.

Deployment: Install azd and run Deployment commands with device authentication:

# Install azd if needed
curl -fsSL https://aka.ms/install-azd.sh | bash && exec bash

# Use device code authentication  
az login --use-device-code
azd auth login --use-device-code

# Continue with deployment commands
5. VS Code Dev Container

Deploy from a containerized environment for team consistency.

Setup:

  1. Install Docker Desktop and Dev Containers extension
  2. Clone repository and open in VS Code
  3. Reopen in container when prompted

Deployment: All tools pre-installed - run Deployment commands directly

6. GitHub Actions (CI/CD)

Automated deployment using the included workflow.

Setup: Configure repository variables and set up service principal with federated credentials

Triggers: Push to main branch or manual workflow dispatch


4. Deployment Commands

One-command deployment - Deploy everything with Azure Developer CLI (prerequisites required):

# Clone and navigate to repository
git clone https://github.com/microsoft/unified-data-foundation-with-fabric-solution-accelerator.git
cd unified-data-foundation-with-fabric-solution-accelerator

# Authenticate (required)
az login
azd auth login

# Optional: Customize workspace name
azd env set FABRIC_WORKSPACE_NAME "My Analytics Platform"

# Deploy everything
azd up

During deployment, you'll specify:

  • Environment name (e.g., "udfwf-dev"). This will be used to build the name of the deployed Azure resources.
  • Azure subscription.
  • Azure resource group.

What you get: Complete medallion architecture with Fabric capacity, lakehouses (Bronze/Silver/Gold), notebooks, sample data, and Power BI reports.

💡 Alternative Deployment Option This guide uses Azure Developer CLI for automated deployment. If you prefer more granular control or have an existing Fabric capacity, see the Manual Deployment Guide.

Next Steps


5. Deployment Results

After successful deployment, you'll have a complete data platform implementing medallion architecture.

Azure Infrastructure

Resource Purpose
Fabric Capacity Dedicated compute for Fabric workloads

Screenshot of deployed Azure resources

Fabric Components

Fabric Workspace

Workspace created with the specified or default name.

Screenshot of resulting Fabric workspace

Folder Structure

your-workspace/
├── lakehouses/          # Bronze, Silver, Gold lakehouses
├── notebooks/           # Data transformation pipelines
│   ├── bronze_to_silver/
│   ├── silver_to_gold/
│   ├── data_management/
│   └── schema/
└── reports/            # Power BI dashboards

Screenshot of resulting Fabric workspace folder structure

Lakehouses

Name Purpose Schema Support
maag_bronze Raw data ingestion Enabled
maag_silver Cleaned, standardized data Enabled
maag_gold Business-ready aggregated data Enabled

Screenshot of resulting Fabric lakehouses

Sample Data

The solution includes sample data for:

  • Finance data: accounts, invoices, payments
  • Sales data: orders, order lines, payments from multiple sources
  • Shared reference data: customers, products, locations, categories

Screenshot of resulting Fabric sample data

Notebooks

Automation Components (50+ notebooks total):

  • Orchestration notebooks (2): run_bronze_to_silver, run_silver_to_gold - Main pipeline execution
  • Bronze to Silver transformations (17): Domain-specific data processing for finance, sales, and shared entities
  • Silver to Gold transformations (19): Business-ready aggregations and data modeling
  • Schema definitions (8): Data models for Silver and Gold layers across all domains
  • Data management utilities (5): Table operations, troubleshooting, and maintenance tools
  • Data Agent configuration (1): AI Data Agent setup notebook

Screenshot of resulting Fabric notebooks

Power BI Reports

Power BI reports are automatically deployed as part of Step 8 of the deployment process. Any .pbix files found in the reports/ directory will be deployed to the workspace's reports folder with automatic dataset configuration:

  • Report Deployment: Scans recursively through the reports directory and uploads each report
  • Conflict Resolution: Uses Create or Overwrite mode for idempotent deployments
  • Dataset Configuration: Automatically configures dataset parameters to connect to the Gold lakehouse SQL endpoint
    • Updates ServerName parameter with lakehouse connection string
    • Updates DatabaseName parameter with lakehouse name
  • Folder Organization: Assigns reports to the reports folder within the workspace
  • Error Handling: Continues deployment if individual reports fail, with detailed logging

Note: Dataset parameter updates require appropriate Power BI API permissions. Service Principals may experience limitations - see Known Limitations for details.

PowerBI files

Screenshot of resulting PowerBI reports

PowerBI Dashboard

Screenshot of resulting PowerBI dashboard


6. Advanced Configuration Options

The solution accelerator provides flexible configuration options to customize your deployment. Parameters can be configured through Azure Developer CLI environment variables (azd env set) for local deployments or through GitHub repository variables for CI/CD deployments.

📁 Configuration Files Reference:

🏗️ Infrastructure Configuration

Configure the Azure infrastructure components through Bicep template parameters defined in main.bicep.

Azure Resources
Parameter AZD Environment Variable GitHub Actions Variable Description Default Example
Solution Name solutionName AZURE_ENV_NAME Friendly name for the application/solution (3-20 chars) udfwfsa mycompany-fabric
Location AZURE_LOCATION AZURE_LOCATION Azure region for resource deployment Resource group location eastus, westus2, westeurope
Fabric Capacity SKU skuName Not directly supported* Fabric capacity tier and performance level F2 F4, F8, F16, F32, F64, F128, F256, F512, F1024, F2048
Enable Telemetry enableTelemetry Not directly supported* Enable/disable usage telemetry collection true false

GitHub Actions can use additional parameters through Bicep parameter files or workflow modifications.

Configuration Examples:

🖥️ Azure Developer CLI
# Set environment variables (used by main.parameters.json)
azd env set AZURE_LOCATION "westeurope"
azd env set skuName "F8"
azd env set enableTelemetry false
azd up
🚀 GitHub Actions

Modify azure-dev.yml Deploy Infrastructure step:

- name: Deploy Infrastructure
  uses: azure/bicep-deploy@v2
  with:
    parameters: |
      {
        "solutionName": "${{ env.AZURE_ENV_NAME_DEV }}",
        "skuName": "F8",
        "enableTelemetry": false
      }

Fabric Capacity SKU Selection Guide:

  • F2-F4: Development and testing environments
  • F8-F32: Small to medium production workloads
  • F64-F256: Large enterprise production workloads
  • F512-F2048: High-performance analytics and data science workloads

For detailed capacity planning, see Fabric capacity planning.

🏢 Fabric Workspace Configuration

Customize the Fabric workspace setup and naming conventions. These parameters are used by the deploy_udf_solution.py script during post-provisioning.

⚠️ Important: Variables marked as "Bicep output" (like AZURE_FABRIC_CAPACITY_NAME, SOLUTION_SUFFIX, AZURE_FABRIC_CAPACITY_ADMINISTRATORS) are automatically set by the deployment process and should NOT be manually configured. These are outputs from main.bicep and will be populated after infrastructure deployment.

Workspace Settings
Parameter AZD Environment Variable GitHub Actions Variable Description Default Example
Capacity Name AZURE_FABRIC_CAPACITY_NAME Bicep output (auto-set) Microsoft Fabric capacity name - DO NOT SET MANUALLY (automatically populated from Bicep deployment) Generated from Bicep fc-udfwfsa-abc123
Workspace Name FABRIC_WORKSPACE_NAME FABRIC_WORKSPACE_NAME Custom name for the Fabric workspace Unified_Data_Foundation_{solution_suffix} "MyCompany Data Foundation", "Analytics Platform - DEV"

Configuration Examples:

🖥️ Azure Developer CLI
azd env set FABRIC_WORKSPACE_NAME "Analytics Platform - DEV"
azd up
🚀 GitHub Actions

Modify azure-dev.yml environment variables:

env:
  FABRIC_WORKSPACE_NAME: "Analytics Platform (dev)"

Workspace Naming Best Practices:

  • Use descriptive names that indicate purpose and environment
  • Consider organizational naming conventions
  • Include environment indicators for multi-environment deployments (Dev, Test, Prod)
  • Avoid special characters that might cause conflicts with Fabric APIs

👥 Fabric Workspace Administrator Configuration

Manage workspace administrators and security permissions for the Fabric workspace. These parameters are processed by both the Bicep template (main.bicep) for capacity-level admins and the Fabric deployment script (deploy_udf_solution.py) for workspace-level admins.

Admin Assignment Options
Parameter AZD Environment Variable GitHub Actions Support Description Format Example
Capacity Administrators AZURE_FABRIC_CAPACITY_ADMINISTRATORS Bicep output (auto-set) DO NOT SET MANUALLY - Automatically populated from Bicep deployment with capacity-level administrators JSON array (read-only) ["user1@contoso.com", "12345678-1234-1234-1234-123456789012"]
Workspace Administrators FABRIC_WORKSPACE_ADMINISTRATORS Via environment variables Comma-separated list of workspace-level administrator identities. Accepts User Principal Names (UPNs) like user@domain.com OR Azure Entra ID Object IDs (GUIDs) obtained from Azure portal Comma-separated string "user@contoso.com, admin@contoso.com" OR "87654321-4321-4321-4321-210987654321, 12345678-1234-1234-1234-123456789012"

GitHub Actions workflow uses Bicep output for admin configuration. See examples below for customization.

Administrator Identity Formats:

FABRIC_WORKSPACE_ADMINISTRATORS accepts flexible identity formats:

  • User Principal Names (UPNs): user@domain.com format for individual users (requires Graph API permissions to resolve)
  • Azure Entra ID Object IDs (GUIDs): 12345678-1234-1234-1234-123456789012 format - recommended when Graph API permissions are unavailable
    • Get user object IDs: az ad user show --id user@contoso.com --query id -o tsv
    • Get service principal object IDs: az ad sp show --id <app-id> --query id -o tsv
  • Mixed Format: Combine UPNs and GUIDs in the same comma-separated list

Configuration Examples:

🖥️ Azure Developer CLI
# NOTE: Do NOT manually set AZURE_FABRIC_CAPACITY_ADMINISTRATORS - it's automatically set by Bicep deployment

# Option 1: Set workspace administrators using UPNs (requires Graph API permissions)
azd env set FABRIC_WORKSPACE_ADMINISTRATORS "user@contoso.com, admin@contoso.com"

# Option 2: Set workspace administrators using Azure Entra ID Object IDs (recommended when Graph API unavailable)
azd env set FABRIC_WORKSPACE_ADMINISTRATORS "87654321-4321-4321-4321-210987654321, 12345678-1234-1234-1234-123456789012"

# Option 3: Mix UPNs and Object IDs
azd env set FABRIC_WORKSPACE_ADMINISTRATORS "user@contoso.com, 12345678-1234-1234-1234-123456789012"

azd up
🚀 GitHub Actions

Option A: Update main.parameters.json:

{
  "parameters": {
    "solutionName": { "value": "${AZURE_ENV_NAME}" },
    "fabricAdminMembers": { "value": ["user@contoso.com"] }
  }
}

Option B: Override in workflow azure-dev.yml:

- name: Deploy Infrastructure  
  uses: azure/bicep-deploy@v2
  with:
    parameters: |
      {
        "fabricAdminMembers": ["user@contoso.com", "sp-guid"]
      }

Administrator Assignment Behavior:

  • Automatic Default Admin: The deployment identity (user or service principal) is automatically added as a Fabric capacity admin
  • Duplicate Detection: Prevents adding the same principal multiple times
  • Fallback Logic: Object ID method tries both User and ServicePrincipal types automatically
  • Graph API Resolution: UPN method uses Microsoft Graph API for identity resolution

Permission Requirements: Administrators configured through these parameters will have Admin role on the Fabric workspace, providing:

  • Full workspace management capabilities
  • Ability to manage workspace items (lakehouses, notebooks, reports)
  • User and permission management within the workspace
  • Workspace settings configuration

🐍 Python Environment Configuration Options

Configure deployment behavior and troubleshooting options. These parameters are handled by the PowerShell orchestration script (Run-PythonScript.ps1).

Deployment Customization

These options are primarily used for configuring the appropriate environment for each deployment process based on elements such as underlying operating system or specialized environments such as containerized deployments or GitHub-hosted runners.

Parameter PowerShell Switch AZD Support GitHub Actions Support Description Use Case
Skip Virtual Environment -SkipPythonVirtualEnvironment Manual override ✅ Used in workflow Use system Python instead of virtual environment System-wide Python management, containerized environments
Skip Dependencies -SkipPythonDependencies Manual override ✅ Used in workflow Skip installing Python packages (assume pre-installed) Pre-configured environments, repeated deployments
Skip Pip Upgrade -SkipPipUpgrade Manual override ✅ Used in workflow Skip upgrading pip to latest version Environments with controlled pip versions

Configuration Examples:

🖥️ Azure Developer CLI

These parameters are automatically optimized in azure.yaml:

hooks:
  postprovision:
    windows:
      shell: pwsh
      run: ./infra/scripts/utils/Run-PythonScript.ps1 -ScriptPath "infra/scripts/fabric/deploy_udf_solution.py"
      interactive: true
      continueOnError: false
    posix:
      shell: pwsh
      run: ./infra/scripts/utils/Run-PythonScript.ps1 -ScriptPath "infra/scripts/fabric/deploy_udf_solution.py" -SkipPythonVirtualEnvironment
      interactive: true
      continueOnError: false
  predown:
    windows:
      shell: pwsh
      run: ./infra/scripts/utils/Run-PythonScript.ps1 -ScriptPath "infra/scripts/fabric/remove_udf_solution.py"
      interactive: true
      continueOnError: false
    posix:
      shell: pwsh
      run: ./infra/scripts/utils/Run-PythonScript.ps1 -ScriptPath "infra/scripts/fabric/remove_udf_solution.py" -SkipPythonVirtualEnvironment
      interactive: true
      continueOnError: false
🚀 GitHub Actions

These parameters are automatically optimized in azure-dev.yml:

- name: Run Fabric Provisioning Script
  run: |
    pwsh ./infra/scripts/utils/Run-PythonScript.ps1 `
      -ScriptPath "infra/scripts/fabric/deploy_udf_solution.py" `
      -SkipPythonVirtualEnvironment `
      -SkipPythonDependencies `
      -SkipPipUpgrade

7. Known Limitations

This section documents known limitations in the deployment process and their workarounds.

🔒 Power BI API Parameter Updates

Issue: Service Principals cannot update Power BI dataset parameters via API, resulting in HTTP 403 errors.

Impact:

  • During automated deployment, if deployment identity is a Service Principal or a Managed Identity, Power BI reports are deployed but dataset parameters (SQL endpoint connection strings) may not be automatically configured
  • Reports may show connection errors until manually configured

Technical Details: Power BI dataset parameter updates are handled by the deployment process. Note that Service Principals may have restricted API access for this operation.

# From the deployment helper modules
try:
    powerbi_client.update_powerbi_dataset_parameters(
        dataset_id=dataset['id'], 
        parameters=[
            {"name": "sqlEndpoint", "newValue": sql_endpoint},
            {"name": "database", "newValue": lakehouse_name}
        ]
    )
    print(f"✅ Dataset parameters updated successfully for '{report_name}'")
except Exception as param_error:
    if "HTTP 403" in str(param_error):
        print(f"⚠️ WARNING: Cannot update dataset parameters automatically")
        print(f"    Manual configuration required in Power BI service")

Workaround:

  • The deployment continues successfully despite this limitation
  • Follow the manual configuration steps in the Power BI Deployment Guide to complete the report setup
  • This typically involves updating the sqlEndpoint and database parameters in the Power BI service

👤 Graph API Principal (user or service principal) Lookup Limitations

Issue: The deployment identity may lack permissions to query user object IDs from Azure Active Directory via Microsoft Graph API.

Impact:

  • When using --fabricAdmins with user principal names (UPNs), the script may fail to resolve user identities
  • Service Principals may successfully create workspaces but fail to add human users as administrators
  • This can result in workspaces that are only accessible to the deployment service principal

Technical Details: The udf_workspace_admins.py helper module implements fallback logic:

def detect_principal_type(admin_identifier, graph_client=None):
    try:
        # Use Graph API to resolve the principal
        principal_type, object_id, principal_data = graph_client.resolve_principal(admin_identifier)
        return principal_type, object_id, principal_data
    except GraphApiError as e:
        # Convert Graph API errors to ValueError for backward compatibility
        print(f"⚠️ WARNING: Graph API lookup failed for '{admin_identifier}': {str(e)}")
        # Fallback to original logic if Graph API is not available
        if is_valid_guid(admin_identifier):
            return "ServicePrincipal", admin_identifier, {"id": admin_identifier, "displayName": "Unknown"}

Workarounds:

  1. Use Object IDs Directly: Configure administrators using their Azure AD object IDs, which don't require Graph API resolution. Get object IDs using Azure CLI:

    # Get user object ID
    az ad user show --id user@contoso.com --query id -o tsv
    
    # Set administrators using object IDs (comma-separated)
    azd env set FABRIC_WORKSPACE_ADMINISTRATORS "87654321-4321-4321-4321-210987654321, 12345678-1234-1234-1234-123456789012"
    azd up

    The helper module automatically detects whether an identifier is a UPN or object ID and handles accordingly.

  2. Post-Deployment Admin Assignment: If Graph API permissions cannot be granted, add administrators manually after deployment through the Fabric portal workspace settings, or use the dedicated helper module udf_workspace_admins.py with appropriate credentials that have Graph API access.


🔐 Fabric REST API Permission Issues

Issue: Service Principals may lack sufficient permissions to access Microsoft Fabric REST APIs.

Impact:

  • Deployment fails during workspace creation or management operations
  • Graceful exit with clear guidance on permission requirements

Technical Details: The deploy_udf_solution.py script provides specific error handling for authorization failures:

except FabricApiError as e:
    if e.status_code == 401:
        print(f"⚠️ WARNING: Unauthorized access to Fabric APIs. Please review your Fabric permissions and ensure you have proper Fabric licensing and permissions.")
        print("   📋 Check the following resources:")
        print("   • Fabric licenses: https://learn.microsoft.com/fabric/enterprise/licenses")
        print("   • Identity support: https://learn.microsoft.com/rest/api/fabric/articles/identity-support")
        print("   • Create Entra app with appropriate Fabric permissions: https://learn.microsoft.com/rest/api/fabric/articles/get-started/create-entra-app")
        sys.exit(0)  # Graceful exit with guidance

Resolution:

  1. Verify Fabric Licensing: Ensure your organization has appropriate Microsoft Fabric licenses
  2. Review Identity Configuration: Follow the Fabric Identity Support documentation
  3. Configure Service Principal: If using a service principal, ensure it's properly configured following Create Entra App guidance
  4. Check API Permissions: Verify the deployment identity has the required Fabric REST API permissions as listed in the prerequisites

The script performs a graceful exit (sys.exit(0)) rather than failing abruptly, allowing you to resolve permissions and retry the deployment.


8. Environment Cleanup

When you no longer need your deployed environment, Azure Developer CLI provides a streamlined approach to completely remove all resources and clean up your Microsoft Fabric workspace.

Complete Environment Removal

The azd down command orchestrates a complete environment cleanup process that:

  1. Removes Fabric Workspace: Safely deletes the Microsoft Fabric workspace and all associated items
  2. Deprovisions Azure Resources: Removes all Azure infrastructure components deployed via Bicep templates
  3. Preserves Local Environment: Keeps your local development environment and configurations intact

Quick cleanup command:

# Navigate to your solution directory
cd unified-data-foundation-with-fabric-solution-accelerator

# Remove everything deployed by azd up
azd down

Cleanup Process Details

Based on the azure.yaml configuration, the cleanup process follows these orchestrated steps:

Phase 1: Fabric Workspace Cleanup (predown hook)

Before removing Azure infrastructure, the cleanup process first handles the Microsoft Fabric workspace:

Windows (PowerShell):

./infra/scripts/utils/Run-PythonScript.ps1 -ScriptPath "infra/scripts/fabric/remove_udf_solution.py"

Unix/Linux (PowerShell Core):

./infra/scripts/utils/Run-PythonScript.ps1 -ScriptPath "infra/scripts/fabric/remove_udf_solution.py" -SkipPythonVirtualEnvironment

This orchestration script (Run-PythonScript.ps1) manages:

  • Python Environment Setup: Creates or reuses Python virtual environment with required dependencies
  • Script Execution: Runs the specified Python script with proper error handling
  • Cross-Platform Support: Handles differences between Windows and Unix-based systems

The core removal logic is handled by remove_udf_solution.py, which:

  • Workspace Lookup: Finds the workspace by name or ID (defaults to "Unified Data Foundation with Fabric workspace")
  • Comprehensive Removal: Deletes all workspace items including notebooks, lakehouses, and datasets
  • Confirmation Prompts: Provides interactive confirmation to prevent accidental deletions
  • Error Handling: Gracefully handles missing workspaces or permission issues

Phase 2: Azure Infrastructure Cleanup

After successful Fabric workspace removal, azd down proceeds to deprovision all Azure resources that were created through the main.bicep template, including:

  • Microsoft Fabric Capacity: Dedicated compute resources
  • Resource Group: Complete resource group removal (if specified)

Safety Features

The cleanup process includes several safety mechanisms:

  • Interactive Confirmation: Prompts before deleting workspaces to prevent accidental removal
  • Graceful Error Handling: Continues with infrastructure cleanup even if Fabric workspace removal fails
  • Detailed Logging: Provides comprehensive output for troubleshooting and audit purposes
  • Non-Destructive Failures: Missing workspaces or permission issues don't prevent infrastructure cleanup

9. Additional Resources

For support, visit the project repository or engage with the Microsoft Fabric community.