Azure Walkthrough ================= This guide walks you through configuring metapod to extract metadata from Azure services and push results to Dataflix. By the end you will have tested 18 Azure probes across 5 services. Prerequisites ------------- - An Azure subscription (free tier with $200 credit works) - Azure CLI installed and logged in (``az login``) - Python 3.10+ .. code-block:: bash pip install metapod-probes[azure] pyodbc Step 1: Service Principal ------------------------- Create a service principal for non-interactive access: .. code-block:: bash az ad sp create-for-rbac --name metapod-automation --role Contributor \ --scopes /subscriptions/ Save the output credentials in a ``.env`` file at the project root: .. code-block:: bash # .env AZURE_TENANT_ID= AZURE_CLIENT_ID= AZURE_CLIENT_SECRET= metapod loads ``.env`` automatically. Step 2: Automated Provisioning ------------------------------ metapod can create all Azure resources for you: .. code-block:: bash metapod provision azure --region westeurope This creates: - **Resource Group** (``metapod-test-``) - **Storage Account** with ADLS Gen2 hierarchical namespace - Blob container with sample CSV/JSON files - ADLS filesystem with sample data - File share with sample documents - **Azure SQL Database** (Basic tier, ~$5/month) - 3 tables, 1 view, 1 stored procedure, sample data - **Azure Data Factory** - Linked service to storage, sample pipeline Config files are auto-generated in ``./configs/``. Step 3: Run Probes ------------------ Run each probe group against the provisioned resources: **Azure Blob Storage** (3 probes): .. code-block:: bash metapod run "blob_storage.*" -c configs/config-blob.yaml --delta Output: containers, blobs with size/type, auto-scan with schema sampling. **ADLS Gen2** (4 probes): .. code-block:: bash metapod run "adls_gen2.*" -c configs/config-adls.yaml --delta Output: filesystems, paths with format detection, ACL permissions, recursive scan. **Azure File Share** (3 probes): .. code-block:: bash metapod run "azure_file_share.*" -c configs/config-fileshare.yaml --delta Output: shares, file listings, auto-scan. **Azure SQL** (5 probes): .. code-block:: bash metapod run "azure_sql.metadata.*" -c configs/config-azuresql.yaml --schema dbo --delta Output: tables, columns, constraints, views with definitions, stored procedures. **Azure Data Factory** (5 probes): .. code-block:: bash metapod run "azure_data_factory.*" -c configs/config-adf.yaml --delta Output: pipelines, linked services, datasets, activities, triggers. **Microsoft Purview** (3 probes): If you have a Purview account: .. code-block:: bash metapod run "purview.metadata.*" -c configs/config-purview.yaml Output: collections, catalog entities, lineage relationships. Step 4: Push to Dataflix ------------------------ Send all results to Dataflix for lineage analysis: .. code-block:: bash metapod push --target https://dataflix.macalab.org \ --project azure-project \ --api-key $DATAFLIX_API_KEY \ --output ./output Dataflix will: 1. Convert tables + columns into CREATE TABLE DDL 2. Store views and procedures as SQL files 3. Archive non-SQL probes (ADF, ADLS, Purview) as reference JSON 4. Run a lineage scan if ``--scan`` is enabled 5. Enrich the graph with probe origin (source_type, cloud_provider, region, resource_group) Step 5: Delta Mode ------------------ On subsequent runs, use ``--delta`` to extract only changes: .. code-block:: bash metapod run "azure_sql.metadata.*" -c configs/config-azuresql.yaml --delta The DeltaEngine compares each record against the previous scan state: - **mtime** -- modification timestamp (DDL changes, file updates) - **hash** -- content hash (procedure source code, view definitions) - **size** -- row count or file size (drift detection) Output shows a delta summary: .. code-block:: text Probe Results Probe Records Duration Status New Mod Del Skip azure_sql.metadata.t. 3 56ms OK 0 0 0 3 azure_sql.metadata.c. 22 85ms OK 0 0 0 22 Delta summary: +0 new, ~0 modified, -0 deleted State is persisted in ``output/.delta/.json``. Step 6: Teardown ---------------- Delete all Azure resources when done: .. code-block:: bash metapod provision teardown This deletes the resource group and all resources within it. Azure handles cleanup asynchronously (takes a few minutes). Configuration Reference ----------------------- **Blob Storage:** .. code-block:: yaml source: type: blob_storage host: token: schema_name: # optional, filters to one container **ADLS Gen2:** .. code-block:: yaml source: type: adls_gen2 host: token: schema_name: Authentication alternatives (instead of ``token``): .. code-block:: yaml source: type: adls_gen2 host: username: # Service Principal password: service_name: **Azure SQL:** .. code-block:: yaml source: type: azure_sql host: .database.windows.net service_name: username: password: Or with Azure AD token: .. code-block:: yaml source: type: azure_sql host: .database.windows.net service_name: token: **Azure Data Factory:** .. code-block:: yaml source: type: azure_data_factory sid: service_name: schema_name: host: username: password: **Microsoft Purview:** .. code-block:: yaml source: type: purview host: service_name: username: password: **Azure File Share:** .. code-block:: yaml source: type: azure_file_share host: token: schema_name: Troubleshooting --------------- **SubscriptionNotFound**: Register the required resource providers: .. code-block:: bash az provider register --namespace Microsoft.Storage --wait az provider register --namespace Microsoft.Sql --wait az provider register --namespace Microsoft.DataFactory --wait az provider register --namespace Microsoft.Purview --wait **IP not allowed (Azure SQL)**: Add your IP to the server firewall: .. code-block:: bash az sql server firewall-rule create \ --resource-group --server \ --name AllowMyIP \ --start-ip-address --end-ip-address **ODBC driver missing**: Install the Microsoft ODBC driver: .. code-block:: bash # Ubuntu/Debian curl https://packages.microsoft.com/keys/microsoft.asc | sudo gpg --dearmor \ -o /usr/share/keyrings/microsoft-prod.gpg echo "deb [signed-by=/usr/share/keyrings/microsoft-prod.gpg] \ https://packages.microsoft.com/ubuntu/22.04/prod jammy main" | \ sudo tee /etc/apt/sources.list.d/mssql-release.list sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18