Azure Walkthrough

This guide walks you through configuring metapod to extract metadata from Azure services and push results to Dataflix. By the end you will have tested 18 Azure probes across 5 services.

Prerequisites

  • An Azure subscription (free tier with $200 credit works)

  • Azure CLI installed and logged in (az login)

  • Python 3.10+

pip install metapod-probes[azure] pyodbc

Step 1: Service Principal

Create a service principal for non-interactive access:

az ad sp create-for-rbac --name metapod-automation --role Contributor \
  --scopes /subscriptions/<SUBSCRIPTION_ID>

Save the output credentials in a .env file at the project root:

# .env
AZURE_TENANT_ID=<tenant-id>
AZURE_CLIENT_ID=<client-id>
AZURE_CLIENT_SECRET=<client-secret>

metapod loads .env automatically.

Step 2: Automated Provisioning

metapod can create all Azure resources for you:

metapod provision azure --region westeurope

This creates:

  • Resource Group (metapod-test-<suffix>)

  • Storage Account with ADLS Gen2 hierarchical namespace - Blob container with sample CSV/JSON files - ADLS filesystem with sample data - File share with sample documents

  • Azure SQL Database (Basic tier, ~$5/month) - 3 tables, 1 view, 1 stored procedure, sample data

  • Azure Data Factory - Linked service to storage, sample pipeline

Config files are auto-generated in ./configs/.

Step 3: Run Probes

Run each probe group against the provisioned resources:

Azure Blob Storage (3 probes):

metapod run "blob_storage.*" -c configs/config-blob.yaml --delta

Output: containers, blobs with size/type, auto-scan with schema sampling.

ADLS Gen2 (4 probes):

metapod run "adls_gen2.*" -c configs/config-adls.yaml --delta

Output: filesystems, paths with format detection, ACL permissions, recursive scan.

Azure File Share (3 probes):

metapod run "azure_file_share.*" -c configs/config-fileshare.yaml --delta

Output: shares, file listings, auto-scan.

Azure SQL (5 probes):

metapod run "azure_sql.metadata.*" -c configs/config-azuresql.yaml --schema dbo --delta

Output: tables, columns, constraints, views with definitions, stored procedures.

Azure Data Factory (5 probes):

metapod run "azure_data_factory.*" -c configs/config-adf.yaml --delta

Output: pipelines, linked services, datasets, activities, triggers.

Microsoft Purview (3 probes):

If you have a Purview account:

metapod run "purview.metadata.*" -c configs/config-purview.yaml

Output: collections, catalog entities, lineage relationships.

Step 4: Push to Dataflix

Send all results to Dataflix for lineage analysis:

metapod push --target https://dataflix.macalab.org \
             --project azure-project \
             --api-key $DATAFLIX_API_KEY \
             --output ./output

Dataflix will:

  1. Convert tables + columns into CREATE TABLE DDL

  2. Store views and procedures as SQL files

  3. Archive non-SQL probes (ADF, ADLS, Purview) as reference JSON

  4. Run a lineage scan if --scan is enabled

  5. Enrich the graph with probe origin (source_type, cloud_provider, region, resource_group)

Step 5: Delta Mode

On subsequent runs, use --delta to extract only changes:

metapod run "azure_sql.metadata.*" -c configs/config-azuresql.yaml --delta

The DeltaEngine compares each record against the previous scan state:

  • mtime – modification timestamp (DDL changes, file updates)

  • hash – content hash (procedure source code, view definitions)

  • size – row count or file size (drift detection)

Output shows a delta summary:

                           Probe Results
Probe                 Records  Duration  Status  New  Mod  Del  Skip
azure_sql.metadata.t.   3       56ms     OK       0    0    0    3
azure_sql.metadata.c.  22       85ms     OK       0    0    0   22

Delta summary: +0 new, ~0 modified, -0 deleted

State is persisted in output/.delta/<probe_name>.json.

Step 6: Teardown

Delete all Azure resources when done:

metapod provision teardown

This deletes the resource group and all resources within it. Azure handles cleanup asynchronously (takes a few minutes).

Configuration Reference

Blob Storage:

source:
  type: blob_storage
  host: <storage-account-name>
  token: <account-key-or-sas>
  schema_name: <container-name>  # optional, filters to one container

ADLS Gen2:

source:
  type: adls_gen2
  host: <storage-account-name>
  token: <account-key>
  schema_name: <filesystem-name>

Authentication alternatives (instead of token):

source:
  type: adls_gen2
  host: <storage-account-name>
  username: <client-id>        # Service Principal
  password: <client-secret>
  service_name: <tenant-id>

Azure SQL:

source:
  type: azure_sql
  host: <server>.database.windows.net
  service_name: <database>
  username: <sql-admin>
  password: <password>

Or with Azure AD token:

source:
  type: azure_sql
  host: <server>.database.windows.net
  service_name: <database>
  token: <bearer-token>

Azure Data Factory:

source:
  type: azure_data_factory
  sid: <subscription-id>
  service_name: <resource-group>
  schema_name: <factory-name>
  host: <tenant-id>
  username: <client-id>
  password: <client-secret>

Microsoft Purview:

source:
  type: purview
  host: <purview-account-name>
  service_name: <tenant-id>
  username: <client-id>
  password: <client-secret>

Azure File Share:

source:
  type: azure_file_share
  host: <storage-account-name>
  token: <account-key>
  schema_name: <share-name>

Troubleshooting

SubscriptionNotFound: Register the required resource providers:

az provider register --namespace Microsoft.Storage --wait
az provider register --namespace Microsoft.Sql --wait
az provider register --namespace Microsoft.DataFactory --wait
az provider register --namespace Microsoft.Purview --wait

IP not allowed (Azure SQL): Add your IP to the server firewall:

az sql server firewall-rule create \
  --resource-group <rg> --server <server> \
  --name AllowMyIP \
  --start-ip-address <your-ip> --end-ip-address <your-ip>

ODBC driver missing: Install the Microsoft ODBC driver:

# Ubuntu/Debian
curl https://packages.microsoft.com/keys/microsoft.asc | sudo gpg --dearmor \
  -o /usr/share/keyrings/microsoft-prod.gpg
echo "deb [signed-by=/usr/share/keyrings/microsoft-prod.gpg] \
  https://packages.microsoft.com/ubuntu/22.04/prod jammy main" | \
  sudo tee /etc/apt/sources.list.d/mssql-release.list
sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18