metapod-probes

Extract metadata from 49 data sources with a single pip install. No raw data. No inbound access. Just metadata, in your network.

pip install metapod-probes
metapod init
metapod run "oracle.metadata.*" --config config.yaml

Output: JSON files in ./output/, ready for any consumer.

197 probes across databases (Oracle, PostgreSQL, Snowflake, Azure SQL, BigQuery, MySQL, DuckDB), cloud platforms (ADLS, Blob Storage, S3, ADF, Unity Catalog), BI tools (Power BI, Tableau, Excel), ETL pipelines (Informatica, Airflow, dbt), governance platforms (Purview, Collibra), data quality (Soda Core, YData Profiling, Schema Contract), compliance (Presidio PII, Access Audit, Data Retention), cost monitoring, and local files (CSV, JSON, Parquet, Avro).

Nine probe categories: metadata, governance, quality, profiling, operations, compliance, cost, schema history, and lineage.

Getting Started

• Installation
  • From PyPI (released version)
  • From source (local development)
  • Air-gapped install (no internet)
  • Database extras
  • Cloud extras
  • Quality, profiling, and compliance extras
  • Everything
  • Verify
• Quickstart
  • Step 1: Install
  • Step 2: Create a test database
  • Step 3: Create config
  • Step 4: Extract
  • Step 5: Inspect
  • Next steps
• Concepts
  • Probe
  • Source
  • Category
  • Envelope
  • Delta Engine
  • Connection Registry

User Guide

• Usage
  • Commands
  • Basic workflow
  • Glob patterns
  • Environment variables
  • Common config examples
• Delta Engine
  • Usage
  • How it works
  • Change detection strategies
  • State persistence
  • API
• Standalone Usage
  • Local file mode (default)
  • Output format
  • Comparing extractions (diff)
  • Incremental extraction (delta mode)
  • Daemon mode (pull)
  • Optional: push to a remote platform
  • Scheduling extractions
• Docker & CI/CD
  • Container deployment
  • Docker Compose
  • GitHub Actions
  • GitLab CI
  • Kubernetes CronJob
  • Credential management

Data Governance

• Data Governance Guide
  • The governance pipeline
  • Prerequisites
  • Step 1: Profile your data
  • Step 2: Detect anomalies
  • Step 3: Run quality checks
  • Step 4: Detect PII
  • Step 5: Classify sensitivity
  • Step 6: Validate schema contract
  • Step 7: Audit access control
  • Step 8: Check retention compliance
  • Step 9: Estimate storage costs
  • Running all steps together
  • Integration with azure-monitoring

Integrations

• Platform Integrations
  • Dataflix (built-in)
  • DataHub
  • OpenMetadata
  • Apache Atlas / Purview
  • Custom REST API
  • Webhook / Event Streaming

Developer Guide

• Creating Custom Probes
  • The BaseProbe interface
  • Connection module
  • File structure
  • ProbeResult envelope
  • Plugin system (entry points)
  • Testing your probe
• Architecture
  • Data flow
  • Directory structure
  • Key design decisions
  • Security model

Probe Reference

• Probe Reference