metapod — Lightweight Metadata Probes

155 probes across 40 data sources. Extract metadata from databases, cloud platforms, BI tools, ETL pipelines, and file systems.

Runs inside your network. Extracts only metadata — no raw data leaves the perimeter. Output is local JSON files that you control.

pip install metapod-probes
metapod run "oracle.metadata.*" --config config.yaml
# Output in ./output/oracle/metadata/*.json — done.

# Optional: push to a catalog platform
metapod push --target https://dataflix.example.com --project my-project

Use cases

• Data catalog population — feed DataHub, OpenMetadata, Atlan, Collibra, or Dataflix

• Schema documentation — extract and version-control your database schema as JSON

• Migration assessment — inventory all objects before a cloud migration

• CI/CD metadata — validate schema changes in pull requests

• Compliance audit — inventory all tables/columns for GDPR/BCBS data mapping

• Incremental monitoring — detect schema drift with delta mode

Getting Started

• Installation
• Standalone Usage
  • Local file mode (default)
  • Output format
  • Comparing extractions (diff)
  • Incremental extraction (delta mode)
  • Daemon mode (pull)
  • Optional: push to a remote platform
  • Scheduling extractions
• Usage
  • Commands
  • Basic workflow
  • Glob patterns
  • Environment variables
  • Common config examples

Guides

• Azure Walkthrough
  • Prerequisites
  • Step 1: Service Principal
  • Step 2: Automated Provisioning
  • Step 3: Run Probes
  • Step 4: Push to Dataflix
  • Step 5: Delta Mode
  • Step 6: Teardown
  • Configuration Reference
  • Troubleshooting
• Docker & CI/CD
  • Container deployment
  • Docker Compose
  • GitHub Actions
  • GitLab CI
  • Kubernetes CronJob
  • Credential management
• Platform Integrations
  • Dataflix (built-in)
  • DataHub
  • OpenMetadata
  • Apache Atlas / Purview
  • Custom REST API
  • Webhook / Event Streaming
• Delta Engine
  • Usage
  • How it works
  • Change detection strategies
  • State persistence
  • API
• Creating Custom Probes
  • The BaseProbe interface
  • Connection module
  • File structure
  • ProbeResult envelope
  • Plugin system (entry points)
  • Testing your probe

Reference

• Available Probes
  • Databases
  • Cloud Platforms
  • BI Tools
  • ETL / Orchestration
  • Files / Other
• Core API
  • ProbeResult
  • BaseProbe
  • register()
  • get_probe()
  • list_probes()
  • get_probes()
  • discover_probes()
  • SourceConfig
  • TargetConfig
  • ProbeConfig
  • load_config()
  • push_probes()
  • push_directory()
  • diff_probes()
  • matches_cron()
  • run_schedule()
• Probe API
  • Oracle
  • PostgreSQL
  • File Scanner

Indices and tables

• Index

• Module Index

• Search Page