Architecture

Data flow

config.yaml
    │
    ▼
Connection Registry ──── probes/{source}/connection.py
    │
    ▼
Probe Execution ──────── probes/{source}/{category}/{name}.py
    │                         │
    │                    DeltaEngine (optional)
    │                         │
    ▼                         ▼
ProbeResult (JSON envelope)
    │
    ├── File output (./output/{source}/{category}/{name}.json)
    ├── Push (metapod push --target URL)
    └── Serve (metapod serve --port 9090)

Directory structure

metapod/
├── core/
│   ├── base_probe.py      # BaseProbe ABC + ProbeResult envelope
│   ├── registry.py        # @register decorator + discovery
│   ├── connections.py     # CONNECTION_REGISTRY (42 source types)
│   ├── config.py          # SourceConfig (SecretStr), ProbeConfig
│   ├── delta.py           # DeltaEngine (incremental extraction)
│   ├── diff.py            # Diff between two extraction runs
│   ├── transport.py       # HTTP push with retry
│   ├── server.py          # Pull-mode daemon
│   ├── scheduler.py       # Cron-based scheduling
│   ├── audit.py           # Extraction audit log
│   ├── openlineage.py     # OpenLineage event emitter
│   └── models.py          # Standard record models
├── probes/
│   ├── oracle/
│   │   ├── connection.py
│   │   ├── metadata/
│   │   │   ├── tables.py
│   │   │   ├── columns.py
│   │   │   └── ...
│   │   ├── quality/
│   │   └── governance/
│   ├── postgres/
│   ├── azure_sql/
│   ├── unity_catalog/
│   └── ... (42 sources)
├── cli.py                 # Typer CLI (run, push, serve, ...)
└── metapod_cli/           # Entry point package

Key design decisions

Stateless probes: Every probe is a pure function — same input, same output. No side effects. The Delta Engine manages state separately.

Zero raw data: Probes query only system catalogs and metadata views. No SELECT * on user tables, ever.

Zero inbound access: In push mode, only HTTPS outbound. The pull-mode daemon is opt-in.

Flat records: Probes return list[dict], not nested objects. This makes JSON output directly consumable by pandas, DuckDB, or any tool.

Auto-discovery: Place a file in probes/{source}/{category}/, decorate with @register, done. No configuration file to edit.

SecretStr credentials: Passwords and tokens use Pydantic SecretStr. They never appear in logs, repr, or crash dumps.

Security model

• Credentials are SecretStr (masked in repr, logs, serialization)

• SQL identifiers are validated with regex before interpolation

• Config supports ${ENV_VAR} substitution (never hardcode secrets)

• Audit log records every extraction (timestamp, probe, source, user)

• No data leaves the network unless you explicitly metapod push