Architecture
============

Data flow
---------

.. code-block:: text

   config.yaml
       │
       ▼
   Connection Registry ──── probes/{source}/connection.py
       │
       ▼
   Probe Execution ──────── probes/{source}/{category}/{name}.py
       │                         │
       │                    DeltaEngine (optional)
       │                         │
       ▼                         ▼
   ProbeResult (JSON envelope)
       │
       ├── File output (./output/{source}/{category}/{name}.json)
       ├── Push (metapod push --target URL)
       └── Serve (metapod serve --port 9090)

Directory structure
-------------------

.. code-block:: text

   metapod/
   ├── core/
   │   ├── base_probe.py      # BaseProbe ABC + ProbeResult envelope
   │   ├── registry.py        # @register decorator + discovery
   │   ├── connections.py     # CONNECTION_REGISTRY (42 source types)
   │   ├── config.py          # SourceConfig (SecretStr), ProbeConfig
   │   ├── delta.py           # DeltaEngine (incremental extraction)
   │   ├── diff.py            # Diff between two extraction runs
   │   ├── transport.py       # HTTP push with retry
   │   ├── server.py          # Pull-mode daemon
   │   ├── scheduler.py       # Cron-based scheduling
   │   ├── audit.py           # Extraction audit log
   │   ├── openlineage.py     # OpenLineage event emitter
   │   └── models.py          # Standard record models
   ├── probes/
   │   ├── oracle/
   │   │   ├── connection.py
   │   │   ├── metadata/
   │   │   │   ├── tables.py
   │   │   │   ├── columns.py
   │   │   │   └── ...
   │   │   ├── quality/
   │   │   └── governance/
   │   ├── postgres/
   │   ├── azure_sql/
   │   ├── unity_catalog/
   │   └── ... (42 sources)
   ├── cli.py                 # Typer CLI (run, push, serve, ...)
   └── metapod_cli/           # Entry point package

Key design decisions
--------------------

**Stateless probes**: Every probe is a pure function — same input, same
output. No side effects. The Delta Engine manages state separately.

**Zero raw data**: Probes query only system catalogs and metadata views.
No ``SELECT *`` on user tables, ever.

**Zero inbound access**: In push mode, only HTTPS outbound. The pull-mode
daemon is opt-in.

**Flat records**: Probes return ``list[dict]``, not nested objects. This
makes JSON output directly consumable by pandas, DuckDB, or any tool.

**Auto-discovery**: Place a file in ``probes/{source}/{category}/``,
decorate with ``@register``, done. No configuration file to edit.

**SecretStr credentials**: Passwords and tokens use Pydantic ``SecretStr``.
They never appear in logs, repr, or crash dumps.

Security model
--------------

- Credentials are ``SecretStr`` (masked in repr, logs, serialization)
- SQL identifiers are validated with regex before interpolation
- Config supports ``${ENV_VAR}`` substitution (never hardcode secrets)
- Audit log records every extraction (timestamp, probe, source, user)
- No data leaves the network unless you explicitly ``metapod push``