Concepts

Probe

A probe is a single metadata extraction unit. It connects to one data source, runs one query (or API call), and returns a list of records.

Each probe is a Python class of ~40 lines with three methods:

  • probe_name() — unique identifier (e.g., oracle.metadata.tables)

  • description() — human-readable description

  • execute() — the extraction logic, returns list[dict]

Naming convention: source.category.name

Source

A data source: a database, cloud platform, BI tool, or file system. Examples: oracle, azure_sql, unity_catalog, excel.

Each source has a connection module (probes/{source}/connection.py) that handles authentication and connection setup.

Category

The type of metadata a probe extracts:

Category

What it extracts

Examples

metadata

Schema, structure, definitions

tables, columns, views, procedures

governance

Access control, permissions

grants, roles, group memberships

quality

Data health signals

null ratios, row counts, freshness

profiling

Statistical summaries

distinct values, correlations

operations

Runtime behavior

query history, access patterns

compliance

Regulatory flags

PII detection, sensitivity labels

cost

Storage and compute

table sizes, blob metrics

schema_history

Change tracking

DDL modification dates

lineage

Data flow

table-to-table, column-to-column

Envelope

Every probe produces a ProbeResult — a standard JSON envelope:

{
  "envelope_version": "1.0",
  "probe": "oracle.metadata.tables",
  "source": {"type": "oracle", "id": "prod-db:1521/ORCLPDB1"},
  "timestamp": "2026-03-29T12:00:00+00:00",
  "records": [...],
  "record_count": 42,
  "duration_ms": 234,
  "error": null
}

The envelope is the same for every probe. The records array shape varies by probe (documented in the probe reference).

Delta Engine

The Delta Engine enables incremental extraction. Instead of extracting all records every time, it compares with the previous run and emits only new, modified, or deleted records.

metapod run "oracle.*" --delta

State is stored in output/.delta/. See Delta Engine.

Connection Registry

All 42 source types are mapped in core/connections.py. Adding a new source means adding one entry to the registry dict:

"my_source": ("probes.my_source.connection", "connect", "source_id")

See Creating Custom Probes for the full guide.