Concepts
========

Probe
-----

A probe is a single metadata extraction unit. It connects to one data source,
runs one query (or API call), and returns a list of records.

Each probe is a Python class of ~40 lines with three methods:

- ``probe_name()`` — unique identifier (e.g., ``oracle.metadata.tables``)
- ``description()`` — human-readable description
- ``execute()`` — the extraction logic, returns ``list[dict]``

Naming convention: ``source.category.name``

Source
------

A data source: a database, cloud platform, BI tool, or file system.
Examples: ``oracle``, ``azure_sql``, ``unity_catalog``, ``excel``.

Each source has a **connection module** (``probes/{source}/connection.py``)
that handles authentication and connection setup.

Category
--------

The type of metadata a probe extracts:

.. list-table::
   :header-rows: 1

   * - Category
     - What it extracts
     - Examples
   * - **metadata**
     - Schema, structure, definitions
     - tables, columns, views, procedures
   * - **governance**
     - Access control, permissions
     - grants, roles, group memberships
   * - **quality**
     - Data health signals
     - null ratios, row counts, freshness
   * - **profiling**
     - Statistical summaries
     - distinct values, correlations
   * - **operations**
     - Runtime behavior
     - query history, access patterns
   * - **compliance**
     - Regulatory flags
     - PII detection, sensitivity labels
   * - **cost**
     - Storage and compute
     - table sizes, blob metrics
   * - **schema_history**
     - Change tracking
     - DDL modification dates
   * - **lineage**
     - Data flow
     - table-to-table, column-to-column

Envelope
--------

Every probe produces a **ProbeResult** — a standard JSON envelope:

.. code-block:: json

   {
     "envelope_version": "1.0",
     "probe": "oracle.metadata.tables",
     "source": {"type": "oracle", "id": "prod-db:1521/ORCLPDB1"},
     "timestamp": "2026-03-29T12:00:00+00:00",
     "records": [...],
     "record_count": 42,
     "duration_ms": 234,
     "error": null
   }

The envelope is the same for every probe. The ``records`` array shape
varies by probe (documented in the probe reference).

Delta Engine
------------

The Delta Engine enables incremental extraction. Instead of extracting
all records every time, it compares with the previous run and emits
only new, modified, or deleted records.

.. code-block:: bash

   metapod run "oracle.*" --delta

State is stored in ``output/.delta/``. See :doc:`delta-engine`.

Connection Registry
-------------------

All 42 source types are mapped in ``core/connections.py``. Adding a new
source means adding one entry to the registry dict:

.. code-block:: python

   "my_source": ("probes.my_source.connection", "connect", "source_id")

See :doc:`extending` for the full guide.