Delta Engine

The Delta Engine enables incremental metadata extraction. Instead of re-extracting all records on every run, it compares the current state against the previous scan and emits only new, modified, or deleted records.

Usage

Add --delta to any metapod run command:

# First run — extracts everything, saves state
metapod run "oracle.metadata.*" --delta

# Second run — only emits changed records
metapod run "oracle.metadata.*" --delta

# Force full re-extract (ignore saved state)
metapod run "oracle.metadata.*" --delta --force

How it works

On each run the DeltaEngine:

Loads previous state from output/.delta/<probe_name>.json
Runs the probe normally (full extraction)
For each record, extracts a unique key and change signals
Compares against previous state to classify: new, modified, unchanged
Detects deletions (previous keys absent from current run)
Saves new state atomically
Returns only changed records

Change detection strategies

The engine auto-selects the strongest available signal per record:

Strategy	Fields detected	Use case
Hash	sql_text, source_code, definition, view_definition	Procedure/view source changes
Mtime	last_modified, modified, last_analyzed, create_date	DDL changes, file updates
Size	row_count, num_rows, size_bytes	Data volume drift
Key only	table_name, name, path, id	New/deleted detection when no other signals exist

Priority: hash > mtime > size. If all signals match, the record is unchanged.

State persistence

State is stored as JSON in output/.delta/:

{
  "probe_name": "oracle.metadata.tables",
  "last_scan": "2026-03-29T12:00:00+00:00",
  "scan_count": 3,
  "resources": {
    "CUSTOMERS": {"key": "CUSTOMERS", "mtime": 1711700000.0, "size": 1500, "hash": ""},
    "ORDERS": {"key": "ORDERS", "mtime": 1711700000.0, "size": 3200, "hash": ""}
  }
}

Writes are atomic (write to .tmp, then rename) to prevent corruption.

API

from core.delta import DeltaEngine

engine = DeltaEngine(state_dir=Path("./output/.delta"))
result, stats = engine.run(probe, connection, schema)

# result.records — only changed records
# stats.new, stats.modified, stats.deleted, stats.unchanged