Standalone Usage
================

metapod works as a standalone tool — no remote platform required.
Extract metadata locally, inspect the JSON output, pipe it to your own
systems, or use it in CI/CD.

Local file mode (default)
-------------------------

Every ``metapod run`` command writes JSON to ``./output/``:

.. code-block:: bash

   metapod run "oracle.metadata.*" --config config.yaml --output ./output

   # Output structure:
   # ./output/
   #   oracle/
   #     metadata/
   #       tables.json
   #       columns.json
   #       views.json
   #       source_code.json
   #       ...

No ``push`` step required. The JSON files are the product.

Output format
-------------

Each probe writes one JSON file — a ``ProbeResult`` envelope:

.. code-block:: json

   {
     "probe": "oracle.metadata.tables",
     "source": {
       "type": "oracle",
       "id": "prod-db:1521/ORCLPDB1/RISK_MGMT"
     },
     "timestamp": "2026-03-29T12:00:00+00:00",
     "records": [
       {
         "table_name": "CUSTOMERS",
         "owner": "RISK_MGMT",
         "row_count": 15000,
         "created": "2025-06-15 10:30:00",
         "modified": "2026-03-28 08:45:00"
       },
       {
         "table_name": "ORDERS",
         "owner": "RISK_MGMT",
         "row_count": 842000,
         "created": "2025-06-15 10:31:00",
         "modified": "2026-03-29 02:15:00"
       }
     ],
     "record_count": 2,
     "duration_ms": 234,
     "error": null
   }

Fields:

- ``probe`` — fully qualified probe name
- ``source.type`` — connector type (oracle, postgres, azure_sql, ...)
- ``source.id`` — connection identifier (host, port, service, schema)
- ``records`` — array of extracted metadata records (shape varies by probe)
- ``record_count`` — length of records array
- ``duration_ms`` — execution time in milliseconds
- ``error`` — null on success, error message on failure

Reading output in Python
~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: python

   import json
   from pathlib import Path

   # Load all probe results
   for jf in Path("./output").rglob("*.json"):
       data = json.load(jf.open())
       print(f"{data['probe']}: {data['record_count']} records")
       for record in data["records"]:
           print(f"  {record}")

Converting to CSV
~~~~~~~~~~~~~~~~~

.. code-block:: bash

   # Quick one-liner: tables to CSV
   python3 -c "
   import json, csv, sys
   data = json.load(open('output/oracle/metadata/tables.json'))
   w = csv.DictWriter(sys.stdout, fieldnames=data['records'][0].keys())
   w.writeheader()
   w.writerows(data['records'])
   " > tables.csv

Converting to pandas DataFrame
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: python

   import json
   import pandas as pd

   data = json.load(open("output/oracle/metadata/columns.json"))
   df = pd.DataFrame(data["records"])
   print(df.head())
   # Filter to a specific table
   print(df[df["table_name"] == "CUSTOMERS"])

Comparing extractions (diff)
----------------------------

Run probes twice and compare:

.. code-block:: bash

   # First extraction
   metapod run "oracle.metadata.*" --output ./output

   # Later...
   mv ./output ./output.prev
   metapod run "oracle.metadata.*" --output ./output

   # Compare
   metapod diff --current ./output --previous ./output.prev

Output shows added, removed, and changed records per probe.

Incremental extraction (delta mode)
------------------------------------

Skip unchanged resources on re-runs:

.. code-block:: bash

   # First run — extracts everything
   metapod run "oracle.metadata.*" --delta --output ./output

   # Second run — only changed tables/columns emitted
   metapod run "oracle.metadata.*" --delta --output ./output

See :doc:`delta-engine` for details on change detection strategies.

Daemon mode (pull)
------------------

Serve probe results via HTTP — other tools call metapod on demand:

.. code-block:: bash

   metapod serve --config config.yaml --port 9090

   # From any consumer:
   curl http://localhost:9090/probes/oracle.metadata.tables
   curl http://localhost:9090/catalog
   curl http://localhost:9090/health

Optional: push to a remote platform
------------------------------------

If you use Dataflix, DataHub, or a custom API, push results after extraction:

.. code-block:: bash

   # To Dataflix
   metapod push --target https://dataflix.example.com --project my-project

   # To any REST API (custom script)
   python3 push_to_datahub.py --input ./output

See :doc:`integrations` for platform-specific guides.

Scheduling extractions
----------------------

Run probes on a cron schedule (blocks forever):

.. code-block:: bash

   metapod schedule "oracle.metadata.*" --cron "0 6 * * *" --config config.yaml

For production, prefer a container scheduler (Kubernetes CronJob, Airflow DAG).
See :doc:`docker-guide`.