YData Profiling
===============

**3 probes**

Integration with `YData Profiling <https://docs.profiling.ydata.ai/>`_
(formerly pandas-profiling) for automated dataset profiling and statistical
anomaly detection.  Profile reports are condensed into metapod-compatible
JSON for downstream alerting and trend analysis.

Install
-------

.. code-block:: bash

   pip install 'metapod-probes[ydata]'

This pulls in ``ydata-profiling`` as a transitive dependency.

Configuration
-------------

.. code-block:: yaml

   sources:
     - type: ydata
       dataset_path: /data/landing/*.parquet
       minimal: true          # fast mode — skip heavy correlations
       anomaly_threshold: 3.0 # z-score threshold for anomaly flagging

Probes
------

.. list-table::
   :header-rows: 1
   :widths: 40 60

   * - Probe
     - Description
   * - ``ydata.profiling.profile``
     - Generate dataset profile (row count, column stats, missing %, duplicates)
   * - ``ydata.profiling.anomalies``
     - Flag statistical anomalies vs. historical baselines (z-score)
   * - ``ydata.profiling.correlations``
     - Extract inter-column correlation matrix and high-correlation warnings

Usage
-----

.. code-block:: bash

   metapod run "ydata.profiling.*" --config config.yaml

Output
------

.. code-block:: json

   {
     "probe": "ydata.profiling.profile",
     "source": "ydata",
     "collected_at": "2026-03-31T08:00:00Z",
     "results": [
       {
         "dataset": "orders.parquet",
         "row_count": 14230,
         "columns": 18,
         "missing_pct": 0.4,
         "duplicate_rows": 12,
         "profiling_duration_s": 3.7
       }
     ]
   }