YData Profiling

3 probes

Integration with YData Profiling (formerly pandas-profiling) for automated dataset profiling and statistical anomaly detection. Profile reports are condensed into metapod-compatible JSON for downstream alerting and trend analysis.

Install

pip install 'metapod-probes[ydata]'

This pulls in ydata-profiling as a transitive dependency.

Configuration

sources:
  - type: ydata
    dataset_path: /data/landing/*.parquet
    minimal: true          # fast mode — skip heavy correlations
    anomaly_threshold: 3.0 # z-score threshold for anomaly flagging

Probes

Probe	Description
`ydata.profiling.profile`	Generate dataset profile (row count, column stats, missing %, duplicates)
`ydata.profiling.anomalies`	Flag statistical anomalies vs. historical baselines (z-score)
`ydata.profiling.correlations`	Extract inter-column correlation matrix and high-correlation warnings

Usage

metapod run "ydata.profiling.*" --config config.yaml

Output

{
  "probe": "ydata.profiling.profile",
  "source": "ydata",
  "collected_at": "2026-03-31T08:00:00Z",
  "results": [
    {
      "dataset": "orders.parquet",
      "row_count": 14230,
      "columns": 18,
      "missing_pct": 0.4,
      "duplicate_rows": 12,
      "profiling_duration_s": 3.7
    }
  ]
}