A pipeline fails at 2 AM. The on-call engineer opens the job logs, sees a generic Spark exception, and starts the ritual: check the source table, check the schema, check the upstream job that feeds it, check whether someone touched the ADF trigger, check Slack for any "quick change" nobody documented. Forty-five minutes later, the cause turns out to be a column rename three hops upstream, in a table this pipeline doesn't even directly reference but depends on transitively.

None of that forty-five minutes was wasted effort exactly — it was necessary detective work. But it was manual detective work, repeated by a different engineer, slightly differently, every time a pipeline breaks. The knowledge of "how we traced the last five incidents" rarely survives past the postmortem doc, if a postmortem doc even gets written.

Three things make root cause analysis (RCA) structurally hard at scale:

• Lineage complexity. A single Gold-layer table might sit six or seven transformations downstream of a dozen source systems. Tracing "what changed upstream" by hand means walking a graph humans were never built to hold in working memory.

• Unstructured logs. Spark stack traces, ADF activity logs, and Databricks job-run history are text-heavy, inconsistent in format, and scattered across systems. Correlating "this job failed" with "that upstream job also had an unusual run at the same time" requires cross-referencing timestamps across sources that don't share a common schema.

• Institutional memory gaps. The senior engineer who instinctively knows "when Table X looks weird, check the CRM sync job first" is a single point of failure. When they're on leave, in a different time zone, or have left the team, that pattern-matching disappears with them.

The war room — pulling in five engineers to stare at dashboards and logs together — is not a process. It's an admission that no single person, and no tooling, can trace the failure alone.

2. The AI Opportunity

RCA is a search-and-correlate problem before it's anything else: given a failure, search the lineage graph for what it depends on, search the logs and metadata for what changed in a relevant time window, and correlate the two. That search-and-correlate step is exactly what AI is well-suited to accelerate — not to replace the engineer's judgment on what the failure means, but to compress the time spent gathering the evidence the judgment is applied to.

Two capabilities matter here, and they do different jobs:

Lineage-graph traversal answers "what could have caused this" by mechanically walking upstream dependencies from the failed asset — tables, jobs, notebooks — and surfacing what else touches the same objects. This is deterministic graph work. No LLM required, no hallucination risk. It's the evidentiary backbone.

LLM-assisted synthesis answers "given this evidence, what's the most likely story" by taking the structured output of the lineage walk plus a bundle of correlated log excerpts and schema-change events, and producing a ranked set of hypotheses in plain language — the kind of narrative a senior engineer would sketch on a whiteboard, but generated in seconds and grounded in the specific evidence retrieved, not general knowledge.

The important architectural point: the LLM never gets to invent the lineage or the log content. It only synthesizes from structured evidence that a deterministic system already retrieved. This is the difference between an RCA assistant and an RCA hallucination generator.

Karta — the doer — is the engineer who used to manually walk every log and every lineage edge. Sakshi — the witness — is the engineer who now reviews a pre-assembled evidence bundle and a ranked hypothesis list, and applies judgment to accept, reject, or redirect. The investigation still requires a human. What changes is where the forty-five minutes goes: from gathering evidence to evaluating it.

3. Implementation Sketch

A practical shape on Azure + Databricks + Unity Catalog: three stages — evidence retrieval, evidence bundling, and LLM-assisted synthesis — kept strictly separate so the LLM only ever reasons over facts you've already retrieved.

Stage 1 — Lineage retrieval. Unity Catalog's system tables expose table- and column-level lineage. Walk upstream from the failed asset within a bounded time window around the failure:

from pyspark.sql import functions as F

failed_table = "prod.sales.gold_daily_revenue"
failure_ts = "2026-07-28T02:14:00Z"
lookback_hours = 24

lineage_df = spark.table("system.access.table_lineage")

upstream = (
    lineage_df
    .filter(F.col("target_table_full_name") == failed_table)
    .filter(F.col("event_time").between(
        F.to_timestamp(F.lit(failure_ts)) - F.expr(f"INTERVAL {lookback_hours} HOURS"),
        F.to_timestamp(F.lit(failure_ts))
    ))
    .select("source_table_full_name", "source_type", "event_time")
    .distinct()
)

Stage 2 — Correlated event bundling. For each upstream table surfaced, pull job-run history and any schema-change events in the same window — the goal is a compact, structured "evidence bundle," not raw logs:

job_runs = spark.table("system.lakeflow.job_run_timeline")

correlated_runs = (
    job_runs
    .join(upstream, job_runs.table_full_name == upstream.source_table_full_name, "inner")
    .filter(job_runs.period_start_time.between(
        F.to_timestamp(F.lit(failure_ts)) - F.expr(f"INTERVAL {lookback_hours} HOURS"),
        F.to_timestamp(F.lit(failure_ts))
    ))
    .select("table_full_name", "run_id", "result_state",
            "period_start_time", "period_end_time")
)

evidence_bundle = {
    "failed_asset": failed_table,
    "failure_time": failure_ts,
    "upstream_assets": [row.source_table_full_name for row in upstream.collect()],
    "correlated_job_runs": [row.asDict() for row in correlated_runs.collect()],
    "error_excerpt": get_error_excerpt(failed_table, failure_ts, max_chars=800),
}

get_error_excerpt is a custom helper, not a native PySpark or system-table function — you write it to pull from wherever your job-run error output actually lands (Databricks job-run API, system.lakeflow.job_run_timeline error fields, or your logging sink), truncate it, and hand back a plain string.

Note the error_excerpt cap and the absence of raw row-level data anywhere in the bundle — only table names, job metadata, and a bounded error message make it into what eventually reaches the LLM.

Stage 3 — LLM-assisted synthesis. Pass the structured bundle — never raw data, never full logs — to the model with a prompt constrained to reason only over what's supplied:

system_prompt = """You are assisting a data engineer with root cause analysis.
You will be given a structured evidence bundle: a failed table, upstream
dependencies, correlated job-run outcomes, and an error excerpt.

Rules:
- Base every hypothesis strictly on the evidence provided. Do not assume
  facts not present in the bundle.
- Produce a ranked list of up to 3 hypotheses with a confidence label
  (high / medium / low) and the specific evidence supporting each.
- For each hypothesis, state one concrete next diagnostic step the
  engineer should take to confirm or rule it out.
- If the evidence is insufficient to form a hypothesis, say so explicitly
  rather than speculating."""

response = call_llm(system_prompt=system_prompt, user_content=json.dumps(evidence_bundle))

The output is a starting hypothesis list with pointers back to evidence — not a verdict. It gets logged alongside the incident and reviewed by the engineer before any action is taken.

4. Limitations & Risks

Hallucination in the synthesis step. Even constrained to a structured bundle, an LLM can still overstate confidence or invent plausible-sounding connections between unrelated events. This is maya again — the same appearance-mistaken-for-substance risk this series keeps returning to, because it keeps showing up wherever an LLM sits between evidence and a decision. The constraint isn't optional: every hypothesis needs a "confirm or rule out" step attached, and no hypothesis should move to remediation without that confirmation.

Lineage completeness gaps. Unity Catalog lineage only captures what runs through Unity Catalog–governed compute. Jobs that read or write outside that boundary — legacy ADF activities hitting non-UC storage, external ETL tools — create blind spots in the graph. A confident-looking lineage walk that's silently incomplete is worse than an honest "lineage unavailable beyond this point."

Correlation is not causation. Two upstream jobs running near the failure time doesn't mean either caused it. The LLM synthesis step can make coincidental timing look like a causal narrative if the prompt and evidence bundle aren't explicit about what "correlated" means. Confidence labels help, but they don't eliminate the risk — engineer review is the actual safeguard, not a formality.

Data exposure through error excerpts. Error messages and stack traces can contain fragments of actual data values — a failed row, a bad key, a customer identifier embedded in an exception message. Sending that verbatim to an external LLM API is a PII exposure path that's easy to miss because it doesn't look like "sending data" the way a table export does.

Cost and latency at incident volume. A single evidence bundle plus LLM call is cheap. Hundreds of failures a week, each triggering lineage walks and LLM synthesis, is not free — and if RCA assist becomes another noisy layer that fires on every transient job retry, it adds cost without adding signal.

5. How to Overcome

Redact before synthesis, not after. Run error excerpts through a redaction pass — pattern-matching known PII formats, masking literal values in exception messages — before they enter the evidence bundle. Treat this as a mandatory pipeline stage, not a best-effort filter.

Scope lineage claims honestly. Have the retrieval stage explicitly flag when it hits the edge of Unity Catalog's visibility, and pass that flag into the evidence bundle so the LLM prompt — and the engineer — knows the lineage picture is partial, not complete.

Require evidence citations in every hypothesis. Enforce, at the prompt level, that each hypothesis names the specific evidence field it draws from. A hypothesis with no traceable evidence pointer gets discarded before it reaches the engineer, not treated as a lower-confidence option.

Gate on severity, not on every failure. Trigger the full RCA-assist pipeline only for failures above a severity threshold — SLA-critical tables, repeated failures on the same asset, failures affecting downstream reporting. Transient retries and known-flaky jobs don't need lineage walks and LLM calls.

Close the loop with a feedback table. Log whether each engineer confirmed, rejected, or modified the top hypothesis. Review this monthly. If confirmed-hypothesis rate drops, that's a signal to revisit the prompt, the evidence bundle composition, or the lineage retrieval scope — not to quietly stop trusting the tool.

6. The Takeaway

For engineers: Next incident you trace manually, write down the actual evidence trail you followed — which tables you checked, in what order, what made you rule things in or out. That trail is the specification for what an RCA-assist bundle needs to contain. You cannot automate a search you haven't first done deliberately by hand.

For leads: Ask your team how postmortem knowledge currently gets reused — is it a doc nobody rereads, or a live system that makes the next incident faster to trace? RCA-assist tooling is only worth building if the evidence-gathering step is genuinely the bottleneck, not the judgment step. Measure time-to-root-cause before and after, not just adoption.

AI doesn't eliminate engineering judgment — it demands better judgment, faster.

Next: Article 5 — Metadata Intelligence: Making Your Catalog Work for You. Turning Unity Catalog from a static inventory into an actively maintained knowledge base with LLM-assisted description generation and PII classification.

Karthik Darbha is a Senior Data Engineering & AI Leader with 23 years of professional experience, including 20+ years building enterprise data platforms across Healthcare, Pharma, Retail, Insurance, and Financial Services. He writes about data engineering, program management, and the intersection of technology and philosophy at tech4nirvana.com.

Most pipeline observability today is a wall of thresholds. Row count dropped below X. Job ran longer than Y minutes. Null percentage exceeded Z. Someone picked those numbers months ago, often by guessing, and the alerts have been drifting out of relevance ever since.

The result is a familiar failure mode: real incidents slip through because they don't cross a static threshold, while noisy, low-value alerts fire constantly and get muted. Engineers stop trusting the monitoring, which means they stop looking at it — until a downstream report is wrong and someone asks why nothing caught it.

This isn't a tooling gap. Most teams already have Databricks system tables, Delta Lake transaction logs, and job-run metadata sitting there, rich with signal. The gap is that nobody's watching it with anything more sophisticated than IF value > threshold.

The AI Opportunity

Observability is fundamentally a pattern-recognition problem, and that's exactly where statistical and ML methods outperform static rules.

Three shifts are now practical at reasonable engineering cost:

• From fixed thresholds to learned baselines. Time-series models (seasonal decomposition, Prophet-style forecasting, or simpler rolling z-scores) learn what "normal" looks like for a given pipeline — including weekly and monthly seasonality — and flag deviations from that learned baseline instead of a number picked six months ago.

• From single-metric checks to correlated signals. A row-count dip alone might be noise. A row-count dip combined with a schema change upstream and a spike in null rates is a pattern worth an engineer's attention. Lightweight anomaly detection across correlated features catches this; single-metric thresholds cannot.

• From "did it fail" to "will it fail." Job duration trending upward over several runs, or memory usage creeping toward cluster limits, is a leading indicator. Forecasting these trends gives you a warning before the 2 AM page, not after.

None of this requires a dedicated ML platform. It requires treating your existing job-run and lineage metadata as a first-class dataset.

Implementation Sketch

A practical starting point on Databricks: build an observability feature table from system tables and job-run history, then apply a lightweight anomaly model per pipeline — one that respects two realities a naive rolling z-score ignores: pipeline metrics are rarely Gaussian, and most have weekly seasonality.

from pyspark.sql import functions as F
from pyspark.sql.window import Window

# Pull job run metrics, tagged by day-of-week to respect weekly seasonality
run_metrics = (
    spark.table("system.lakeflow.job_run_timeline")
    .filter(F.col("result_state") == "SUCCESS")
    .withColumn("run_date", F.to_date("period_start_time"))
    .withColumn("day_of_week", F.dayofweek("run_date"))
    .groupBy("job_id", "run_date", "day_of_week")
    .agg(F.avg("run_duration_seconds").alias("avg_duration"))
)

# Baseline computed per (job_id, day_of_week) — trailing 8 same-weekday
# occurrences (~8 weeks), so a Saturday batch run is compared to past
# Saturdays, not to Tuesday's lighter load.
baseline_window = (
    Window.partitionBy("job_id", "day_of_week")
    .orderBy("run_date")
    .rowsBetween(-8, -1)
)

scored = (
    run_metrics
    # Median + MAD instead of mean + stddev: robust to the skewed,
    # multi-modal distributions that duration and row-count metrics
    # typically have (batch-size variance, cache effects, cluster
    # contention). A rolling z-score on raw mean/stddev is the single
    # most common way this kind of check misfires in production.
    .withColumn("baseline_median", F.expr("percentile_approx(avg_duration, 0.5)").over(baseline_window))
    .withColumn("abs_deviation", F.abs(F.col("avg_duration") - F.col("baseline_median")))
    .withColumn("mad", F.expr("percentile_approx(abs_deviation, 0.5)").over(baseline_window))
    .withColumn("history_count", F.count("*").over(baseline_window))
    .withColumn(
        # 0.6745 scales MAD to be comparable to a stddev-based z-score
        "robust_z_score",
        F.when(F.col("mad") > 0, 0.6745 * F.col("abs_deviation") / F.col("mad"))
    )
    # Automated cold-start handling: a pipeline only enters adaptive
    # mode once it has 8 same-weekday data points; until then it's
    # explicitly flagged for static-threshold monitoring instead of
    # silently producing an unreliable score.
    .withColumn(
        "monitoring_mode",
        F.when(F.col("history_count") >= 8, F.lit("adaptive")).otherwise(F.lit("static_fallback"))
    )
    .filter((F.col("monitoring_mode") == "adaptive") & (F.abs(F.col("robust_z_score")) > 3.5))
)

This is deliberately simple — median/MAD and day-of-week partitioning, not a neural network. For most pipeline-level metrics (duration, row count, null rate), this level of sophistication catches the majority of real anomalies without the distributional assumptions a plain z-score makes. Reserve heavier models (isolation forests, autoencoders, or explicit seasonal decomposition) for high-dimensional cases like multi-column data drift or pipelines with irregular, non-weekly cycles, where a single metric or a fixed weekday lag can't capture the pattern.

One scale caveat: percentile_approx inside an open-ended window clause is a non-additive, shuffle-heavy operation. Recomputing it over raw run-logs inline is fine for dozens of pipelines; at thousands of jobs running daily, pre-aggregate into a daily per-(job_id, day_of_week) state table and compute the rolling median/MAD against that smaller table instead of the raw log.

Route flagged anomalies into a Delta table feeding your existing alerting channel, tagged with the contributing signals — not just "duration anomaly" but "duration anomaly, coincides with upstream schema change at 03:14."

Limitations and Risks

• Cold start, now explicit rather than hidden. The monitoring_mode flag stops a pipeline from getting an unreliable score during its first 8 weeks, but someone still has to wire the static_fallback branch to an actual threshold check — the code makes the state visible, it doesn't eliminate the manual setup.

• MAD is more robust, not distribution-free. Median/MAD tolerates skew and outliers far better than mean/stddev, but it still assumes a single dominant mode. A pipeline with genuinely bimodal runtime behavior — a fast path when Spark hits cache versus a slow path when it spills to disk — will settle the median between or on one mode, producing erratic scores either way. That needs a segmented or mixture-aware approach, not this one.

• Shuffle cost at scale. percentile_approx in an open-ended window clause is non-additive and shuffle-heavy. Fine for dozens of pipelines computed inline against raw logs; at thousands of jobs running daily, pre-aggregate into a daily state table first rather than recomputing the full window over raw run-logs.

• Baseline drift as false comfort. If a pipeline's behavior degrades slowly, a rolling baseline "learns" the degradation as the new normal and stops flagging it. This is the single biggest failure mode of adaptive monitoring — it can quietly raise your tolerance for bad behavior instead of catching it.

• Alert fatigue, relocated not solved. Swapping static thresholds for statistical ones doesn't guarantee fewer false positives if the underlying metric is inherently noisy (e.g., source system volume genuinely varies day to day). Tuning still matters.

• Who watches the watcher. An anomaly-detection layer is itself a system that can fail silently — a stale baseline table, a broken feature pipeline. It needs its own basic health check, or it becomes a false sense of security.

How to Overcome

• Cap baseline windows and set a floor (e.g., require 30 days of history before adaptive scoring kicks in; use static thresholds before that).

• Add a long-window drift check alongside the short-window anomaly check — compare this month's baseline to the baseline from three months ago. A quiet, gradual shift over that longer window is worth a human look even if no single day tripped an alert.

• Keep a human-in-the-loop feedback mechanism: when someone dismisses an alert as noise, capture that as a label. Be honest about what this buys you — feeding binary dismissal labels back into a median/MAD threshold isn't a natural fit; treat the labeled data as the seed of a future supervised model, not a live input to today's statistical check.

• Monitor the monitoring pipeline itself with a simple heartbeat check — last successful run time, row counts written — using the same basic tooling you'd apply to any production job.

The Takeaway

For engineers: start with your job-run system tables and one pipeline's duration metric. A rolling z-score is a day of work, not a project, and it will outperform a static threshold within a month.

For leads: the real gain isn't fewer alerts — it's alerts that mean something. Track the ratio of actioned-to-dismissed alerts before and after adaptive monitoring goes in; that number, not alert volume, is the metric that tells you whether observability is actually working.

In Advaita Vedanta, avidya isn't the absence of information — it's the presence of information you've stopped truly seeing. A threshold breached ten times a day for six months isn't ignorance; it's information the team has learned to look past. Adaptive observability doesn't add data. It restores the seeing.

Karthik Darbha is a Senior Data Engineering & AI Leader with 23 years of professional experience, including 21+ years building enterprise data platforms across Healthcare, Pharma, Retail, Insurance, and Financial Services. He writes about data engineering, program management, and the intersection of technology and philosophy at tech4nirvana.com.

A critical dashboard starts showing numbers that don't feel right. An analyst flags it Friday afternoon. The on-call engineer traces it back — a source table's null rate tripled two days ago. The rule-based data quality tests passed. The pipeline ran green. Nothing fired.

The rules said the data was fine. The data was not fine.

This is the fundamental failure mode of rule-based data quality: it can only catch what someone thought to check for, in advance, expressed as a threshold someone was willing to defend. At the scale and velocity of modern data platforms, that contract breaks constantly — and silently.

1. The Problem — What Rules Cannot See

Rule-based quality systems are built on a foundational assumption: you know what "bad" looks like before it happens. You express that knowledge as constraints: not null, within range, referential integrity, row count within ±10% of yesterday's.

This works well for known failure modes. It is structurally incapable of catching unknown ones.

Consider what rules miss:

Distribution drift — A revenue column's values quietly shift from a mean of \(1.2M to \)1.9M over three weeks. Every individual row is within the defined range. The aggregate distribution has fundamentally changed. No threshold fires.

Seasonal baseline collapse — A weekly batch table that normally lands with 2.1M rows on Monday arrives with 1.8M. That's within the ±10% rule. But your team doesn't know that last Monday was a holiday in the source system and 1.8M is now the new normal. The rule no longer means anything.

Correlated column anomalies — A transaction table where amount and discount_pct move in unexpected correlation — not individually out of range, but together, statistically improbable. Rule sets do not model joint distributions.

Silent schema migrations — A source system begins zero-padding a product code column. String lengths change. Values are still non-null and string-typed. Every rule passes. Downstream joins silently fail.

In Advaita, tamas describes the quality of inertia, opacity, and entropy — the state in which things lose their clarity and coherence without the obvious appearance of breaking. Rule-based data quality operates in a tamasic mode: it responds only when something crosses an explicit threshold, while the underlying signal degrades unremarked.

The math of rule maintenance compounds this. A platform with 500 Delta tables, each averaging 40 monitored columns, requires 20,000 rules — each of which must be authored, calibrated, and updated when distributions legitimately shift. Most teams maintain a fraction of that, concentrated on tables someone already knows are important.

The rest of the estate runs on trust.

2. The AI Opportunity — What's Now Tractable

The shift AI enables is from explicit constraint to learned expectation. Instead of asking "does this value violate a rule I wrote?", the system asks "does this value behave the way this column historically has?"

That reframe opens up several capabilities that were not practically achievable before:

Statistical anomaly detection per column — Models that learn the distribution of each column over time (mean, variance, skew, null rate, value frequency) and flag statistically significant deviations. No threshold authoring required.

Multivariate anomaly detection — Models like Isolation Forest or Autoencoders that learn the joint distribution across columns and flag rows that are individually plausible but collectively improbable. This catches correlated anomalies that column-level rules miss entirely.

Adaptive baselines — Instead of a static ±10% row count rule, a time-series model learns day-of-week, week-of-month, and seasonal patterns. It flags Monday arrivals against the distribution of past Monday arrivals — accounting for holidays, promotions, and growth trends.

Automated profiling at catalog scale — Profile generation (column statistics, value distributions, null rates, cardinality) can run as a background Databricks job across your entire Delta Lake, feeding both anomaly models and Unity Catalog enrichment.

Confidence-aware alerting — Instead of binary pass/fail, ML-based systems can emit anomaly scores. Downstream logic can tier the response: high-confidence anomaly triggers quarantine; medium-confidence triggers review; low-confidence logs for trend analysis.

The practical entry point for most Azure/Databricks teams is a two-layer architecture: keep your existing rule-based tests (they catch hard constraints cheaply), and add an ML anomaly detection layer that operates on distributions and scores deviations. The rules handle known failure modes. The models handle the unknown.

3. Implementation Sketch — Databricks + MLflow Anomaly Scorer on Delta

The following is a working implementation pattern, not production-ready code. It demonstrates the architecture decisions; your team will adapt thresholds, scheduling, and alerting to your context.

Architecture

ADF Pipeline Trigger
      │
      ▼
Databricks Job: Profile & Score
      │
      ├── Step 1: Column Profiling (PySpark → Delta stats table)
      ├── Step 2: Feature Engineering (rolling stats, drift metrics)
      ├── Step 3: MLflow Model Inference (Isolation Forest per table)
      └── Step 4: Write anomaly scores → Delta quality log table
                        │
                        ├── High-score rows → Quarantine path
                        └── Alerts → Teams/PagerDuty via ADF webhook

Step 1 — Profile a Delta Table

from pyspark.sql import functions as F
from pyspark.sql import DataFrame

def profile_delta_table(df: DataFrame, table_name: str, run_ts: str) -> DataFrame:
    """
    Compute per-column statistics for anomaly feature generation.
    Returns a single-row DataFrame of profile metrics.
    """
    numeric_cols = [f.name for f in df.schema.fields
                    if f.dataType.typeName() in ("double", "float", "long", "integer", "decimal")]

    profile = {"table_name": table_name, "run_ts": run_ts, "row_count": df.count()}

    for col in numeric_cols:
        stats = df.select(
            F.mean(col).alias("mean"),
            F.stddev(col).alias("stddev"),
            F.expr(f"percentile({col}, 0.25)").alias("p25"),
            F.expr(f"percentile({col}, 0.75)").alias("p75"),
            (F.sum(F.when(F.col(col).isNull(), 1).otherwise(0)) / F.count("*")).alias("null_rate")
        ).first()
        profile[f"{col}_mean"]      = stats["mean"]
        profile[f"{col}_stddev"]    = stats["stddev"]
        profile[f"{col}_p25"]       = stats["p25"]
        profile[f"{col}_p75"]       = stats["p75"]
        profile[f"{col}_null_rate"] = stats["null_rate"]

    return spark.createDataFrame([profile])

Production note: On multi-terabyte Delta tables, exact percentile() triggers a full data shuffle across the cluster. For large-scale deployments, prefer Delta Lake's transaction log column statistics (available via DESCRIBE DETAIL) for min/max/null counts, and use percentile_approx() with a t-digest algorithm for distribution estimates. This significantly reduces compute cost without meaningful loss in anomaly detection accuracy.

Step 2 — Train and Register an Isolation Forest Model

import mlflow
import mlflow.sklearn
import pandas as pd
from sklearn.ensemble import IsolationForest

def train_anomaly_model(profile_history_df: pd.DataFrame, table_name: str):
    """
    Train an Isolation Forest on historical profile snapshots.
    Log model to MLflow with table-scoped experiment.
    """
    # Drop non-feature columns
    feature_df = profile_history_df.drop(columns=["table_name", "run_ts", "row_count"])
    feature_df = feature_df.fillna(feature_df.median())

    model = IsolationForest(
        n_estimators=100,
        contamination=0.05,   # assume ~5% of historical snapshots were anomalous
        random_state=42
    )
    model.fit(feature_df)

    mlflow.set_experiment(f"/data_quality/{table_name}")
    with mlflow.start_run(run_name="isolation_forest_v1"):
        mlflow.log_param("contamination", 0.05)
        mlflow.log_param("n_estimators", 100)
        mlflow.sklearn.log_model(model, artifact_path="model",
                                  registered_model_name=f"dq_anomaly_{table_name}")
    return model

Step 3 — Score a New Batch Profile

def score_profile(current_profile: pd.DataFrame, model) -> dict:
    """
    Score the current run's profile against the trained model.
    Returns anomaly flag and raw score.
    """
    feature_df = current_profile.drop(columns=["table_name", "run_ts", "row_count"])
    feature_df = feature_df.fillna(0)

    score   = model.decision_function(feature_df)[0]   # higher = more normal
    is_anomaly = model.predict(feature_df)[0] == -1    # -1 = anomaly

    return {
        "anomaly_score": float(score),
        "is_anomaly": bool(is_anomaly),
        "severity": "HIGH" if score < -0.15 else "MEDIUM" if score < -0.05 else "LOW"
    }

When is_anomaly is True, surface the contributing columns using SHAP before routing the alert:

import shap

def explain_anomaly(feature_df: pd.DataFrame, model, top_n: int = 3) -> list:
    """
    Use SHAP to identify the top contributing columns to an anomaly score.
    Returns a ranked list of (column, shap_value) tuples for alert enrichment.
    """
    explainer = shap.TreeExplainer(model)
    shap_values = explainer.shap_values(feature_df)

    # shap_values shape: (n_samples, n_features)
    feature_importance = sorted(
        zip(feature_df.columns, abs(shap_values[0])),
        key=lambda x: x[1],
        reverse=True
    )
    return feature_importance[:top_n]

# Usage: enrich the alert payload
if score_result["is_anomaly"]:
    top_contributors = explain_anomaly(feature_df, model)
    score_result["top_contributors"] = [
        {"column": col, "shap_value": round(val, 4)}
        for col, val in top_contributors
    ]

This transforms the alert payload from "transaction_fact looks 12% anomalous" into "transaction_fact anomaly — top drivers: amount_mean (0.34), discount_pct_null_rate (0.21), row_count (0.18)" — an actionable engineering ticket rather than an abstract flag.

Step 4 — Write Results to Delta Quality Log

from delta.tables import DeltaTable

result = {
    "table_name":     table_name,
    "run_ts":         run_ts,
    "anomaly_score":  score_result["anomaly_score"],
    "is_anomaly":     score_result["is_anomaly"],
    "severity":       score_result["severity"],
    "model_version":  model_version
}

spark.createDataFrame([result]).write \
    .format("delta") \
    .mode("append") \
    .saveAsTable("catalog.data_quality.anomaly_log")

Scheduling and Alerting

Trigger this job from ADF immediately after each pipeline run completes. Add a downstream ADF activity that reads the anomaly log, filters is_anomaly = true AND severity = 'HIGH', and calls a webhook to route alerts to Teams or PagerDuty. Medium-severity alerts can land in a Slack channel for async review.

Unity Catalog integration: surface the latest anomaly score as a custom metadata property on each table, so catalog users see quality status inline alongside lineage and descriptions.

4. Limitations & Risks

Cold start problem. Isolation Forest needs 30–60 historical profile snapshots to establish a meaningful baseline. On new tables, the model will either surface noise or fail to detect genuine anomalies. Rule-based tests are strictly better here until sufficient history accumulates.

Model drift. Data distributions legitimately evolve — business launches, seasonal cycles, source system migrations. An anomaly model trained on six-month-old history will generate false positives against a platform that has legitimately changed. Models need periodic retraining cadences and triggered resets on known change events.

Explainability gap. An Isolation Forest flags "this profile looks anomalous" without saying which column drove the score. Engineers need to investigate — SHAP values or simple statistical comparison against the historical baseline can help surface the likely culprit column, but this is not automatic. The SHAP enrichment in Step 3 above addresses this for the scoring path; extend it to alert payloads and runbooks for full coverage.

Wide table dimensionality. Isolation Forest degrades on tables with hundreds of columns — the profile history DataFrame becomes highly wide, triggering the curse of dimensionality. For such tables, apply PCA as a preprocessing step to reduce feature space before training, or switch to an Autoencoder-based anomaly detector which handles high-dimensional feature spaces more robustly.

False positives at scale. A model with 5% contamination assumption, applied across 500 tables on a daily batch, generates 25 anomaly alerts per run under normal operation — before any real issues. Without severity tiering and alert routing discipline, this recreates the alert fatigue problem you were trying to solve.

PII and metadata exposure. Sending column-level statistics (even aggregated) to external LLM APIs for enrichment or RCA narration can expose sensitive schema topology. Keep profile data and model training within your Azure tenant. MLflow on Databricks Managed MLflow keeps all model artifacts within your workspace.

Sattva — the quality of clarity, purity, and coherent signal — is not the default state of a data platform. It is achieved through continuous attention. An ML anomaly system that drifts, hallucinates confidence, or floods engineers with noise does not produce sattva. It produces a more sophisticated form of tamas: entropy wrapped in the appearance of intelligence.

5. How to Overcome

Hybrid layer: rules + ML. Do not decommission rule-based tests. Let them handle hard constraints (not null, referential integrity, known format rules) cheaply. The ML layer handles distribution-level anomalies. Two layers, two purposes, different cost profiles.

Gated automation. Start with anomaly scoring and human review. Do not auto-quarantine rows until your model has a track record. Move to automated quarantine only after validating precision on your specific tables over 4–6 weeks. Trust must be earned incrementally.

Column importance ranking. Not all columns are equal. Run SHAP on a sample of historical anomaly events to identify which columns contribute most to anomaly scores. Prioritize those for explainability tooling and manual review.

Retraining triggers. Define retraining events: a known source system migration, a business rule change, a data model restructuring. Add these as pipeline parameters that trigger automatic model retraining and version registration in MLflow. Don't wait for scheduled retraining cycles when you know the baseline has changed.

Model performance tracking. Log confirmed anomalies (true positives) and false positives into a feedback table. Track precision and recall over time using MLflow experiment comparison. Treat model degradation as an incident, not a background observation.

Phased rollout. Week 1–2: profile top 20 critical tables, establish baselines. Week 3–4: run models in shadow mode (score but don't alert). Week 5–6: enable alerting on HIGH severity only. Month 2: expand to broader estate.

6. The Takeaway

For engineers: This week, pick one high-value Delta table your team cares about and run a basic profiling job on the last 30 days of data. Get the column distributions on paper — mean, stddev, null rate, row counts by day. You cannot detect anomalies without a baseline. The baseline is the work that makes everything else possible.

For leads: Review your current data quality SLAs and ask: are they defined in terms of rule coverage (we have X assertions), or outcome quality (our analysts trust the numbers)? The shift from rules to reasoning is not primarily a tooling decision. It is a quality definition decision. Redefine quality as business confidence, not test passage rate — and the right tooling follows.

AI doesn't eliminate engineering judgment — it demands better judgment, faster.

Next: Article 3 — Observability That Thinks: AI for Pipeline Monitoring. AI-assisted classification of volume drift, freshness failures, and schema anomalies — with Databricks and Azure Monitor.

There is a quiet crisis in most data engineering teams.

Pipelines fail at 2 AM. The on-call engineer spends three hours tracing a root cause that turns out to be an upstream schema change no one documented. Data quality checks pass, but a business analyst flags numbers that "feel off." The data catalog has 4,000 assets — and 3,800 of them have no description, no owner, and no lineage.

None of this is a talent problem. These teams are capable. The problem is scale mismatch: the volume, velocity, and complexity of modern data infrastructure has outpaced what humans can monitor, document, and govern manually.

Avidya in Advaita does not mean ignorance in the ordinary sense. It means perceiving a thing without seeing its true nature. We have not built broken platforms — we have built platforms that outpaced our ability to see them clearly. AI, used well, closes that gap.

This is where AI enters — not as a replacement for engineering judgment, but as a force multiplier for it.

Where Manual DE Practices Hit Their Ceiling

Traditional data engineering runs on three pillars:

Rules and thresholds — Great Expectations suites, dbt tests, SQL assertions. Effective when you know what to check. Brittle when data distributions shift, business logic evolves, or new sources arrive.

Human-in-the-loop monitoring — Dashboards, alerts, war rooms. Effective when incidents are infrequent and teams are small. Breaks down at scale, across time zones, and under alert fatigue.

Documentation by convention — Wiki pages, Confluence, Notion. Effective when someone has time to write and keep them current. In practice, they decay within weeks of a system change.

These practices were designed for a world where a single team owned a handful of pipelines. The modern GCC data platform might have hundreds of pipelines, dozens of source systems, petabytes under management, and SLAs measured in minutes.

The math no longer works.

What AI Actually Changes

AI does not eliminate the need for engineering judgment. It changes where that judgment gets applied.

The shift is from production of insight to evaluation of it. Advaita calls this the movement from karta to sakshi — from doer to witness. The engineer is no longer the one who generates every answer. They become the one who discerns: which output is trustworthy, which needs correction, which reveals something the system itself cannot see. That is not a diminished role. It is a more demanding one.

The Opportunity Landscape

Across the articles in this series, we will explore five concrete domains where AI creates measurable leverage in data engineering:

1. Data Quality — Moving from static rule sets to adaptive anomaly detection. ML models that learn what "normal" looks like for a given table, column, or pipeline, and flag deviations before they surface in business reports.

2. Observability — From dashboards you reactively check, to systems that proactively surface signals. AI-assisted classification of volume drift, freshness failures, and schema anomalies — with prioritized alerts, not noise.

3. Root Cause Analysis — Correlating pipeline failures with upstream events, schema changes, and historical patterns. LLMs parsing structured logs and traversing lineage graphs to surface probable causes in minutes, not hours.

4. Metadata & Lineage Intelligence — Auto-generating column descriptions, inferring data classifications, enriching catalogs with semantic context. Turning your Unity Catalog or data catalog from a static inventory into an intelligent knowledge base.

5. Cost & Performance Optimization — Analyzing Spark query plans, cluster utilization patterns, and job run histories to recommend rightsizing, caching strategies, and partition improvements — before your cloud bill arrives.

Limitations and Risks You Must Reckon With

This series will not paper over the challenges. Every article will address them head-on because they are real, and ignoring them is how AI initiatives fail quietly.

Hallucination and low-confidence outputs. LLMs generating metadata descriptions or RCA hypotheses can be confidently wrong. Without a human review layer, bad outputs enter your catalog or runbooks as facts. This is maya in its most technically precise form — appearance without substance, mistaken for ground truth. The antidote is not distrust of AI, but structured discernment: every AI output in a production context needs a verification gate.

Observability of the AI layer itself. If your AI system monitors your pipelines, who monitors the AI system? Model drift, stale training data, and changing data distributions can silently degrade AI effectiveness. The irony is real.

Data privacy and PII exposure. Sending pipeline metadata, column names, or log excerpts to external LLM APIs may expose sensitive information — depending on your data contracts, regulatory environment (GDPR, HIPAA, DPDPA), and API provider's data retention policies. This is not theoretical.

False confidence from high-accuracy models. A model that flags anomalies with 92% precision sounds reliable. The 8% false positives on a high-volume pipeline still generate dozens of incorrect alerts per day. Worse, false negatives — missed anomalies — may not surface until business impact is visible.

Skill gap reality. Most data engineering teams are not ML teams. Deploying and maintaining AI components requires skills in model evaluation, feature engineering, and MLOps that typical DE profiles do not carry. Tooling alone will not close this gap.

Cost of AI at scale. LLM API calls on large metadata catalogs, high-frequency pipelines, or real-time log streams accumulate cost rapidly. An AI observability layer that itself becomes a significant line item is a governance failure.

The Governing Principle for This Series

Advaita speaks of viveka — discriminative discernment — as the foundational faculty for a serious practitioner. Not blind acceptance. Not reflexive rejection. The capacity to distinguish the real from the apparently real, the signal from the noise.

That is exactly what AI-augmented data engineering demands.

AI doesn't eliminate engineering judgment — it demands better judgment, faster.

Every recommendation in this series will be held to that standard. We will not advocate for AI because it is fashionable. We will advocate for it where it demonstrably reduces toil, improves reliability, or surfaces insight that humans structurally cannot produce at the required speed and scale.

And we will call out, explicitly, when a human-in-the-loop is not optional — it is the safeguard.

What to Expect

Each subsequent article in this series follows a consistent structure:

• The Problem — what breaks or costs without AI

• The AI Opportunity — what's now tractable and why

• Implementation Sketch — concept + lightweight code or architecture (primarily Azure / Databricks stack)

• Limitations & Risks — honest constraints, not footnotes

• How to Overcome — mitigation patterns, guardrails, phased adoption

• The Takeaway — one action for engineers, one for leads

Next in the series: Article 2 — AI-Driven Data Quality: From Rules to Reasoning. We'll move beyond dbt tests and static thresholds into ML-based anomaly detection on Delta tables, with an implementation sketch using Databricks and MLflow.

Karthik Darbha is a Senior Data Engineering & AI Leader with 23 years of professional experience, including 20+ years building enterprise data platforms across Healthcare, Pharma, Retail, Insurance, and Financial Services. He writes about data engineering, program management, and the intersection of technology and philosophy at tech4nirvana.com.

Most data platform programs don't fail loudly. They fail quietly — one missed dependency, one unreported pipeline issue, one status update that said 'green' while the underlying data quality was red.

After two decades in IT delivery and six-plus years building data platforms on Azure Databricks, I have seen a consistent pattern: technical execution is rarely the bottleneck. Visibility is.

The Technical Program Manager (TPM) on a Databricks or Azure data platform program is not just a project tracker. The TPM is the connective tissue between engineering reality and business expectation. And the primary tool of that role is structured, layered visibility.

This article is a practitioner's guide to building that visibility layer — from pipeline health to steering committee reporting — drawn from real delivery experience on medallion architecture rollouts, Unity Catalog migrations, and large-scale ADF-based ingestion programs.

The Visibility Stack: Four Layers That Matter

Effective program visibility is not a single dashboard. It is a stack of four interconnected layers, each serving a different audience and time horizon.

Each layer feeds the one above it. A pipeline failure at Layer 1 becomes a milestone risk at Layer 2, a dependency flag at Layer 3, and — if unresolved — a red RAG item at Layer 4. The TPM's job is to manage the signal flow across all four layers simultaneously.

Layer 1: Pipeline Health Monitoring

Databricks Job Monitoring

In a production Databricks environment, job health is the ground truth of program status. The key instrumentation points are:

• Job run success/failure rates tracked via Databricks Workflows UI or REST API
• Cluster utilization and auto-termination anomalies — unexpected terminations often signal memory pressure or misconfigured autoscaling
• Lakeflow Spark Declarative Pipelines event logs — specifically quarantine metrics and data quality expectation failures
• Structured Streaming lag metrics for near-real-time pipelines — consumer lag is a leading indicator of downstream SLA breach

As a TPM, you do not need to debug these yourself. You need to ensure your engineering team has a monitoring contract — agreed thresholds, owners, and escalation triggers — before the pipeline goes to production. The absence of a monitoring contract is itself a program risk.

ADF Run Status

Azure Data Factory pipelines are typically the ingestion layer in a medallion architecture. Key monitoring practices:

• Use ADF Monitor with alert rules on pipeline failure — do not rely on manual checks
• Track watermark drift: if the high-watermark timestamp in your control table is not advancing, data freshness is silently degrading
• Distinguish transient failures (network timeouts, throttling) from structural failures (schema drift, source unavailability) — they have different resolution paths and different stakeholder implications

TPM Principle: A pipeline failure that surfaces in a steering committee meeting before it surfaces in your monitoring layer is a program governance failure, not a technical one.

Layer 2: Milestone Tracking for Data Platform Migrations

Medallion Architecture Rollout as Milestone Anchors

A medallion architecture migration — Bronze → Silver → Gold — provides a natural milestone structure that is legible to both engineers and business stakeholders. The key is to define exit criteria for each layer transition, not just completion dates.

This dual-criteria approach prevents the most common milestone failure in data programs: engineering marking a phase complete while business stakeholders have not validated the output.

Unity Catalog Migration Milestones

Unity Catalog migrations carry specific governance complexity. Structure milestones around these control points:

• Metastore provisioning and account-level admin alignment — often blocked by IT governance, not engineering
• Workspace attachment and existing cluster migration — plan for a deprecation window, not a hard cutover
• Data Access Control migration — moving from legacy table ACLs to Unity Catalog privileges requires a privilege audit first. The audit should enumerate all existing GRANT statements at the database, table, and view level; map them to Unity Catalog securable objects (catalog → schema → table); and identify orphaned permissions with no active principal. Budget at least one sprint for this exercise on programs with more than 20 tables and multiple team-level access groups. Skipping it results in either over-permissioned production catalogs or broken access after cutover — both are compliance incidents in regulated environments.
• External location and storage credential setup — validate with the cloud infrastructure team before scheduling migration windows
• Lineage and audit log enablement — confirm with compliance that the System Catalog meets audit requirements

Governance note: Unity Catalog migrations in regulated environments (BFSI, Healthcare) must align metastore boundaries with data residency requirements. This is a TPM dependency item, not an engineering decision.

Layer 3: Dependency Exposure

In large data platform programs, dependencies are the primary source of schedule risk — not technical complexity. The TPM's job is to make dependencies visible before they become blockers.

Dependency Mapping Framework

Categorize dependencies across three dimensions:

Dependency Visibility in the Sprint

In programs involving multiple delivery teams, dependency risk compounds when teams are optimizing for different sprint goals. The TPM must ensure that cross-team dependency work is explicitly ticketed and assigned in the sprint — not just logged in a dependency register. An integration task that exists only in a RAID log has no owner and no deadline. Make it a sprint ticket, or it will not get done.

Layer 4: Stakeholder Reporting Cadences

Two Reports, Two Languages

Leadership does not read engineering dashboards. Engineers do not need executive summaries. The TPM authors two distinct artifacts:

• Weekly Engineering Pulse: pipeline metrics, sprint velocity, open blockers, dependency status — shared in the team channel or stand-up
• Monthly Steering Committee One-Pager: RAG status, milestone trend (on track / at risk / delayed), top 3 risks with mitigation status, business impact summary — presented to sponsors

RAG Status Template for Data Platform Programs

Three rules for RAG status credibility: never go from green to red in one reporting cycle without a prior amber; always include a trend arrow alongside the RAG colour; and always pair a red status with a documented mitigation action and revised date. The Cloud Spend row is not optional — leadership in cloud-native programs increasingly treats DBU consumption vs. value delivered as a primary health signal, not a finance footnote.

Risk Escalation for Pipeline Failures

Classifying Pipeline Failures

Not all pipeline failures are equal. The TPM must help engineering leads apply consistent classification to avoid both under-escalation (hiding problems) and over-escalation (noise fatigue in leadership).

Escalation discipline note: A P1 that the TPM learns about from a business stakeholder — rather than from the engineering team — indicates a broken escalation contract. Establish the escalation chain in program kickoff, not after the first incident.

The RAID Log for Databricks + Azure Programs

A RAID log (Risks, Assumptions, Issues, Dependencies) is the TPM's primary program governance artifact. For data platform programs, the standard RAID template needs calibration to capture data engineering-specific risks accurately.

Common RAID Items in Databricks / Azure Programs

The RAID log is a living document. Review it in every weekly engineering sync. An item that has not been updated in two weeks is either resolved (and should be closed) or forgotten (and is now a hidden risk).

The TPM as Connective Tissue

A data platform program is a complex system. Databricks clusters, ADF pipelines, Delta Lake tables, Unity Catalog policies, business stakeholders, compliance requirements, and delivery teams — all interdependent, all operating at different speeds and speaking different languages.

The TPM does not build the platform. The TPM builds the visibility layer that allows the platform to be built reliably. Without that layer, even the best engineering team will eventually deliver the wrong thing, at the wrong time, with the wrong stakeholders informed.

Program visibility is not administrative overhead. It is a delivery capability — as important as data architecture, and far more often the differentiator between programs that succeed and programs that recover.

If your data platform program is on track and you cannot explain why in three bullet points that a business sponsor would understand, your visibility layer needs work.

Karthik Darbha is a Senior Data Engineering & AI Leader with 23 years of professional experience, including 20+ years building enterprise data platforms across Healthcare, Pharma, Retail, Insurance, and Financial Services. He writes about data engineering, program management, and the intersection of technology and philosophy at tech4nirvana.com.

The Bhagavad Gita Problem in Data Engineering

Arjuna's dilemma on the battlefield of Kurukshetra is, at its core, a crisis of control. He wants to manage outcomes — to know, in advance, exactly what will happen if he acts. Krishna's answer is radical: act without attachment to results. Do what the moment demands, not what your anxiety prescribes.

Data engineers face a structurally similar problem when they reach for ZORDER BY. Every time they write to a large table, they feel compelled to run OPTIMIZE + ZORDER BY — to impose order, manually, comprehensively. It feels responsible. It feels controlled. But it is expensive, often redundant, and operationally unsustainable at scale.

Liquid Clustering is the engineering equivalent of nishkama karma — action without the compulsive need to orchestrate every outcome.

Key idea: Liquid Clustering does not optimize everything. It optimizes what needs optimizing. This is not laziness — it is precision grounded in awareness.

What Is Liquid Clustering? (The Technical Grounding)

Liquid Clustering replaces ZORDER BY with an incremental, stateful optimization that tracks file-level clustering health. Available from Delta 3.x (Databricks Runtime 13.3+), it rewrites only data files that have drifted from the clustering target — not the whole table.

In practical terms: if you have a 10 TB sales_transactions table clustered by rep_id and sale_date, and an hourly ingestion job appends 5 GB of new records, a subsequent OPTIMIZE touches only the new files — not all 10 TB. The already-clustered data is left undisturbed.

Maya and the Illusion of Total Order

Advaita Vedanta teaches that Maya — often translated as illusion — is not the claim that the world does not exist, but that we mistake partial appearances for ultimate reality. We see a table with millions of rows and believe that perfect, complete, always-current physical ordering is both achievable and necessary. This is Maya at the engineering layer.

There is a precise geometric reason why this illusion breaks down. ZORDER BY uses a Z-order curve (also called a Morton curve) — a mathematical technique that maps multi-dimensional data onto a single linear sequence while attempting to preserve locality. It works reasonably well in two or three dimensions. But as you add columns, the curve's locality-preserving property degrades rapidly. Points that are close in multi-dimensional space end up far apart on the linear curve. The ordering becomes increasingly arbitrary. You are not achieving global order — you are achieving the appearance of order, at increasing cost.

Liquid Clustering abandons this pretension entirely. It uses a multidimensional clustering approach that does not attempt to collapse all dimensions into a single line. Instead, it tracks clustering health per file, per column set, and acts only where the data has genuinely drifted. It does not chase a Z-curve that was never fully achievable. This is why the 4-column limit that cripples ZORDER BY does not apply in the same way to Liquid Clustering — the underlying geometry is different.

The truth is that queries do not need perfect global order. They need sufficient local clustering — enough that the query planner can skip irrelevant files. Liquid Clustering understands this. It does not chase an impossible ideal; it tracks the real state of each file and acts only where action is warranted.

Enabling Liquid Clustering (SQL):

CREATE TABLE pharma.silver.sales_transactions
  CLUSTER BY (rep_id, sale_date)
AS SELECT * FROM bronze.sales_transactions_raw;

Enabling via PySpark:

from delta.tables import DeltaTable

DeltaTable.createOrReplace(spark) \
  .tableName('pharma.silver.sales_transactions') \
  .addColumn('rep_id', 'STRING') \
  .addColumn('sale_date', 'DATE') \
  .addColumn('product_code', 'STRING') \
  .addColumn('territory', 'STRING') \
  .addColumn('revenue', 'DOUBLE') \
  .clusterBy('rep_id', 'sale_date') \
  .execute()

Alter an existing table:

ALTER TABLE pharma.silver.sales_transactions
  CLUSTER BY (rep_id, sale_date);

The Witness Consciousness of OPTIMIZE

In Advaita, the concept of Sakshi — the witness — describes a mode of awareness that observes without compulsive intervention. The witness knows what is happening without needing to control every outcome. It acts when necessary; it rests when not.

This is precisely how OPTIMIZE behaves on a Liquid Clustering-enabled table. It inspects file statistics. It evaluates clustering health. It writes only what needs rewriting. It is not passive — it is precisely calibrated.

-- OPTIMIZE as Sakshi: acts only where action is needed
OPTIMIZE pharma.silver.sales_transactions;

-- Verify the witness's state
DESCRIBE DETAIL pharma.silver.sales_transactions;

Do not mix OPTIMIZE + ZORDER BY on a table with CLUSTER BY. ZORDER BY overrides the clustering key temporarily — the Sakshi becomes confused. Trust the system.

Choosing Cluster Keys: Neti Neti in Practice

Shankara's method of Neti Neti — "not this, not this" — is a process of elimination that reveals truth by discarding what does not qualify. Choosing cluster keys works the same way.

• Not this: low-cardinality columns like region or therapeutic_area — skip files at too coarse a granularity
• Not this: columns that never appear in WHERE clauses — irrelevant to query planning
• Not this: more than 3–4 columns — clustering effectiveness degrades with dimensionality
• This: high-cardinality identifiers — rep_id, product_code — used in point lookups
• This: date/time columns — sale_date, order_date — used in range scans

Pharma example — sales transactions table:

-- Queries almost always filter by rep_id and sale_date range
CREATE TABLE pharma.silver.sales_transactions
  CLUSTER BY (rep_id, sale_date)
AS SELECT * FROM bronze.sales_transactions_raw;

Turiya: The State Beneath All States

Vedanta describes four states of consciousness: waking, dreaming, deep sleep — and Turiya, the fourth, which is not a state at all but the ground from which the other three arise. It is always present, unchanged, whether you are awake or asleep.

Delta Lake's transaction log is the Turiya of your data platform. Every OPTIMIZE, every write, every schema change is recorded there as a discrete action. But the log itself does not change — it only grows. It witnesses all transformations without being transformed. Liquid Clustering, at its foundation, relies on this immutable log to know what has changed and what has not.

Crucially, the witness does not merely observe — it records measurable state. Each data file's entry in the _delta_log JSON contains a stats column that captures per-column min/max values and null counts for the first 32 columns by default. This is the metadata the query planner uses for file skipping — the ability to eliminate entire Parquet files from a scan without reading them. When your cluster keys are well-chosen and Liquid Clustering is healthy, the engine consults these statistics and skips the irrelevant. The Sakshi has already noted what is present and what is not.

-- Inspect raw file statistics in the transaction log
SELECT
  add.path,
  add.stats
FROM json.`abfss://silver@<storage>.dfs.core.windows.net/sales_transactions/_delta_log/*.json`
WHERE add IS NOT NULL
LIMIT 10;

When you understand your data platform this way — as a system with a permanent witness and measurable, transient transformations — you stop being anxious about every write. The log holds truth. OPTIMIZE acts on it. The table reflects reality at any given point in time.

The Engineer Who Does Not Over-Optimize

The Bhagavad Gita's message to Arjuna was not "do nothing." It was "act from clarity, not from fear." ZORDER BY on a large, frequently-written table is often an act of fear — the anxiety that the data is not ordered enough, that queries will be slow, that something will go wrong.

Liquid Clustering asks you to trust the system. Define your cluster keys thoughtfully. Run OPTIMIZE. Let the engine determine what needs rewriting. Observe the results. Intervene only when the evidence demands it.

This is not passivity. This is precision. And it scales.

na hi jnanena sadrsam pavitram iha vidyate

There is no purifier in this world equal to knowledge. — Bhagavad Gita 4.38

Glossary

Technical Terms

Cardinality — The number of distinct values in a column. High cardinality means many unique values (e.g. rep_id); low cardinality means few (e.g. region = North/South/East/West). High-cardinality columns make better cluster keys because they allow the query engine to skip more files.

DBU (Databricks Unit) — The unit of processing capacity used to measure and bill Databricks workloads. OPTIMIZE operations consume DBUs, so minimising unnecessary reclustering has a direct cost impact.

Delta Lake — An open-source storage layer that brings ACID transactions, schema enforcement, and time travel to data lakes. It sits on top of cloud object storage (e.g. Azure Data Lake Storage) and underpins all Databricks table operations.

Delta Transaction Log (_delta_log) — An append-only JSON log that records every operation performed on a Delta table — writes, deletes, schema changes, and optimizations. It is the authoritative source of truth for the table's current and historical state.

File Skipping — A query optimization technique where the engine reads per-file statistics (min/max values, null counts) from the transaction log and eliminates data files that cannot contain rows matching the query's filter predicates — without reading those files at all.

Liquid Clustering — A Delta Lake optimization feature (DBR 13.3+) that incrementally reclusters only data files that have drifted from the defined clustering target, replacing the need for full-table ZORDER BY runs.

OPTIMIZE — A Databricks SQL command that compacts small files and, when Liquid Clustering is enabled, reclusters files that have drifted. On a clustered table, it is idempotent and safe to run repeatedly.

Parquet — The columnar file format used by Delta Lake to store data on disk. Columnar storage means the engine can read only the columns required by a query, and file skipping works at the Parquet file granularity.

Sargability — A query property describing whether a filter predicate can be resolved using an index or statistics (Search ARGument ABLE). Cluster keys should be chosen from sargable columns — those that appear in WHERE clauses and benefit from file-level min/max statistics.

Z-order Curve (Morton Curve) — A mathematical space-filling curve that maps multi-dimensional data onto a single linear sequence while attempting to preserve locality. Used internally by ZORDER BY. Its locality-preserving property degrades significantly beyond 3–4 dimensions, which is why ZORDER BY becomes less effective with more columns.

ZORDER BY — A Databricks OPTIMIZE sub-command that physically co-locates related data using a Z-order curve. Requires a full table scan on every run and becomes geometrically less effective as the number of clustering columns increases.

Vedantic Terms

Advaita Vedanta — One of the principal schools of Hindu philosophy, associated with Adi Shankaracharya (8th century CE). Advaita means non-dual — the teaching that Brahman (ultimate reality) alone exists, and that the apparent multiplicity of the world arises through Maya.

Maya — Often translated as illusion, but more precisely: the power by which ultimate reality appears as the phenomenal world of multiplicity. Maya does not mean the world is unreal — it means we mistake the appearance for the ground. In this post, it refers to the mistaken belief that perfect, exhaustive physical ordering of data is both achievable and necessary.

Neti Neti — Sanskrit for "not this, not this." A method of inquiry attributed to the Brihadaranyaka Upanishad and developed by Shankara, in which truth is approached by systematically negating everything that does not qualify — rather than by positive assertion. Used here as a framework for eliminating poor cluster key candidates.

Nishkama Karma — Sanskrit for "desireless action" or "action without attachment to results." A central teaching of the Bhagavad Gita (Chapter 3): act fully and precisely, but without anxiety about controlling every outcome. Used here to describe Liquid Clustering's approach — act on what needs acting on, leave the rest undisturbed.

Sakshi — Sanskrit for "witness." In Advaita, the Sakshi is the pure awareness that observes all mental and physical phenomena without being modified by them. It is always present, always aware, never reactive. Used here to describe OPTIMIZE's role: it inspects file statistics and acts with precision, not compulsion.

Turiya — Sanskrit for "the fourth." The fourth state of consciousness in Advaita, beyond waking (jagrat), dreaming (svapna), and deep sleep (sushupti). Turiya is not itself a state but the unchanging ground of awareness from which the other three arise and into which they dissolve. Used here to describe the Delta transaction log — the immutable ground that witnesses all table transformations without itself being transformed.

Karthik Darbha is a Senior Data Engineering & AI Leader with 23 years of professional experience, including 20+ years building enterprise data platforms across Healthcare, Pharma, Retail, Insurance, and Financial Services.

The Problem of Many Selves

In the Advaita Vedanta tradition, the root cause of all suffering is Avidya (अविद्या) — ignorance. Not ignorance in the ordinary sense of not knowing facts, but a more fundamental confusion: the mistaking of the many for the one.

The individual soul — Jivatman — believes itself to be separate, bounded, and independent. It clings to its uniqueness, defends its boundaries, and experiences the world as a collection of distinct, competing objects. This is the delusion that Advaita seeks to dissolve — not through argument alone, but through direct recognition: there is only one reality, and that reality is Brahman.

I have been a data engineer for over two decades. I have worked in Healthcare, Pharma, Financial Services, and Retail. I have seen data architectures built with enormous care and technical sophistication fail — not because the tools were wrong, but because the underlying philosophy was fragmented.

And I have come to believe that most of these failures are not technical failures at all. They are philosophical ones. They are the failures of a mind that sees separation where there is unity, multiplicity where there is one source.

Advaita Vedanta — the philosophy of non-duality — offers a lens that cuts through this complexity with a clarity I have found nowhere else in the technical literature. This post is my attempt to make that connection explicit.

A note on Sanskrit terms: This article draws on several concepts from Advaita Vedanta. Each term is defined on first use, but for quick reference: Brahman (ultimate reality), Maya (appearance/illusion), Avidya (ignorance), Adhikara (qualification/eligibility), Pratibimba (reflection), Viveka (discriminative wisdom), Neti Neti (not this, not this — iterative negation), Dharma (right action in context), Tat tvam asi (Thou art That — the identity of self and ultimate reality). No prior knowledge of Vedanta is required to follow the technical argument.

I. Brahman and the Lakehouse: The One Source of Truth

The central claim of Advaita Vedanta, articulated most powerfully by Adi Shankaracharya in the eighth century, is deceptively simple: there is only one reality — Brahman. Everything we perceive as separate — the chair, the tree, your thoughts, my words — is a modification of this one undivided ground. The apparent multiplicity of the world is Maya, the appearance of difference superimposed upon unity.

Now consider the defining problem of enterprise data architecture: the proliferation of truth.

Every department maintains its own definition of a customer. Sales counts by active accounts. Finance counts by billing entities. Marketing counts by email subscriptions. The data warehouse has a customers table. The CRM has another. The data lake has three more. Each one is confidently called the source of truth, and none of them agree.

In Vedantic terms, this is precisely the confusion of Maya — mistaking the modifications for the ground, the shadows on the wall for the light itself.

The Lakehouse architecture — and more specifically, the Unity Catalog pattern in Databricks — is, philosophically, an attempt to establish Brahman in the data estate. One catalog. One lineage. One governed source from which all downstream consumption derives. Not many truths dressed up as one, but a single ontological ground from which all analytical perspectives emerge as views.

When I first encountered Unity Catalog, my Vedantic instinct recognised it immediately: this is the architecture of non-duality made operational. The Bronze layer is the unmanifest — raw, unprocessed, the Nirguna Brahman (Brahman without attributes). Silver is the first differentiation, cleansed and conformed. Gold is Saguna Brahman — Brahman with attributes, ready to be perceived and used by the world. The medallion architecture is not just a data pattern. It is a cosmology.

II. Maya and the Schema: Why Data Is Always an Approximation

One of Vedanta's subtler insights is that Maya is not illusion in the sense of falsehood. The world is not false. It is a real appearance — a functional reality that operates perfectly within its own domain, even if it is not the whole story. Your coffee cup is real for the purposes of drinking coffee. But at a deeper level, it is mostly empty space and probabilistic quantum fields.

Data engineers experience this tension every day, though few name it.

Every schema is Maya. Every data model is an approximation — a useful fiction that captures reality adequately while inevitably leaving out angles that will matter to someone, somewhere, at some future point.

A data model is not reality. It is a perspective on reality. Advaita calls this vivartavada — the appearance of transformation. The rope that appears as a snake. The schema that appears as a business.

Schema evolution is not a technical problem — it is an epistemological one. The schema must always change because our understanding of the business deepens. Fighting schema change is fighting the nature of knowledge itself.

Delta Lake's schema evolution features, the MERGE INTO pattern, the CLONE operations — these are not convenience features alone. They are a structural acknowledgment that every model is temporary. They are the data engineer's Neti Neti (नेति नेति) — not this, not this — iteratively approaching truth without ever claiming to have finally captured it.

-- Schema evolution: acknowledging that the model is never final
ALTER TABLE silver.customers ADD COLUMNS (
  preferred_channel STRING,
  lifetime_value_band STRING
);

-- MERGE INTO: reconciling multiple partial truths into one governed table
MERGE INTO silver.customers AS target
USING (
  SELECT customer_id, email, preferred_channel
  FROM bronze.raw_crm_events
  WHERE event_date = current_date()
) AS source
ON target.customer_id = source.customer_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

The Unity Catalog metastore records every one of these changes — not as failures, but as the natural evolution of understanding. The lineage graph in Unity Catalog is, philosophically, a record of Viveka — discriminative wisdom — applied iteratively over time.

III. Adhikara: Why Not Everyone Should See Everything

Advaita Vedanta has a concept that is often misunderstood by those unfamiliar with the tradition: Adhikara (अधिकार) — qualification or eligibility. The deepest teachings of non-duality are not presented indiscriminately to every seeker. There is a gradation — a recognition that different levels of inquiry require different levels of preparation, and that premature exposure to the highest teachings can confuse rather than illuminate.

This is not elitism. It is epistemological honesty.

The data governance challenge in regulated industries — Healthcare, Financial Services, Pharma — is precisely an Adhikara problem. Not because the data is being hidden maliciously, but because different roles have different legitimate needs and different levels of qualification to handle sensitive information responsibly.

The HIPAA-compliant healthcare data platform does not expose raw patient records to every analyst. The financial platform does not expose individual transaction details to every business user. Adhikara — qualification — determines access.

Unity Catalog operationalises Adhikara with surgical precision:

-- Column masking: the data exists, but its form is appropriate to the viewer
CREATE FUNCTION mask_ssn(ssn STRING)
  RETURNS STRING
  RETURN IF(IS_MEMBER('pii-approved-analysts'), ssn, 'XXX-XX-' || RIGHT(ssn, 4));

ALTER TABLE silver.patients ALTER COLUMN ssn
  SET MASK mask_ssn;

-- Row-level security: each user sees only the universe they are qualified to see
CREATE ROW FILTER region_filter ON gold.patient_outcomes
  USING (analyst_region = current_user_region());

-- Fine-grained GRANT: Adhikara encoded as permissions
GRANT SELECT ON TABLE gold.patient_outcomes
  TO `clinical-analytics-team`;

REVOKE SELECT ON TABLE silver.raw_claims
  FROM `business-analysts`;

The beauty of Unity Catalog's approach is that the data is not duplicated for different access levels. The same underlying reality — Brahman, if you will — is presented in forms appropriate to the qualification of each observer. The senior data engineer sees the full raw record. The business analyst sees the governed, masked, aggregated view. The external partner sees only the Delta Shared subset. One reality, multiple valid perspectives, each appropriate to its Adhikara.

IV. Delta Sharing as Vasudhaiva Kutumbakam

The ancient Sanskrit principle Vasudhaiva Kutumbakam (वसुधैव कुटुम्बकम्) — the world is one family — expresses the Advaitic insight that boundaries between self and other are ultimately illusory. At the deepest level, we are all one.

Unity Catalog's Delta Sharing protocol is Vasudhaiva Kutumbakam for the data ecosystem.

Delta Sharing allows you to share live, governed data across:

• Organisational boundaries (share with partners, vendors, customers)

• Cloud boundaries (share from Azure to AWS to GCP)

• Platform boundaries (share with non-Databricks consumers)

No data copying. No replication. No loss of governance. The data remains in one place — governed by one Unity Catalog — but its benefits are shared across the whole family of consumers.

# Delta Sharing: one governed source, shared with the whole family
import delta_sharing

# The recipient needs only a profile file — no platform dependency
client = delta_sharing.SharingClient("config.share")

# Access the shared table — from any platform, any cloud
df = delta_sharing.load_as_pandas(
    "config.share#partner_share.gold.aggregated_outcomes"
)

The philosophical alignment is precise: Delta Sharing does not dissolve the boundaries of governance (the organisations remain distinct, as Jivatmans remain apparently distinct). But it recognises the underlying unity — the shared data reality — and enables participation in that unity without demanding merger. This is exactly the Advaitic position: the apparent multiplicity is real at the vyavaharika (conventional) level, but the underlying unity is the paramarthika (ultimate) truth.

V. The Pratibimba: Reflection Without Separation

One of the most beautiful concepts in Advaita Vedanta is Pratibimba (प्रतिबिम्ब) — the reflection. When Brahman appears as the individual soul, it is like the sun reflected in a pot of water. The reflection is real — it illuminates, it warms, it functions. But it is not separate from the original sun. When the pot is broken (when Avidya is dissolved), the reflection merges back into the original.

Unity Catalog's views and materialised views are Pratibimba — reflections of the underlying data reality.

A Gold table in the serving layer is a reflection of the Silver tables below it, which are reflections of the Bronze tables below them, which are reflections of the source systems at the root. Each layer is a real, functional, useful representation. But none of them is the ultimate truth — they are all expressions of the underlying data reality, governed and unified through the one Catalog.

-- A materialised view is Pratibimba: real, functional, but not the source
CREATE MATERIALIZED VIEW gold.customer_ltv_summary
  COMMENT 'Reflection of silver.transactions and silver.customers'
AS
SELECT
  c.customer_id,
  c.segment,
  SUM(t.transaction_value) AS lifetime_value,
  COUNT(t.transaction_id) AS total_transactions,
  MAX(t.transaction_date) AS last_activity_date
FROM silver.customers c
JOIN silver.transactions t ON c.customer_id = t.customer_id
GROUP BY c.customer_id, c.segment;

Unity Catalog's data lineage automatically tracks these Pratibimba relationships — recording which views depend on which tables, which downstream models derive from which upstream sources. The lineage graph is the map of Pratibimba across the entire data estate.

Implementing the Unified Self: A Practical Migration Path

Recognising the Advaitic truth of Unity Catalog is one thing. Migrating from the world of siloed Hive Metastores to unified governance is another.

Here is a migration framework I have applied in regulated environments:

Phase 1 — Inventory (Sravana: listening)

# Enumerate the current fragmented reality
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

# List all databases in Hive Metastore
databases = spark.sql("SHOW DATABASES").collect()

for db in databases:
    tables = spark.sql(f"SHOW TABLES IN {db.databaseName}").collect()
    print(f"Database: {db.databaseName} | Tables: {len(tables)}")

Phase 2 — Classify (Manana: reflection)

# Classify tables by sensitivity before migrating
# Not all data has the same Adhikara requirements
sensitivity_map = {
    "bronze.raw_patient_events": "PII_HIGH",
    "silver.patient_demographics": "PII_MEDIUM",
    "gold.aggregated_outcomes": "PUBLIC_INTERNAL"
}

Phase 3 — Migrate and Govern (Nididhyasana: realisation)

-- Upgrade to Unity Catalog namespace
CREATE CATALOG IF NOT EXISTS prod_healthcare;
CREATE SCHEMA IF NOT EXISTS prod_healthcare.silver;

-- Migrate with governance from day one
CREATE TABLE prod_healthcare.silver.patients
  LOCATION 'abfss://silver@yourstorage.dfs.core.windows.net/patients'
  AS SELECT * FROM hive_metastore.legacy_db.patients;

-- Apply Adhikara immediately
GRANT SELECT ON TABLE prod_healthcare.silver.patients
  TO `clinical-data-scientists`;

The three phases map directly to the Vedantic path of Sravana (hearing/understanding), Manana (deep reflection), and Nididhyasana (direct realisation). You cannot skip phases. The organisation that tries to govern without first understanding what it has will fail, just as the seeker who claims realisation without genuine enquiry is merely performing wisdom.

A note on migration realism. The framework above is conceptually clean. Real migrations are not. In practice, expect friction at four points:

• Workspace attachment sequencing — A Unity Catalog metastore is regional and account-scoped. Attaching multiple workspaces to the same metastore must be planned carefully; workspaces previously using different Hive Metastores will have namespace collisions that require manual resolution before migration proceeds.

• External location conflicts — Tables created in Hive Metastore with LOCATION pointing to ADLS Gen2 paths need those paths registered as External Locations in Unity Catalog before they can be referenced. Unregistered paths will cause PERMISSION_DENIED errors that are not always immediately obvious in their root cause.

• HMS sync and managed table ownership — Managed tables in the legacy Hive Metastore are owned by the workspace; after migration, Unity Catalog requires explicit ownership assignment at catalog, schema, and table levels. Missing this step leads to silent governance gaps where tables exist but have no effective steward.

• Privilege inheritance gaps — Unity Catalog does not automatically inherit Hive Metastore ACLs. Every permission must be explicitly re-granted. In regulated environments, this is a compliance event, not just a technical step — it should be logged, reviewed, and signed off.

None of these friction points invalidate the framework. But acknowledging them is part of Manana — honest reflection on what the path actually involves, not just what it looks like on a whiteboard.

The Unified Self in Production

When Unity Catalog is implemented with integrity — when the three-level namespace is consistently applied, when Adhikara is encoded at the column level, when Delta Sharing enables Vasudhaiva Kutumbakam with partners — something remarkable happens.

The data estate stops feeling like a collection of separate systems and begins to feel like a single, coherent intelligence. Analysts from different teams can trust each other's data because they share a common governance layer. Engineers spend less time negotiating access and more time building insight. The organisation stops managing multiplicity and starts experiencing unity.

This is not a metaphor. It is a measurable operational outcome.

But the Vedantic framing adds something that the purely technical framing misses: it reminds us why this matters. The fragmentation of data is not just a technical debt problem. It is a reflection of a fragmented organisational mind — a mind that has forgotten its own unity and is experiencing the suffering of Maya.

Unity Catalog does not just solve a technical problem. It is an invitation to a different way of thinking — one where the data estate is understood as a unified whole, where governance is understood as Dharma (right action in context), and where the role of the data engineer is not just to move bytes but to establish clarity where confusion reigns.

Tat tvam asi — Thou art That. The data and the business are not separate. The engineer and the organisation are not separate. The governance layer and the governed data are not separate. When this is truly understood — not as a slogan but as a lived architectural principle — the unified self emerges in production.*

A Note on Tools vs. Principles

This article uses Databricks Unity Catalog as its primary example — deliberately, because it is the most complete implementation of unified data governance available today on a cloud lakehouse platform. But the philosophical principles are not Databricks-specific.

The same Advaitic framework applies to any serious data governance implementation: Apache Atlas for metadata management, AWS Glue Data Catalog for AWS-native estates, Microsoft Purview for Azure-wide governance, or a Data Mesh architecture where federated computational governance replaces centralised control. The specific tool enforces the principle; it does not originate it.

What Unity Catalog offers is a particularly coherent operationalisation of the non-dual ideal — one catalog, one lineage, one governed ground. If your organisation uses a different stack, the question to ask is the same: does your governance layer establish one source of ontological truth aka Single Source of Truth (SSOT), or does it manage the proliferation of many? The answer determines whether your architecture reflects Brahman or perpetuates Maya — regardless of the vendor logo on the dashboard.

Conclusion

Unity Catalog is, technically, a centralised metadata and governance layer for Databricks workspaces. It solves real problems: cross-workspace data sharing, fine-grained access control, lineage tracking, and audit compliance.

But at a deeper level, it is an architectural expression of Advaitic wisdom: the recognition that what appears as many is, at its root, one — and that the role of good architecture, like the role of good philosophy, is to make that unity visible, governable, and available to all who are qualified to receive it.

Build with Unity Catalog. Build with unity.

Karthik Darbha is a Senior Data Engineering & AI Leader with 23 years of professional experience, including 20+ years building enterprise data platforms across Healthcare, Pharma, Retail, Insurance, and Financial Services.

Why This Migration Is Non-Trivial

Earlier, I worked as Product Owner and Data Architect on a SAS to Databricks migration for a Pharma Supply Chain and Manufacturing client. One deliverable stood out: migrating Statistical Process Control (SPC) logic — specifically the 8 SPC Run Rules — from SAS Data Step to PySpark.

SPC is a regulatory obligation in pharmaceutical manufacturing. Run rules operationalize this — they catch statistical signals before a measurement breaches a hard specification limit.

Why 8 points for Rule 2? Rule 2 uses 8 consecutive points on one side of the mean. This reflects a deliberate sensitivity trade-off widely adopted in pharma LIMS/QMS systems — a slightly more sensitive threshold where the cost of a missed shift outweighs the cost of an extra investigation.

The 8 SPC Run Rules

The SAS Paradigm

SAS Data Step processes one row at a time. The RETAIN statement persists values across iterations — making run-counter logic trivial:

/* Rule 2 in SAS — naturally sequential */
data spc_out;
  set process_data;
  retain run_count 0 last_side ' ';

  /* All three cases must be explicit — on-mean points break the run */
  if      value > mean then side = 'A';
  else if value < mean then side = 'B';
  else                      side = 'C';

  if side = 'C' then do;
    run_count = 0;
    last_side = 'C';
  end;
  else if side = last_side then run_count + 1;
  else do;
    run_count = 1;
    last_side = side;
  end;

  /* fires EXACTLY at the 8th point — onset semantics are free */
  rule_2 = (run_count = 8);
run;

Three properties make SAS the natural host:

• Implicit cursor — PDV advances one row at a time

• Persistent state — via RETAIN, free and automatic

• Onset detection — trivially correct; fires when run_count == 8, resets on side-change or on-mean point

Four Spark Challenges

1. No shared cursor

A Spark DataFrame is distributed across many executors — there is no single sequential pass. Solution: Window.partitionBy('batch_id', 'parameter_name').orderBy('measurement_timestamp') guarantees correct ordering within a partition-window. Ensure that batch_id and parameter_name define complete logical boundaries and that chronological ordering is never disrupted at partition edges. Validate with synthetic boundary-crossing test data before deploying to production.

2. No implicit state — and memory pressure

SAS RETAIN has no Spark equivalent. The idiomatic bridge: collect_list() over a Window frame + Higher-Order Functions (aggregate(), forall(), slice()) applied to the resulting ordered array.

One operational constraint is important: collect_list() pulls all values in the window into a single executor's memory. For an SPC batch with millions of sensor readings per batch_id, this can trigger OutOfMemory errors. The mitigation is straightforward — use a bounded window (rowsBetween(-14, 0)) rather than unbounded preceding. Since no SPC rule requires more than 15 contiguous observations, capping the array at 15 elements eliminates the memory risk without any loss of rule accuracy.

For very high-scale deployments, a Pandas UDF approach can achieve significantly better throughput — see Vectorized Approach below.

3. Onset detection is not free

A naive HOF check (exists(), forall()) fires at positions N, N+1, N+2 — re-triggering on the same run. The fix: a named_struct accumulator inside aggregate() that tracks (last_side, run_len), incrementing on continuation and resetting on side-change or on-mean observation. The rule fires only when run_len == exactly N.

4. Cross-batch state loss

In a batch SPC pipeline, if a run of 8 points straddles two incremental loads — 5 points in Job A, 3 points in Job B — the window-based accumulator loses state between jobs and will not fire. For near-real-time or incremental SPC monitoring, the correct pattern is Stateful Structured Streaming with mapGroupsWithState(), which persists run state across micro-batches via a checkpoint. See Cross-Batch Runs below.

PySpark Implementation

Setup

from pyspark.sql import functions as F
from pyspark.sql.window import Window
from pyspark.sql.types import StructType, StructField, DoubleType, IntegerType

# Bounded window — caps array at 15 elements, satisfying all SPC rule windows
# and preventing executor OOM on large batches
w_bounded = Window.partitionBy('batch_id', 'parameter_name') \
                  .orderBy('measurement_timestamp') \
                  .rowsBetween(-14, 0)

vals     = F.collect_list('value').over(w_bounded)
sigma    = F.first('sigma').over(w_bounded)
mean_val = F.first('mean_val').over(w_bounded)

# Normalised z-score array
z_arr = F.transform(vals, lambda x: (x - mean_val) / sigma)

Handling missing values: SPC rules are sensitive to gaps in measurement_timestamp. A gap in the time series should reset run counters — the run has been interrupted. Pre-process the DataFrame to detect gaps before windowing:

w_order = Window.partitionBy('batch_id', 'parameter_name') \
                .orderBy('measurement_timestamp')

df_gap_aware = df.withColumn(
    'prev_timestamp',
    F.lag('measurement_timestamp').over(w_order)
).withColumn(
    'gap_exceeded',
    (F.unix_timestamp('measurement_timestamp') -
     F.unix_timestamp('prev_timestamp')) > 300  # 5-minute threshold — adjust per SOP
)

# Partition on gap boundaries so each continuous segment is windowed independently
# Implementation depends on your SOP — filter, flag, or introduce a segment_id column

Document the gap threshold in your data dictionary. This is a business rule, not a Spark concern.

Rule 1 — Single point beyond 3σ

rule1 = F.abs(z_arr.getItem(F.size(z_arr) - 1)) > 3.0

Rule 2 — Onset-correct 8-point run

Values exactly at the mean (z = 0) break the run — the same behaviour as the SAS side = 'C' case. The accumulator resets to 0 on an on-mean observation, resets to 1 on a side change, and increments on continuation. The rule fires at exactly the 8th consecutive point — not before, not after.

def side_of(z):
    """Return 1 (above mean), -1 (below mean), or 0 (on mean)."""
    return F.when(z > 0, F.lit(1)) \
            .when(z < 0, F.lit(-1)) \
            .otherwise(F.lit(0))

rule2 = F.aggregate(
    z_arr,
    F.named_struct(
        F.lit('last_side'), F.lit(0),
        F.lit('run_len'),   F.lit(0)
    ),
    lambda acc, z: F.named_struct(
        F.lit('last_side'), side_of(z),
        F.lit('run_len'),
        F.when(
            (side_of(z) != F.lit(0)) & (side_of(z) == acc['last_side']),
            acc['run_len'] + 1          # Continue run on same side
        ).when(
            side_of(z) != F.lit(0),     # Side change
            F.lit(1)                    # Start new run at 1
        ).otherwise(
            F.lit(0)                    # On-mean: reset
        )
    ),
    lambda acc: acc['run_len'] == 8     # Fire at exactly the 8th point
)

Validation contract:

• 7 points above mean → rule2 = False

• 8 points above mean → rule2 = True at position 8 only

• 9 points above mean → rule2 = False (no re-fire)

• 8 above, 1 on-mean, 8 above → fires at positions 8 and 17 (two separate runs)

Rule 3 — 6 consecutive trending points (onset detection)

The accumulator tracks {prev, dir, run}. Direction is computed once per step from the current and previous z-score, stored in dir, and compared against acc['dir'] in the next step. A flat step (equal consecutive values) breaks the trend and resets the streak.

# Rule 3: 6 consecutive points strictly increasing or decreasing
# Accumulator fields:
#   prev — previous z-score (null sentinel on first observation)
#   dir  — +1 increasing, -1 decreasing, 0 flat or first step
#   run  — current streak length

def _current_dir(z, prev):
    """Direction from prev to z. Null prev → no direction."""
    return (
        F.when(prev.isNull(), F.lit(0))
         .when(z > prev,      F.lit(1))
         .when(z < prev,      F.lit(-1))
         .otherwise(F.lit(0))
    )

rule3 = F.aggregate(
    z_arr,
    F.named_struct(
        F.lit('prev'), F.lit(None).cast('double'),
        F.lit('dir'),  F.lit(0),
        F.lit('run'),  F.lit(0)
    ),
    lambda acc, z: F.named_struct(
        F.lit('prev'), z,
        F.lit('dir'),  _current_dir(z, acc['prev']),
        F.lit('run'),
        F.when(
            acc['prev'].isNotNull() &
            (_current_dir(z, acc['prev']) == acc['dir']) &
            (acc['dir'] != F.lit(0)),
            acc['run'] + 1              # Continue trend
        ).otherwise(
            F.lit(1)                    # New direction, flat, or first step
        )
    ),
    lambda acc: (acc['run'] == 6) & (acc['dir'] != 0)  # Fire at exactly 6th point
)

Rule 4 — 14 alternating points

For each inner point in the last 14 observations, the product of the left and right deltas must be negative — confirming a direction reversal at every step.

# Rule 4: 14 consecutive points alternating up/down
# last14 indices: 0 .. 13
# Inner loop: i = 1 .. 12, accessing i-1 (0..11), i (1..12), i+1 (2..13) — all safe

last14 = F.slice(z_arr, F.size(z_arr) - 13, 14)

rule4 = (F.size(z_arr) >= 14) & F.aggregate(
    F.sequence(F.lit(1), F.lit(12)),    # 12 inner points; accesses indices 0..13
    F.lit(True),
    lambda acc, i: acc & (
        ((last14.getItem(i)     - last14.getItem(i - 1)) *
         (last14.getItem(i + 1) - last14.getItem(i)))     < 0
    )
)

Rules 5, 6, 7, 8 — Window patterns

# Rule 5: 2 of 3 consecutive beyond ±2σ, same side
last3  = F.slice(z_arr, F.size(z_arr) - 2, 3)
above2 = F.aggregate(last3, F.lit(0), lambda a, z: a + F.when(z >  2.0, F.lit(1)).otherwise(F.lit(0)))
below2 = F.aggregate(last3, F.lit(0), lambda a, z: a + F.when(z < -2.0, F.lit(1)).otherwise(F.lit(0)))
rule5  = (F.size(z_arr) >= 3) & ((above2 >= 2) | (below2 >= 2))

# Rule 6: 4 of 5 consecutive beyond ±1σ, same side
last5  = F.slice(z_arr, F.size(z_arr) - 4, 5)
above1 = F.aggregate(last5, F.lit(0), lambda a, z: a + F.when(z >  1.0, F.lit(1)).otherwise(F.lit(0)))
below1 = F.aggregate(last5, F.lit(0), lambda a, z: a + F.when(z < -1.0, F.lit(1)).otherwise(F.lit(0)))
rule6  = (F.size(z_arr) >= 5) & ((above1 >= 4) | (below1 >= 4))

# Rule 7: 15 consecutive within ±1σ (stratification)
last15 = F.slice(z_arr, F.size(z_arr) - 14, 15)
rule7  = (F.size(z_arr) >= 15) & F.forall(last15, lambda z: F.abs(z) <= 1.0)

# Rule 8: 8 consecutive beyond ±1σ, either side (mixture)
last8  = F.slice(z_arr, F.size(z_arr) - 7, 8)
rule8  = (F.size(z_arr) >= 8) & F.forall(last8, lambda z: F.abs(z) > 1.0)

Mutual exclusivity — priority waterfall

Each observation receives exactly one rule label. Priority: R1 → R2 → R5 → R6 → R3 → R4 → R7 → R8 — severity-first, consistent with pharma QMS convention.

df_out = df.withColumn('spc_rule',
    F.when(rule1, F.lit('R1_3sigma'))
     .when(rule2, F.lit('R2_run8_same_side'))
     .when(rule5, F.lit('R5_2of3_beyond_2sigma'))
     .when(rule6, F.lit('R6_4of5_beyond_1sigma'))
     .when(rule3, F.lit('R3_trend6'))
     .when(rule4, F.lit('R4_alternating14'))
     .when(rule7, F.lit('R7_stratification15'))
     .when(rule8, F.lit('R8_mixture8'))
     .otherwise(F.lit('NO_SIGNAL'))
).withColumn('rule_fired_at_timestamp', F.current_timestamp())

Vectorized Approach (Pandas UDF)

For high-scale deployments — millions of sensor readings per batch — Pandas UDFs execute NumPy operations in vectorized batches on the executor, avoiding JVM serialization overhead. This can be significantly faster than nested Spark SQL HOFs for complex multi-rule logic.

When to prefer Pandas UDFs:

• Processing 1M+ rows per batch_id

• Rules require NumPy/SciPy (e.g., EWMA, rolling z-test)

• Multiple rules computed in a single pass

import pandas as pd
import numpy as np
from pyspark.sql.functions import pandas_udf
from pyspark.sql.types import StructType, StructField, BooleanType

@pandas_udf(
    StructType([
        StructField('rule2_fired', BooleanType(), True),
        StructField('rule3_fired', BooleanType(), True),
    ])
)
def spc_rules_vectorized(z_values: pd.Series) -> pd.DataFrame:
    """Compute Rule 2 and Rule 3 for an ordered window of z-scores."""
    z = z_values.values  # NumPy array

    # Rule 2: 8 consecutive same side (onset detection)
    rule2_result = np.zeros(len(z), dtype=bool)
    run_len, last_side = 0, 0
    for i, zi in enumerate(z):
        side = 1 if zi > 0 else (-1 if zi < 0 else 0)
        if side == 0:
            run_len, last_side = 0, 0
        elif side == last_side:
            run_len += 1
        else:
            run_len, last_side = 1, side
        rule2_result[i] = (run_len == 8)

    # Rule 3: 6 consecutive trending (onset detection)
    rule3_result = np.zeros(len(z), dtype=bool)
    run_len, last_dir, prev_z = 0, 0, None
    for i, zi in enumerate(z):
        if prev_z is None:
            prev_z = zi
            continue
        cur_dir = 1 if zi > prev_z else (-1 if zi < prev_z else 0)
        if cur_dir == 0:
            run_len, last_dir = 0, 0
        elif cur_dir == last_dir:
            run_len += 1
        else:
            run_len, last_dir = 1, cur_dir
        rule3_result[i] = (run_len == 6) and (last_dir != 0)
        prev_z = zi

    return pd.DataFrame({'rule2_fired': rule2_result, 'rule3_fired': rule3_result})

Cross-Batch Runs (Stateful Streaming)

Production pharma SPC systems often operate in near-real-time: measurements arrive as they are taken, not in daily batch files. If a run of 8 points straddles two micro-batches — 5 points in the first, 3 in the second — the window-based accumulator loses state at the batch boundary and will not fire the rule.

The correct pattern is mapGroupsWithState() in Spark Structured Streaming, which persists (last_side, run_len) in a managed state store across micro-batches.

from dataclasses import dataclass
from typing import Iterator
from pyspark.sql.streaming import GroupState, GroupStateTimeout
from pyspark.sql import Row

@dataclass
class SpcRunState:
    last_side: int = 0
    run_len:   int = 0

def update_spc_state(
    group_key: tuple,           # (batch_id, parameter_name)
    measurements: Iterator[Row],
    state: GroupState
) -> Iterator[Row]:
    """
    Stateful SPC Rule 2 processor.
    Preserves run state across Spark micro-batches.
    """
    current = state.get if state.exists else SpcRunState()

    results = []
    for row in sorted(measurements, key=lambda r: r.measurement_timestamp):
        z    = row.z_score
        side = 1 if z > 0 else (-1 if z < 0 else 0)

        if side == 0:
            current.run_len, current.last_side = 0, 0
            rule2_fired = False
        elif side == current.last_side:
            current.run_len += 1
            rule2_fired = (current.run_len == 8)
        else:
            current.run_len, current.last_side = 1, side
            rule2_fired = False

        results.append({**row.asDict(), 'rule2_fired': rule2_fired})

    state.update(current)
    yield from results

# Apply:
df_stream = (
    spark.readStream
    .schema(input_schema)
    .load('/mnt/bronze/sensor_stream')
    .withColumn('z_score', (F.col('value') - F.col('mean')) / F.col('sigma'))
    .groupby('batch_id', 'parameter_name')
    .applyInPandasWithState(
        update_spc_state,
        output_schema,
        state_schema,
        'append',
        GroupStateTimeout.ProcessingTimeTimeout()
    )
)

df_stream.writeStream \
    .option('checkpointLocation', '/checkpoints/spc_stream') \
    .option('mergeSchema', 'true') \
    .toTable('gold_spc_alerts')
    .start()

Unit Testing Edge Runs

Before deploying to a validated pharma environment, test the accumulator logic explicitly against boundary conditions. These tests should be part of your CI/CD pipeline.

def test_rule2_fires_at_exactly_8():
    """No early fire, no re-fire after position 8."""
    z = [0.5] * 9
    result = compute_rule2(z)
    assert result[6] == False, "Must not fire at position 7"
    assert result[7] == True,  "Must fire at position 8"
    assert result[8] == False, "Must not re-fire at position 9"

def test_rule2_on_mean_resets():
    """On-mean point resets the run; second run fires independently."""
    z = [0.5] * 8 + [0.0] + [0.5] * 8
    result = compute_rule2(z)
    assert result[7]  == True,  "First run fires at position 8"
    assert result[16] == True,  "Second run fires at position 17"

def test_rule3_flat_breaks_trend():
    """A flat (equal) step resets the trend counter."""
    z = [0.5, 0.6, 0.7, 0.8, 0.9, 1.0,   # 5 increasing steps
         1.0,                               # flat — breaks trend
         1.1, 1.2, 1.3, 1.4, 1.5, 1.6]    # new run of 6
    result = compute_rule3(z)
    assert result[5]  == False, "Broken trend must not fire"
    assert result[12] == True,  "New run of 6 must fire"

def test_rule4_alternating_14():
    """14 alternating points fires Rule 4."""
    z = [0.5, -0.5, 0.6, -0.6, 0.7, -0.7, 0.8, -0.8,
         0.9, -0.9, 1.0, -1.0, 1.1, -1.1]
    result = compute_rule4(z)
    assert result[13] == True, "Rule 4 must fire at 14th alternating point"

def test_rule5_minimum_window():
    """Rule 5 requires at least 3 points."""
    z = [2.5]
    result = compute_rule5(z)
    assert result[0] == False, "Rule 5 must not fire on 1-point window"

def test_cross_batch_rule2():
    """Run spanning two batches fires at overall position 8 via stateful processor."""
    state_after_a = process_batch_stateful([0.5] * 5)   # 5 above in Batch A
    result_b      = process_batch_stateful([0.5] * 3, initial_state=state_after_a)
    assert result_b[2] == True, "Rule 2 must fire at the 3rd point of Batch B (8th overall)"

SAS vs PySpark

Medallion Architecture Placement

Unity Catalog governance: Catalog pharma_manufacturing → schemas raw, curated, analytics. Delta Lake time-travel supports regulatory investigation. ZORDER on (batch_id, parameter_name, measurement_timestamp) eliminates file scanning on QMS query patterns.

Production Design Considerations

Schema enforcement

Do not hard-code sigma and mean_val as literals. Use broadcast variables for small control limits tables or joins for dynamically computed limits:

# Broadcast: for small, infrequently changing control limits
control_limits_bc = spark.sparkContext.broadcast(
    control_limits_df.collect()
)

# Join: for dynamically computed batch-level statistics
df_with_stats = df.join(
    control_stats,
    on=['batch_id', 'parameter_name'],
    how='left'
)

Sigma estimator

Document which estimator was used to compute sigma — moving range, sample SD, pooled, or UWMA. This is a regulatory and statistical decision, not a Spark concern. Example using moving range (the SAS default for individuals charts):

w_order = Window.partitionBy('batch_id', 'parameter_name') \
                .orderBy('measurement_timestamp')

df_sigma = df.withColumn(
    'moving_range',
    F.abs(F.col('value') - F.lag('value').over(w_order))
).withColumn(
    'sigma_estimate',
    F.avg('moving_range').over(w_order) / 1.128  # d2 constant for n=2
)

Partition alignment

Repartition on (batch_id, parameter_name) before writing Silver and apply ZORDER on timestamp to ensure physical file layout aligns with logical query patterns:

df.repartition('batch_id', 'parameter_name') \
    .write.format('delta') \
    .mode('overwrite') \
    .option('zorderBy', 'batch_id,parameter_name,measurement_timestamp') \
    .saveAsTable('silver_spc_zscores')

Alert de-duplication

In operational deployments, the same rule can fire on consecutive rows within a run. Apply a de-duplication layer before writing to your LIMS alerting table:

df_alerts = (
    df_out
    .filter(F.col('spc_rule') != 'NO_SIGNAL')
    .withColumn('alert_id', F.md5(
        F.concat_ws('|',
            F.col('batch_id'),
            F.col('parameter_name'),
            F.col('spc_rule'),
            F.date_trunc('hour', F.col('rule_fired_at_timestamp'))
        )
    ))
    .dropDuplicates(['alert_id'])
)

The Witness and the Process Stream

Patanjali's Yoga Sutras II.17 identifies the root of suffering as the association between the Seer (Drashtri) and the Seen (Drishya). Advaita Vedanta resolves this through Sakshi — the Witness that observes all phenomena without identification or reaction.

A well-designed SPC monitor is, in a small engineering sense, a Sakshi. The process stream flows — measurements rise, fall, drift, oscillate. The system neither panics nor ignores. Rule 1 fires not because the system is alarmed, but because it has accurately perceived what is. The onset-correct accumulator fires exactly once, at the right moment, on the right signal, without noise.

The transition from SAS to Spark mirrors a deeper shift: from implicit state (RETAIN) to explicit state (aggregate(), mapGroupsWithState()). That explicitness demands clarity — you must name your assumptions, test your edge cases, and document your intentions. That rigour is a form of witness-consciousness in code.

Reliable observation requires correct architecture. Turīya — the fourth state of the Mandukya Upanishad — witnesses the three states (waking, dream, deep sleep) without being any of them. The Gold layer, sitting above Bronze and Silver, witnesses the process without being the process.

— tech4nirvana.com

References

• FDA 21 CFR Part 211 — Current Good Manufacturing Practice for Finished Pharmaceuticals.

• ICH Q10: Pharmaceutical Quality System. 2008.

• ICH Q14: Analytical Procedure Development. 2023.

• Apache Spark Documentation — Window Functions and Higher-Order Functions.

• Databricks Documentation — Delta Lake Time Travel, Unity Catalog, Structured Streaming, mapGroupsWithState.

• Apache Spark — Pandas UDFs with Arrow: https://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html

• Patanjali. Yoga Sutras II.17: Drashtri-drishyayoh samyogo heya-hetuh.

• Shankaracharya. Mandukya Upanishad Bhashya — on Turīya as the fourth (witnessing) state.

There is an ancient Sanskrit phrase that has guided seekers of truth for over a thousand years: "Tat Tvam Asi" — You are That. At its core, Advaita Vedanta, the non-dualist school of Indian philosophy codified by Adi Shankaracharya in the 8th century, teaches that the apparent multiplicity of the world is an illusion. Beneath all diversity lies a single, undivided reality — Brahman.

As a data engineer who has spent over two decades building pipelines, architecting data platforms, and debugging production failures at 2am, I have come to realize something quietly profound: the principles of Advaita Vedanta map onto the challenges of modern data engineering with remarkable precision.

This is not mysticism. This is epistemology — the study of how we know what we know. And data engineering, at its heart, is an epistemological discipline.

Maya: The Illusion of Raw Data

In Advaita Vedanta, Maya (माया) refers to the cosmic illusion — the tendency of the mind to mistake the appearance of things for their ultimate reality. The world we perceive through our senses is real in a practical sense, but it conceals a deeper truth.

In data engineering, raw data is Maya.

A source system presents you with a table of transactions. It looks real. It looks complete. But dig deeper and you find:

• Duplicate records from retry logic

• NULL values where business rules demand non-null

• Timestamps in five different formats across three source systems

• Currency values without denomination codes

• Customer IDs that changed silently after a system migration

The raw data is not lying to you — it is simply presenting its surface reality. The data engineer's job is to pierce the veil of Maya, to look past the apparent truth of the source and ask: what is the actual business reality this data represents?

The Medallion Architecture — Bronze, Silver, Gold — is, in this sense, a structured practice of moving from Maya toward truth. Bronze is raw reality as it arrives. Silver is cleansed, conformed reality. Gold is the curated truth the business actually needs.

Viveka: The Practice of Discrimination

Viveka (विवेक) is one of the four qualifications (Sadhana Chatushtaya) that Shankara prescribed for a serious student of Vedanta. It means discrimination — the ability to distinguish the real from the unreal, the permanent from the impermanent, the essential from the incidental.

In data engineering, Viveka is your data quality framework.

Every day, a data engineer exercises Viveka:

• Is this NULL a missing value or a legitimate unknown?

• Is this spike in the metric a real business event or a pipeline anomaly?

• Is this schema change backward compatible or breaking?

• Should this logic live in the transformation layer or the serving layer?

Without Viveka, data pipelines become swamps of technical debt. Every table gets every column. Every pipeline carries every edge case. The system grows heavy with the unreal mistaken for the real.

The practice of Viveka in data engineering means building systems that know what they are for, and refusing to carry what they are not.

Neti Neti: The Power of Elimination

One of the most powerful methods in Advaita Vedanta is Neti Neti (नेति नेति) — Not this, not this. Rather than trying to define Brahman positively, the seeker systematically eliminates everything that Brahman is not. What remains, when all the unreal has been stripped away, is the truth.

In data engineering, Neti Neti is your schema design and debugging philosophy.

When designing a dimensional model, you ask:

• Is this a fact? Neti — it changes too slowly.

• Is this a dimension? Neti — it has no independent existence without a transaction.

• Is this a measure? Neti — it cannot be aggregated meaningfully.

When debugging a pipeline failure:

• Is it the source system? Neti — the raw data looks clean.

• Is it the transformation logic? Neti — unit tests pass.

• Is it the infrastructure? Iti — yes, the Spark executor ran out of memory due to data skew.

The senior data engineer is not the one who immediately knows the answer. The senior data engineer is the one who knows how to eliminate systematically until the truth reveals itself.

Brahman: The Single Source of Truth

In Advaita Vedanta, Brahman (ब्रह्मन्) is the ultimate reality — the single, undivided, infinite consciousness that underlies all apparent multiplicity. Everything that exists is, in its deepest nature, Brahman.

In data engineering, Brahman is your Single Source of Truth.

Every enterprise data platform is, in a sense, a temple to Brahman. The goal is to create one authoritative, trusted, governed representation of business reality — whether that is:

• A unified customer identity across CRM, billing, and support systems

• A canonical product hierarchy reconciled across ERP and e-commerce

• A single financial ledger that the CFO, auditors, and analysts all agree on

The tragedy of most data platforms is that they multiply Atman instead of realizing Brahman. Every team builds its own mart. Every analyst has their own definition of "active customer." Every dashboard shows a slightly different revenue number.

Unity Catalog in Databricks, data contracts, semantic layers — these are not just technical tools. They are institutional practices of non-duality. They assert: there is one truth, and we will govern access to it, not multiply it.

Upadesha Saram: The Essence of the Teaching

Ramana Maharshi's Upadesha Saram (उपदेश सारम्) distills the entirety of Vedantic practice into 30 verses. Its central teaching is self-inquiry: rather than seeking truth outside, turn attention inward and ask "Who am I?"

For a data engineer, self-inquiry means questioning your own assumptions before building anything:

• Why does this data exist?

• Who will use this output and how?

• What breaks if this is wrong?

• Am I solving the real problem or the stated problem?

The greatest data engineering failures I have witnessed in 22 years were not technical failures. They were failures of inquiry — teams that built what was asked without asking why, that optimized pipelines for throughput without asking whether the data was trusted.

Self-inquiry in data engineering is not navel-gazing. It is the highest form of rigor.

Conclusion: The Engineer as Seeker

Advaita Vedanta does not ask you to abandon the world. It asks you to engage with the world with clarity — to act effectively in the empirical realm while remaining anchored in the understanding of ultimate truth.

This is precisely what great data engineering demands.

Build your pipelines. Design your schemas. Optimize your Spark jobs. But do all of this with Viveka. Pierce the Maya of raw data. Apply Neti Neti to eliminate the inessential. And always pursue Brahman — the single, unified, trusted truth your organization can build decisions upon.

The data platform is not just infrastructure. It is, in its highest aspiration, an instrument of clarity.

Tat Tvam Asi. That is what the data, in its deepest truth, is trying to say.

Karthik Darbha is a Data Engineering & AI Leader with over 22 years of experience in Healthcare, Pharma, Retail, Insurance, and Financial Services. He writes at tech4nirvana.com, exploring the intersection of data architecture and timeless wisdom.