The Bhagavad Gita Problem in Data Engineering

Arjuna's dilemma on the battlefield of Kurukshetra is, at its core, a crisis of control. He wants to manage outcomes — to know, in advance, exactly what will happen if he acts. Krishna's answer is radical: act without attachment to results. Do what the moment demands, not what your anxiety prescribes.

Data engineers face a structurally similar problem when they reach for ZORDER BY. Every time they write to a large table, they feel compelled to run OPTIMIZE + ZORDER BY — to impose order, manually, comprehensively. It feels responsible. It feels controlled. But it is expensive, often redundant, and operationally unsustainable at scale.

Liquid Clustering is the engineering equivalent of nishkama karma — action without the compulsive need to orchestrate every outcome.

Key idea: Liquid Clustering does not optimize everything. It optimizes what needs optimizing. This is not laziness — it is precision grounded in awareness.

What Is Liquid Clustering? (The Technical Grounding)

Liquid Clustering replaces ZORDER BY with an incremental, stateful optimization that tracks file-level clustering health. Available from Delta 3.x (Databricks Runtime 13.3+), it rewrites only data files that have drifted from the clustering target — not the whole table.

In practical terms: if you have a 10 TB sales_transactions table clustered by rep_id and sale_date, and an hourly ingestion job appends 5 GB of new records, a subsequent OPTIMIZE touches only the new files — not all 10 TB. The already-clustered data is left undisturbed.

Maya and the Illusion of Total Order

Advaita Vedanta teaches that Maya — often translated as illusion — is not the claim that the world does not exist, but that we mistake partial appearances for ultimate reality. We see a table with millions of rows and believe that perfect, complete, always-current physical ordering is both achievable and necessary. This is Maya at the engineering layer.

There is a precise geometric reason why this illusion breaks down. ZORDER BY uses a Z-order curve (also called a Morton curve) — a mathematical technique that maps multi-dimensional data onto a single linear sequence while attempting to preserve locality. It works reasonably well in two or three dimensions. But as you add columns, the curve's locality-preserving property degrades rapidly. Points that are close in multi-dimensional space end up far apart on the linear curve. The ordering becomes increasingly arbitrary. You are not achieving global order — you are achieving the appearance of order, at increasing cost.

Liquid Clustering abandons this pretension entirely. It uses a multidimensional clustering approach that does not attempt to collapse all dimensions into a single line. Instead, it tracks clustering health per file, per column set, and acts only where the data has genuinely drifted. It does not chase a Z-curve that was never fully achievable. This is why the 4-column limit that cripples ZORDER BY does not apply in the same way to Liquid Clustering — the underlying geometry is different.

The truth is that queries do not need perfect global order. They need sufficient local clustering — enough that the query planner can skip irrelevant files. Liquid Clustering understands this. It does not chase an impossible ideal; it tracks the real state of each file and acts only where action is warranted.

Enabling Liquid Clustering (SQL):

CREATE TABLE pharma.silver.sales_transactions
  CLUSTER BY (rep_id, sale_date)
AS SELECT * FROM bronze.sales_transactions_raw;

Enabling via PySpark:

from delta.tables import DeltaTable

DeltaTable.createOrReplace(spark) \
  .tableName('pharma.silver.sales_transactions') \
  .addColumn('rep_id', 'STRING') \
  .addColumn('sale_date', 'DATE') \
  .addColumn('product_code', 'STRING') \
  .addColumn('territory', 'STRING') \
  .addColumn('revenue', 'DOUBLE') \
  .clusterBy('rep_id', 'sale_date') \
  .execute()

Alter an existing table:

ALTER TABLE pharma.silver.sales_transactions
  CLUSTER BY (rep_id, sale_date);

The Witness Consciousness of OPTIMIZE

In Advaita, the concept of Sakshi — the witness — describes a mode of awareness that observes without compulsive intervention. The witness knows what is happening without needing to control every outcome. It acts when necessary; it rests when not.

This is precisely how OPTIMIZE behaves on a Liquid Clustering-enabled table. It inspects file statistics. It evaluates clustering health. It writes only what needs rewriting. It is not passive — it is precisely calibrated.

-- OPTIMIZE as Sakshi: acts only where action is needed
OPTIMIZE pharma.silver.sales_transactions;

-- Verify the witness's state
DESCRIBE DETAIL pharma.silver.sales_transactions;

Do not mix OPTIMIZE + ZORDER BY on a table with CLUSTER BY. ZORDER BY overrides the clustering key temporarily — the Sakshi becomes confused. Trust the system.

Choosing Cluster Keys: Neti Neti in Practice

Shankara's method of Neti Neti — "not this, not this" — is a process of elimination that reveals truth by discarding what does not qualify. Choosing cluster keys works the same way.

• Not this: low-cardinality columns like region or therapeutic_area — skip files at too coarse a granularity
• Not this: columns that never appear in WHERE clauses — irrelevant to query planning
• Not this: more than 3–4 columns — clustering effectiveness degrades with dimensionality
• This: high-cardinality identifiers — rep_id, product_code — used in point lookups
• This: date/time columns — sale_date, order_date — used in range scans

Pharma example — sales transactions table:

-- Queries almost always filter by rep_id and sale_date range
CREATE TABLE pharma.silver.sales_transactions
  CLUSTER BY (rep_id, sale_date)
AS SELECT * FROM bronze.sales_transactions_raw;

Turiya: The State Beneath All States

Vedanta describes four states of consciousness: waking, dreaming, deep sleep — and Turiya, the fourth, which is not a state at all but the ground from which the other three arise. It is always present, unchanged, whether you are awake or asleep.

Delta Lake's transaction log is the Turiya of your data platform. Every OPTIMIZE, every write, every schema change is recorded there as a discrete action. But the log itself does not change — it only grows. It witnesses all transformations without being transformed. Liquid Clustering, at its foundation, relies on this immutable log to know what has changed and what has not.

Crucially, the witness does not merely observe — it records measurable state. Each data file's entry in the _delta_log JSON contains a stats column that captures per-column min/max values and null counts for the first 32 columns by default. This is the metadata the query planner uses for file skipping — the ability to eliminate entire Parquet files from a scan without reading them. When your cluster keys are well-chosen and Liquid Clustering is healthy, the engine consults these statistics and skips the irrelevant. The Sakshi has already noted what is present and what is not.

-- Inspect raw file statistics in the transaction log
SELECT
  add.path,
  add.stats
FROM json.`abfss://silver@<storage>.dfs.core.windows.net/sales_transactions/_delta_log/*.json`
WHERE add IS NOT NULL
LIMIT 10;

When you understand your data platform this way — as a system with a permanent witness and measurable, transient transformations — you stop being anxious about every write. The log holds truth. OPTIMIZE acts on it. The table reflects reality at any given point in time.

The Engineer Who Does Not Over-Optimize

The Bhagavad Gita's message to Arjuna was not "do nothing." It was "act from clarity, not from fear." ZORDER BY on a large, frequently-written table is often an act of fear — the anxiety that the data is not ordered enough, that queries will be slow, that something will go wrong.

Liquid Clustering asks you to trust the system. Define your cluster keys thoughtfully. Run OPTIMIZE. Let the engine determine what needs rewriting. Observe the results. Intervene only when the evidence demands it.

This is not passivity. This is precision. And it scales.

na hi jnanena sadrsam pavitram iha vidyate

There is no purifier in this world equal to knowledge. — Bhagavad Gita 4.38

Glossary

Technical Terms

Cardinality — The number of distinct values in a column. High cardinality means many unique values (e.g. rep_id); low cardinality means few (e.g. region = North/South/East/West). High-cardinality columns make better cluster keys because they allow the query engine to skip more files.

DBU (Databricks Unit) — The unit of processing capacity used to measure and bill Databricks workloads. OPTIMIZE operations consume DBUs, so minimising unnecessary reclustering has a direct cost impact.

Delta Lake — An open-source storage layer that brings ACID transactions, schema enforcement, and time travel to data lakes. It sits on top of cloud object storage (e.g. Azure Data Lake Storage) and underpins all Databricks table operations.

Delta Transaction Log (_delta_log) — An append-only JSON log that records every operation performed on a Delta table — writes, deletes, schema changes, and optimizations. It is the authoritative source of truth for the table's current and historical state.

File Skipping — A query optimization technique where the engine reads per-file statistics (min/max values, null counts) from the transaction log and eliminates data files that cannot contain rows matching the query's filter predicates — without reading those files at all.

Liquid Clustering — A Delta Lake optimization feature (DBR 13.3+) that incrementally reclusters only data files that have drifted from the defined clustering target, replacing the need for full-table ZORDER BY runs.

OPTIMIZE — A Databricks SQL command that compacts small files and, when Liquid Clustering is enabled, reclusters files that have drifted. On a clustered table, it is idempotent and safe to run repeatedly.

Parquet — The columnar file format used by Delta Lake to store data on disk. Columnar storage means the engine can read only the columns required by a query, and file skipping works at the Parquet file granularity.

Sargability — A query property describing whether a filter predicate can be resolved using an index or statistics (Search ARGument ABLE). Cluster keys should be chosen from sargable columns — those that appear in WHERE clauses and benefit from file-level min/max statistics.

Z-order Curve (Morton Curve) — A mathematical space-filling curve that maps multi-dimensional data onto a single linear sequence while attempting to preserve locality. Used internally by ZORDER BY. Its locality-preserving property degrades significantly beyond 3–4 dimensions, which is why ZORDER BY becomes less effective with more columns.

ZORDER BY — A Databricks OPTIMIZE sub-command that physically co-locates related data using a Z-order curve. Requires a full table scan on every run and becomes geometrically less effective as the number of clustering columns increases.

Vedantic Terms

Advaita Vedanta — One of the principal schools of Hindu philosophy, associated with Adi Shankaracharya (8th century CE). Advaita means non-dual — the teaching that Brahman (ultimate reality) alone exists, and that the apparent multiplicity of the world arises through Maya.

Maya — Often translated as illusion, but more precisely: the power by which ultimate reality appears as the phenomenal world of multiplicity. Maya does not mean the world is unreal — it means we mistake the appearance for the ground. In this post, it refers to the mistaken belief that perfect, exhaustive physical ordering of data is both achievable and necessary.

Neti Neti — Sanskrit for "not this, not this." A method of inquiry attributed to the Brihadaranyaka Upanishad and developed by Shankara, in which truth is approached by systematically negating everything that does not qualify — rather than by positive assertion. Used here as a framework for eliminating poor cluster key candidates.

Nishkama Karma — Sanskrit for "desireless action" or "action without attachment to results." A central teaching of the Bhagavad Gita (Chapter 3): act fully and precisely, but without anxiety about controlling every outcome. Used here to describe Liquid Clustering's approach — act on what needs acting on, leave the rest undisturbed.

Sakshi — Sanskrit for "witness." In Advaita, the Sakshi is the pure awareness that observes all mental and physical phenomena without being modified by them. It is always present, always aware, never reactive. Used here to describe OPTIMIZE's role: it inspects file statistics and acts with precision, not compulsion.

Turiya — Sanskrit for "the fourth." The fourth state of consciousness in Advaita, beyond waking (jagrat), dreaming (svapna), and deep sleep (sushupti). Turiya is not itself a state but the unchanging ground of awareness from which the other three arise and into which they dissolve. Used here to describe the Delta transaction log — the immutable ground that witnesses all table transformations without itself being transformed.

Karthik Darbha is a Senior Data Engineering & AI Leader with 23 years of professional experience, including 20+ years building enterprise data platforms across Healthcare, Pharma, Retail, Insurance, and Financial Services.

The Problem of Many Selves

In the Advaita Vedanta tradition, the root cause of all suffering is Avidya (अविद्या) — ignorance. Not ignorance in the ordinary sense of not knowing facts, but a more fundamental confusion: the mistaking of the many for the one.

The individual soul — Jivatman — believes itself to be separate, bounded, and independent. It clings to its uniqueness, defends its boundaries, and experiences the world as a collection of distinct, competing objects. This is the delusion that Advaita seeks to dissolve — not through argument alone, but through direct recognition: there is only one reality, and that reality is Brahman.

I have been a data engineer for over two decades. I have worked in Healthcare, Pharma, Financial Services, and Retail. I have seen data architectures built with enormous care and technical sophistication fail — not because the tools were wrong, but because the underlying philosophy was fragmented.

And I have come to believe that most of these failures are not technical failures at all. They are philosophical ones. They are the failures of a mind that sees separation where there is unity, multiplicity where there is one source.

Advaita Vedanta — the philosophy of non-duality — offers a lens that cuts through this complexity with a clarity I have found nowhere else in the technical literature. This post is my attempt to make that connection explicit.

A note on Sanskrit terms: This article draws on several concepts from Advaita Vedanta. Each term is defined on first use, but for quick reference: Brahman (ultimate reality), Maya (appearance/illusion), Avidya (ignorance), Adhikara (qualification/eligibility), Pratibimba (reflection), Viveka (discriminative wisdom), Neti Neti (not this, not this — iterative negation), Dharma (right action in context), Tat tvam asi (Thou art That — the identity of self and ultimate reality). No prior knowledge of Vedanta is required to follow the technical argument.

I. Brahman and the Lakehouse: The One Source of Truth

The central claim of Advaita Vedanta, articulated most powerfully by Adi Shankaracharya in the eighth century, is deceptively simple: there is only one reality — Brahman. Everything we perceive as separate — the chair, the tree, your thoughts, my words — is a modification of this one undivided ground. The apparent multiplicity of the world is Maya, the appearance of difference superimposed upon unity.

Now consider the defining problem of enterprise data architecture: the proliferation of truth.

Every department maintains its own definition of a customer. Sales counts by active accounts. Finance counts by billing entities. Marketing counts by email subscriptions. The data warehouse has a customers table. The CRM has another. The data lake has three more. Each one is confidently called the source of truth, and none of them agree.

In Vedantic terms, this is precisely the confusion of Maya — mistaking the modifications for the ground, the shadows on the wall for the light itself.

The Lakehouse architecture — and more specifically, the Unity Catalog pattern in Databricks — is, philosophically, an attempt to establish Brahman in the data estate. One catalog. One lineage. One governed source from which all downstream consumption derives. Not many truths dressed up as one, but a single ontological ground from which all analytical perspectives emerge as views.

When I first encountered Unity Catalog, my Vedantic instinct recognised it immediately: this is the architecture of non-duality made operational. The Bronze layer is the unmanifest — raw, unprocessed, the Nirguna Brahman (Brahman without attributes). Silver is the first differentiation, cleansed and conformed. Gold is Saguna Brahman — Brahman with attributes, ready to be perceived and used by the world. The medallion architecture is not just a data pattern. It is a cosmology.

II. Maya and the Schema: Why Data Is Always an Approximation

One of Vedanta's subtler insights is that Maya is not illusion in the sense of falsehood. The world is not false. It is a real appearance — a functional reality that operates perfectly within its own domain, even if it is not the whole story. Your coffee cup is real for the purposes of drinking coffee. But at a deeper level, it is mostly empty space and probabilistic quantum fields.

Data engineers experience this tension every day, though few name it.

Every schema is Maya. Every data model is an approximation — a useful fiction that captures reality adequately while inevitably leaving out angles that will matter to someone, somewhere, at some future point.

A data model is not reality. It is a perspective on reality. Advaita calls this vivartavada — the appearance of transformation. The rope that appears as a snake. The schema that appears as a business.

Schema evolution is not a technical problem — it is an epistemological one. The schema must always change because our understanding of the business deepens. Fighting schema change is fighting the nature of knowledge itself.

Delta Lake's schema evolution features, the MERGE INTO pattern, the CLONE operations — these are not convenience features alone. They are a structural acknowledgment that every model is temporary. They are the data engineer's Neti Neti (नेति नेति) — not this, not this — iteratively approaching truth without ever claiming to have finally captured it.

-- Schema evolution: acknowledging that the model is never final
ALTER TABLE silver.customers ADD COLUMNS (
  preferred_channel STRING,
  lifetime_value_band STRING
);

-- MERGE INTO: reconciling multiple partial truths into one governed table
MERGE INTO silver.customers AS target
USING (
  SELECT customer_id, email, preferred_channel
  FROM bronze.raw_crm_events
  WHERE event_date = current_date()
) AS source
ON target.customer_id = source.customer_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

The Unity Catalog metastore records every one of these changes — not as failures, but as the natural evolution of understanding. The lineage graph in Unity Catalog is, philosophically, a record of Viveka — discriminative wisdom — applied iteratively over time.

III. Adhikara: Why Not Everyone Should See Everything

Advaita Vedanta has a concept that is often misunderstood by those unfamiliar with the tradition: Adhikara (अधिकार) — qualification or eligibility. The deepest teachings of non-duality are not presented indiscriminately to every seeker. There is a gradation — a recognition that different levels of inquiry require different levels of preparation, and that premature exposure to the highest teachings can confuse rather than illuminate.

This is not elitism. It is epistemological honesty.

The data governance challenge in regulated industries — Healthcare, Financial Services, Pharma — is precisely an Adhikara problem. Not because the data is being hidden maliciously, but because different roles have different legitimate needs and different levels of qualification to handle sensitive information responsibly.

The HIPAA-compliant healthcare data platform does not expose raw patient records to every analyst. The financial platform does not expose individual transaction details to every business user. Adhikara — qualification — determines access.

Unity Catalog operationalises Adhikara with surgical precision:

-- Column masking: the data exists, but its form is appropriate to the viewer
CREATE FUNCTION mask_ssn(ssn STRING)
  RETURNS STRING
  RETURN IF(IS_MEMBER('pii-approved-analysts'), ssn, 'XXX-XX-' || RIGHT(ssn, 4));

ALTER TABLE silver.patients ALTER COLUMN ssn
  SET MASK mask_ssn;

-- Row-level security: each user sees only the universe they are qualified to see
CREATE ROW FILTER region_filter ON gold.patient_outcomes
  USING (analyst_region = current_user_region());

-- Fine-grained GRANT: Adhikara encoded as permissions
GRANT SELECT ON TABLE gold.patient_outcomes
  TO `clinical-analytics-team`;

REVOKE SELECT ON TABLE silver.raw_claims
  FROM `business-analysts`;

The beauty of Unity Catalog's approach is that the data is not duplicated for different access levels. The same underlying reality — Brahman, if you will — is presented in forms appropriate to the qualification of each observer. The senior data engineer sees the full raw record. The business analyst sees the governed, masked, aggregated view. The external partner sees only the Delta Shared subset. One reality, multiple valid perspectives, each appropriate to its Adhikara.

IV. Delta Sharing as Vasudhaiva Kutumbakam

The ancient Sanskrit principle Vasudhaiva Kutumbakam (वसुधैव कुटुम्बकम्) — the world is one family — expresses the Advaitic insight that boundaries between self and other are ultimately illusory. At the deepest level, we are all one.

Unity Catalog's Delta Sharing protocol is Vasudhaiva Kutumbakam for the data ecosystem.

Delta Sharing allows you to share live, governed data across:

• Organisational boundaries (share with partners, vendors, customers)

• Cloud boundaries (share from Azure to AWS to GCP)

• Platform boundaries (share with non-Databricks consumers)

No data copying. No replication. No loss of governance. The data remains in one place — governed by one Unity Catalog — but its benefits are shared across the whole family of consumers.

# Delta Sharing: one governed source, shared with the whole family
import delta_sharing

# The recipient needs only a profile file — no platform dependency
client = delta_sharing.SharingClient("config.share")

# Access the shared table — from any platform, any cloud
df = delta_sharing.load_as_pandas(
    "config.share#partner_share.gold.aggregated_outcomes"
)

The philosophical alignment is precise: Delta Sharing does not dissolve the boundaries of governance (the organisations remain distinct, as Jivatmans remain apparently distinct). But it recognises the underlying unity — the shared data reality — and enables participation in that unity without demanding merger. This is exactly the Advaitic position: the apparent multiplicity is real at the vyavaharika (conventional) level, but the underlying unity is the paramarthika (ultimate) truth.

V. The Pratibimba: Reflection Without Separation

One of the most beautiful concepts in Advaita Vedanta is Pratibimba (प्रतिबिम्ब) — the reflection. When Brahman appears as the individual soul, it is like the sun reflected in a pot of water. The reflection is real — it illuminates, it warms, it functions. But it is not separate from the original sun. When the pot is broken (when Avidya is dissolved), the reflection merges back into the original.

Unity Catalog's views and materialised views are Pratibimba — reflections of the underlying data reality.

A Gold table in the serving layer is a reflection of the Silver tables below it, which are reflections of the Bronze tables below them, which are reflections of the source systems at the root. Each layer is a real, functional, useful representation. But none of them is the ultimate truth — they are all expressions of the underlying data reality, governed and unified through the one Catalog.

-- A materialised view is Pratibimba: real, functional, but not the source
CREATE MATERIALIZED VIEW gold.customer_ltv_summary
  COMMENT 'Reflection of silver.transactions and silver.customers'
AS
SELECT
  c.customer_id,
  c.segment,
  SUM(t.transaction_value) AS lifetime_value,
  COUNT(t.transaction_id) AS total_transactions,
  MAX(t.transaction_date) AS last_activity_date
FROM silver.customers c
JOIN silver.transactions t ON c.customer_id = t.customer_id
GROUP BY c.customer_id, c.segment;

Unity Catalog's data lineage automatically tracks these Pratibimba relationships — recording which views depend on which tables, which downstream models derive from which upstream sources. The lineage graph is the map of Pratibimba across the entire data estate.

Implementing the Unified Self: A Practical Migration Path

Recognising the Advaitic truth of Unity Catalog is one thing. Migrating from the world of siloed Hive Metastores to unified governance is another.

Here is a migration framework I have applied in regulated environments:

Phase 1 — Inventory (Sravana: listening)

# Enumerate the current fragmented reality
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

# List all databases in Hive Metastore
databases = spark.sql("SHOW DATABASES").collect()

for db in databases:
    tables = spark.sql(f"SHOW TABLES IN {db.databaseName}").collect()
    print(f"Database: {db.databaseName} | Tables: {len(tables)}")

Phase 2 — Classify (Manana: reflection)

# Classify tables by sensitivity before migrating
# Not all data has the same Adhikara requirements
sensitivity_map = {
    "bronze.raw_patient_events": "PII_HIGH",
    "silver.patient_demographics": "PII_MEDIUM",
    "gold.aggregated_outcomes": "PUBLIC_INTERNAL"
}

Phase 3 — Migrate and Govern (Nididhyasana: realisation)

-- Upgrade to Unity Catalog namespace
CREATE CATALOG IF NOT EXISTS prod_healthcare;
CREATE SCHEMA IF NOT EXISTS prod_healthcare.silver;

-- Migrate with governance from day one
CREATE TABLE prod_healthcare.silver.patients
  LOCATION 'abfss://silver@yourstorage.dfs.core.windows.net/patients'
  AS SELECT * FROM hive_metastore.legacy_db.patients;

-- Apply Adhikara immediately
GRANT SELECT ON TABLE prod_healthcare.silver.patients
  TO `clinical-data-scientists`;

The three phases map directly to the Vedantic path of Sravana (hearing/understanding), Manana (deep reflection), and Nididhyasana (direct realisation). You cannot skip phases. The organisation that tries to govern without first understanding what it has will fail, just as the seeker who claims realisation without genuine enquiry is merely performing wisdom.

A note on migration realism. The framework above is conceptually clean. Real migrations are not. In practice, expect friction at four points:

• Workspace attachment sequencing — A Unity Catalog metastore is regional and account-scoped. Attaching multiple workspaces to the same metastore must be planned carefully; workspaces previously using different Hive Metastores will have namespace collisions that require manual resolution before migration proceeds.

• External location conflicts — Tables created in Hive Metastore with LOCATION pointing to ADLS Gen2 paths need those paths registered as External Locations in Unity Catalog before they can be referenced. Unregistered paths will cause PERMISSION_DENIED errors that are not always immediately obvious in their root cause.

• HMS sync and managed table ownership — Managed tables in the legacy Hive Metastore are owned by the workspace; after migration, Unity Catalog requires explicit ownership assignment at catalog, schema, and table levels. Missing this step leads to silent governance gaps where tables exist but have no effective steward.

• Privilege inheritance gaps — Unity Catalog does not automatically inherit Hive Metastore ACLs. Every permission must be explicitly re-granted. In regulated environments, this is a compliance event, not just a technical step — it should be logged, reviewed, and signed off.

None of these friction points invalidate the framework. But acknowledging them is part of Manana — honest reflection on what the path actually involves, not just what it looks like on a whiteboard.

The Unified Self in Production

When Unity Catalog is implemented with integrity — when the three-level namespace is consistently applied, when Adhikara is encoded at the column level, when Delta Sharing enables Vasudhaiva Kutumbakam with partners — something remarkable happens.

The data estate stops feeling like a collection of separate systems and begins to feel like a single, coherent intelligence. Analysts from different teams can trust each other's data because they share a common governance layer. Engineers spend less time negotiating access and more time building insight. The organisation stops managing multiplicity and starts experiencing unity.

This is not a metaphor. It is a measurable operational outcome.

But the Vedantic framing adds something that the purely technical framing misses: it reminds us why this matters. The fragmentation of data is not just a technical debt problem. It is a reflection of a fragmented organisational mind — a mind that has forgotten its own unity and is experiencing the suffering of Maya.

Unity Catalog does not just solve a technical problem. It is an invitation to a different way of thinking — one where the data estate is understood as a unified whole, where governance is understood as Dharma (right action in context), and where the role of the data engineer is not just to move bytes but to establish clarity where confusion reigns.

Tat tvam asi — Thou art That. The data and the business are not separate. The engineer and the organisation are not separate. The governance layer and the governed data are not separate. When this is truly understood — not as a slogan but as a lived architectural principle — the unified self emerges in production.*

A Note on Tools vs. Principles

This article uses Databricks Unity Catalog as its primary example — deliberately, because it is the most complete implementation of unified data governance available today on a cloud lakehouse platform. But the philosophical principles are not Databricks-specific.

The same Advaitic framework applies to any serious data governance implementation: Apache Atlas for metadata management, AWS Glue Data Catalog for AWS-native estates, Microsoft Purview for Azure-wide governance, or a Data Mesh architecture where federated computational governance replaces centralised control. The specific tool enforces the principle; it does not originate it.

What Unity Catalog offers is a particularly coherent operationalisation of the non-dual ideal — one catalog, one lineage, one governed ground. If your organisation uses a different stack, the question to ask is the same: does your governance layer establish one source of ontological truth aka Single Source of Truth (SSOT), or does it manage the proliferation of many? The answer determines whether your architecture reflects Brahman or perpetuates Maya — regardless of the vendor logo on the dashboard.

Conclusion

Unity Catalog is, technically, a centralised metadata and governance layer for Databricks workspaces. It solves real problems: cross-workspace data sharing, fine-grained access control, lineage tracking, and audit compliance.

But at a deeper level, it is an architectural expression of Advaitic wisdom: the recognition that what appears as many is, at its root, one — and that the role of good architecture, like the role of good philosophy, is to make that unity visible, governable, and available to all who are qualified to receive it.

Build with Unity Catalog. Build with unity.

Karthik Darbha is a Senior Data Engineering & AI Leader with 23 years of professional experience, including 20+ years building enterprise data platforms across Healthcare, Pharma, Retail, Insurance, and Financial Services.

Why This Migration Is Non-Trivial

Earlier, I worked as Product Owner and Data Architect on a SAS to Databricks migration for a Pharma Supply Chain and Manufacturing client. One deliverable stood out: migrating Statistical Process Control (SPC) logic — specifically the 8 SPC Run Rules — from SAS Data Step to PySpark.

SPC is a regulatory obligation in pharmaceutical manufacturing. Run rules operationalize this — they catch statistical signals before a measurement breaches a hard specification limit.

Why 8 points for Rule 2? Rule 2 uses 8 consecutive points on one side of the mean. This reflects a deliberate sensitivity trade-off widely adopted in pharma LIMS/QMS systems — a slightly more sensitive threshold where the cost of a missed shift outweighs the cost of an extra investigation.

The 8 SPC Run Rules

The SAS Paradigm

SAS Data Step processes one row at a time. The RETAIN statement persists values across iterations — making run-counter logic trivial:

/* Rule 2 in SAS — naturally sequential */
data spc_out;
  set process_data;
  retain run_count 0 last_side ' ';

  /* All three cases must be explicit — on-mean points break the run */
  if      value > mean then side = 'A';
  else if value < mean then side = 'B';
  else                      side = 'C';

  if side = 'C' then do;
    run_count = 0;
    last_side = 'C';
  end;
  else if side = last_side then run_count + 1;
  else do;
    run_count = 1;
    last_side = side;
  end;

  /* fires EXACTLY at the 8th point — onset semantics are free */
  rule_2 = (run_count = 8);
run;

Three properties make SAS the natural host:

• Implicit cursor — PDV advances one row at a time

• Persistent state — via RETAIN, free and automatic

• Onset detection — trivially correct; fires when run_count == 8, resets on side-change or on-mean point

Four Spark Challenges

1. No shared cursor

A Spark DataFrame is distributed across many executors — there is no single sequential pass. Solution: Window.partitionBy('batch_id', 'parameter_name').orderBy('measurement_timestamp') guarantees correct ordering within a partition-window. Ensure that batch_id and parameter_name define complete logical boundaries and that chronological ordering is never disrupted at partition edges. Validate with synthetic boundary-crossing test data before deploying to production.

2. No implicit state — and memory pressure

SAS RETAIN has no Spark equivalent. The idiomatic bridge: collect_list() over a Window frame + Higher-Order Functions (aggregate(), forall(), slice()) applied to the resulting ordered array.

One operational constraint is important: collect_list() pulls all values in the window into a single executor's memory. For an SPC batch with millions of sensor readings per batch_id, this can trigger OutOfMemory errors. The mitigation is straightforward — use a bounded window (rowsBetween(-14, 0)) rather than unbounded preceding. Since no SPC rule requires more than 15 contiguous observations, capping the array at 15 elements eliminates the memory risk without any loss of rule accuracy.

For very high-scale deployments, a Pandas UDF approach can achieve significantly better throughput — see Vectorized Approach below.

3. Onset detection is not free

A naive HOF check (exists(), forall()) fires at positions N, N+1, N+2 — re-triggering on the same run. The fix: a named_struct accumulator inside aggregate() that tracks (last_side, run_len), incrementing on continuation and resetting on side-change or on-mean observation. The rule fires only when run_len == exactly N.

4. Cross-batch state loss

In a batch SPC pipeline, if a run of 8 points straddles two incremental loads — 5 points in Job A, 3 points in Job B — the window-based accumulator loses state between jobs and will not fire. For near-real-time or incremental SPC monitoring, the correct pattern is Stateful Structured Streaming with mapGroupsWithState(), which persists run state across micro-batches via a checkpoint. See Cross-Batch Runs below.

PySpark Implementation

Setup

from pyspark.sql import functions as F
from pyspark.sql.window import Window
from pyspark.sql.types import StructType, StructField, DoubleType, IntegerType

# Bounded window — caps array at 15 elements, satisfying all SPC rule windows
# and preventing executor OOM on large batches
w_bounded = Window.partitionBy('batch_id', 'parameter_name') \
                  .orderBy('measurement_timestamp') \
                  .rowsBetween(-14, 0)

vals     = F.collect_list('value').over(w_bounded)
sigma    = F.first('sigma').over(w_bounded)
mean_val = F.first('mean_val').over(w_bounded)

# Normalised z-score array
z_arr = F.transform(vals, lambda x: (x - mean_val) / sigma)

Handling missing values: SPC rules are sensitive to gaps in measurement_timestamp. A gap in the time series should reset run counters — the run has been interrupted. Pre-process the DataFrame to detect gaps before windowing:

w_order = Window.partitionBy('batch_id', 'parameter_name') \
                .orderBy('measurement_timestamp')

df_gap_aware = df.withColumn(
    'prev_timestamp',
    F.lag('measurement_timestamp').over(w_order)
).withColumn(
    'gap_exceeded',
    (F.unix_timestamp('measurement_timestamp') -
     F.unix_timestamp('prev_timestamp')) > 300  # 5-minute threshold — adjust per SOP
)

# Partition on gap boundaries so each continuous segment is windowed independently
# Implementation depends on your SOP — filter, flag, or introduce a segment_id column

Document the gap threshold in your data dictionary. This is a business rule, not a Spark concern.

Rule 1 — Single point beyond 3σ

rule1 = F.abs(z_arr.getItem(F.size(z_arr) - 1)) > 3.0

Rule 2 — Onset-correct 8-point run

Values exactly at the mean (z = 0) break the run — the same behaviour as the SAS side = 'C' case. The accumulator resets to 0 on an on-mean observation, resets to 1 on a side change, and increments on continuation. The rule fires at exactly the 8th consecutive point — not before, not after.

def side_of(z):
    """Return 1 (above mean), -1 (below mean), or 0 (on mean)."""
    return F.when(z > 0, F.lit(1)) \
            .when(z < 0, F.lit(-1)) \
            .otherwise(F.lit(0))

rule2 = F.aggregate(
    z_arr,
    F.named_struct(
        F.lit('last_side'), F.lit(0),
        F.lit('run_len'),   F.lit(0)
    ),
    lambda acc, z: F.named_struct(
        F.lit('last_side'), side_of(z),
        F.lit('run_len'),
        F.when(
            (side_of(z) != F.lit(0)) & (side_of(z) == acc['last_side']),
            acc['run_len'] + 1          # Continue run on same side
        ).when(
            side_of(z) != F.lit(0),     # Side change
            F.lit(1)                    # Start new run at 1
        ).otherwise(
            F.lit(0)                    # On-mean: reset
        )
    ),
    lambda acc: acc['run_len'] == 8     # Fire at exactly the 8th point
)

Validation contract:

• 7 points above mean → rule2 = False

• 8 points above mean → rule2 = True at position 8 only

• 9 points above mean → rule2 = False (no re-fire)

• 8 above, 1 on-mean, 8 above → fires at positions 8 and 17 (two separate runs)

Rule 3 — 6 consecutive trending points (onset detection)

The accumulator tracks {prev, dir, run}. Direction is computed once per step from the current and previous z-score, stored in dir, and compared against acc['dir'] in the next step. A flat step (equal consecutive values) breaks the trend and resets the streak.

# Rule 3: 6 consecutive points strictly increasing or decreasing
# Accumulator fields:
#   prev — previous z-score (null sentinel on first observation)
#   dir  — +1 increasing, -1 decreasing, 0 flat or first step
#   run  — current streak length

def _current_dir(z, prev):
    """Direction from prev to z. Null prev → no direction."""
    return (
        F.when(prev.isNull(), F.lit(0))
         .when(z > prev,      F.lit(1))
         .when(z < prev,      F.lit(-1))
         .otherwise(F.lit(0))
    )

rule3 = F.aggregate(
    z_arr,
    F.named_struct(
        F.lit('prev'), F.lit(None).cast('double'),
        F.lit('dir'),  F.lit(0),
        F.lit('run'),  F.lit(0)
    ),
    lambda acc, z: F.named_struct(
        F.lit('prev'), z,
        F.lit('dir'),  _current_dir(z, acc['prev']),
        F.lit('run'),
        F.when(
            acc['prev'].isNotNull() &
            (_current_dir(z, acc['prev']) == acc['dir']) &
            (acc['dir'] != F.lit(0)),
            acc['run'] + 1              # Continue trend
        ).otherwise(
            F.lit(1)                    # New direction, flat, or first step
        )
    ),
    lambda acc: (acc['run'] == 6) & (acc['dir'] != 0)  # Fire at exactly 6th point
)

Rule 4 — 14 alternating points

For each inner point in the last 14 observations, the product of the left and right deltas must be negative — confirming a direction reversal at every step.

# Rule 4: 14 consecutive points alternating up/down
# last14 indices: 0 .. 13
# Inner loop: i = 1 .. 12, accessing i-1 (0..11), i (1..12), i+1 (2..13) — all safe

last14 = F.slice(z_arr, F.size(z_arr) - 13, 14)

rule4 = (F.size(z_arr) >= 14) & F.aggregate(
    F.sequence(F.lit(1), F.lit(12)),    # 12 inner points; accesses indices 0..13
    F.lit(True),
    lambda acc, i: acc & (
        ((last14.getItem(i)     - last14.getItem(i - 1)) *
         (last14.getItem(i + 1) - last14.getItem(i)))     < 0
    )
)

Rules 5, 6, 7, 8 — Window patterns

# Rule 5: 2 of 3 consecutive beyond ±2σ, same side
last3  = F.slice(z_arr, F.size(z_arr) - 2, 3)
above2 = F.aggregate(last3, F.lit(0), lambda a, z: a + F.when(z >  2.0, F.lit(1)).otherwise(F.lit(0)))
below2 = F.aggregate(last3, F.lit(0), lambda a, z: a + F.when(z < -2.0, F.lit(1)).otherwise(F.lit(0)))
rule5  = (F.size(z_arr) >= 3) & ((above2 >= 2) | (below2 >= 2))

# Rule 6: 4 of 5 consecutive beyond ±1σ, same side
last5  = F.slice(z_arr, F.size(z_arr) - 4, 5)
above1 = F.aggregate(last5, F.lit(0), lambda a, z: a + F.when(z >  1.0, F.lit(1)).otherwise(F.lit(0)))
below1 = F.aggregate(last5, F.lit(0), lambda a, z: a + F.when(z < -1.0, F.lit(1)).otherwise(F.lit(0)))
rule6  = (F.size(z_arr) >= 5) & ((above1 >= 4) | (below1 >= 4))

# Rule 7: 15 consecutive within ±1σ (stratification)
last15 = F.slice(z_arr, F.size(z_arr) - 14, 15)
rule7  = (F.size(z_arr) >= 15) & F.forall(last15, lambda z: F.abs(z) <= 1.0)

# Rule 8: 8 consecutive beyond ±1σ, either side (mixture)
last8  = F.slice(z_arr, F.size(z_arr) - 7, 8)
rule8  = (F.size(z_arr) >= 8) & F.forall(last8, lambda z: F.abs(z) > 1.0)

Mutual exclusivity — priority waterfall

Each observation receives exactly one rule label. Priority: R1 → R2 → R5 → R6 → R3 → R4 → R7 → R8 — severity-first, consistent with pharma QMS convention.

df_out = df.withColumn('spc_rule',
    F.when(rule1, F.lit('R1_3sigma'))
     .when(rule2, F.lit('R2_run8_same_side'))
     .when(rule5, F.lit('R5_2of3_beyond_2sigma'))
     .when(rule6, F.lit('R6_4of5_beyond_1sigma'))
     .when(rule3, F.lit('R3_trend6'))
     .when(rule4, F.lit('R4_alternating14'))
     .when(rule7, F.lit('R7_stratification15'))
     .when(rule8, F.lit('R8_mixture8'))
     .otherwise(F.lit('NO_SIGNAL'))
).withColumn('rule_fired_at_timestamp', F.current_timestamp())

Vectorized Approach (Pandas UDF)

For high-scale deployments — millions of sensor readings per batch — Pandas UDFs execute NumPy operations in vectorized batches on the executor, avoiding JVM serialization overhead. This can be significantly faster than nested Spark SQL HOFs for complex multi-rule logic.

When to prefer Pandas UDFs:

• Processing 1M+ rows per batch_id

• Rules require NumPy/SciPy (e.g., EWMA, rolling z-test)

• Multiple rules computed in a single pass

import pandas as pd
import numpy as np
from pyspark.sql.functions import pandas_udf
from pyspark.sql.types import StructType, StructField, BooleanType

@pandas_udf(
    StructType([
        StructField('rule2_fired', BooleanType(), True),
        StructField('rule3_fired', BooleanType(), True),
    ])
)
def spc_rules_vectorized(z_values: pd.Series) -> pd.DataFrame:
    """Compute Rule 2 and Rule 3 for an ordered window of z-scores."""
    z = z_values.values  # NumPy array

    # Rule 2: 8 consecutive same side (onset detection)
    rule2_result = np.zeros(len(z), dtype=bool)
    run_len, last_side = 0, 0
    for i, zi in enumerate(z):
        side = 1 if zi > 0 else (-1 if zi < 0 else 0)
        if side == 0:
            run_len, last_side = 0, 0
        elif side == last_side:
            run_len += 1
        else:
            run_len, last_side = 1, side
        rule2_result[i] = (run_len == 8)

    # Rule 3: 6 consecutive trending (onset detection)
    rule3_result = np.zeros(len(z), dtype=bool)
    run_len, last_dir, prev_z = 0, 0, None
    for i, zi in enumerate(z):
        if prev_z is None:
            prev_z = zi
            continue
        cur_dir = 1 if zi > prev_z else (-1 if zi < prev_z else 0)
        if cur_dir == 0:
            run_len, last_dir = 0, 0
        elif cur_dir == last_dir:
            run_len += 1
        else:
            run_len, last_dir = 1, cur_dir
        rule3_result[i] = (run_len == 6) and (last_dir != 0)
        prev_z = zi

    return pd.DataFrame({'rule2_fired': rule2_result, 'rule3_fired': rule3_result})

Cross-Batch Runs (Stateful Streaming)

Production pharma SPC systems often operate in near-real-time: measurements arrive as they are taken, not in daily batch files. If a run of 8 points straddles two micro-batches — 5 points in the first, 3 in the second — the window-based accumulator loses state at the batch boundary and will not fire the rule.

The correct pattern is mapGroupsWithState() in Spark Structured Streaming, which persists (last_side, run_len) in a managed state store across micro-batches.

from dataclasses import dataclass
from typing import Iterator
from pyspark.sql.streaming import GroupState, GroupStateTimeout
from pyspark.sql import Row

@dataclass
class SpcRunState:
    last_side: int = 0
    run_len:   int = 0

def update_spc_state(
    group_key: tuple,           # (batch_id, parameter_name)
    measurements: Iterator[Row],
    state: GroupState
) -> Iterator[Row]:
    """
    Stateful SPC Rule 2 processor.
    Preserves run state across Spark micro-batches.
    """
    current = state.get if state.exists else SpcRunState()

    results = []
    for row in sorted(measurements, key=lambda r: r.measurement_timestamp):
        z    = row.z_score
        side = 1 if z > 0 else (-1 if z < 0 else 0)

        if side == 0:
            current.run_len, current.last_side = 0, 0
            rule2_fired = False
        elif side == current.last_side:
            current.run_len += 1
            rule2_fired = (current.run_len == 8)
        else:
            current.run_len, current.last_side = 1, side
            rule2_fired = False

        results.append({**row.asDict(), 'rule2_fired': rule2_fired})

    state.update(current)
    yield from results

# Apply:
df_stream = (
    spark.readStream
    .schema(input_schema)
    .load('/mnt/bronze/sensor_stream')
    .withColumn('z_score', (F.col('value') - F.col('mean')) / F.col('sigma'))
    .groupby('batch_id', 'parameter_name')
    .applyInPandasWithState(
        update_spc_state,
        output_schema,
        state_schema,
        'append',
        GroupStateTimeout.ProcessingTimeTimeout()
    )
)

df_stream.writeStream \
    .option('checkpointLocation', '/checkpoints/spc_stream') \
    .option('mergeSchema', 'true') \
    .toTable('gold_spc_alerts')
    .start()

Unit Testing Edge Runs

Before deploying to a validated pharma environment, test the accumulator logic explicitly against boundary conditions. These tests should be part of your CI/CD pipeline.

def test_rule2_fires_at_exactly_8():
    """No early fire, no re-fire after position 8."""
    z = [0.5] * 9
    result = compute_rule2(z)
    assert result[6] == False, "Must not fire at position 7"
    assert result[7] == True,  "Must fire at position 8"
    assert result[8] == False, "Must not re-fire at position 9"

def test_rule2_on_mean_resets():
    """On-mean point resets the run; second run fires independently."""
    z = [0.5] * 8 + [0.0] + [0.5] * 8
    result = compute_rule2(z)
    assert result[7]  == True,  "First run fires at position 8"
    assert result[16] == True,  "Second run fires at position 17"

def test_rule3_flat_breaks_trend():
    """A flat (equal) step resets the trend counter."""
    z = [0.5, 0.6, 0.7, 0.8, 0.9, 1.0,   # 5 increasing steps
         1.0,                               # flat — breaks trend
         1.1, 1.2, 1.3, 1.4, 1.5, 1.6]    # new run of 6
    result = compute_rule3(z)
    assert result[5]  == False, "Broken trend must not fire"
    assert result[12] == True,  "New run of 6 must fire"

def test_rule4_alternating_14():
    """14 alternating points fires Rule 4."""
    z = [0.5, -0.5, 0.6, -0.6, 0.7, -0.7, 0.8, -0.8,
         0.9, -0.9, 1.0, -1.0, 1.1, -1.1]
    result = compute_rule4(z)
    assert result[13] == True, "Rule 4 must fire at 14th alternating point"

def test_rule5_minimum_window():
    """Rule 5 requires at least 3 points."""
    z = [2.5]
    result = compute_rule5(z)
    assert result[0] == False, "Rule 5 must not fire on 1-point window"

def test_cross_batch_rule2():
    """Run spanning two batches fires at overall position 8 via stateful processor."""
    state_after_a = process_batch_stateful([0.5] * 5)   # 5 above in Batch A
    result_b      = process_batch_stateful([0.5] * 3, initial_state=state_after_a)
    assert result_b[2] == True, "Rule 2 must fire at the 3rd point of Batch B (8th overall)"

SAS vs PySpark

Medallion Architecture Placement

Unity Catalog governance: Catalog pharma_manufacturing → schemas raw, curated, analytics. Delta Lake time-travel supports regulatory investigation. ZORDER on (batch_id, parameter_name, measurement_timestamp) eliminates file scanning on QMS query patterns.

Production Design Considerations

Schema enforcement

Do not hard-code sigma and mean_val as literals. Use broadcast variables for small control limits tables or joins for dynamically computed limits:

# Broadcast: for small, infrequently changing control limits
control_limits_bc = spark.sparkContext.broadcast(
    control_limits_df.collect()
)

# Join: for dynamically computed batch-level statistics
df_with_stats = df.join(
    control_stats,
    on=['batch_id', 'parameter_name'],
    how='left'
)

Sigma estimator

Document which estimator was used to compute sigma — moving range, sample SD, pooled, or UWMA. This is a regulatory and statistical decision, not a Spark concern. Example using moving range (the SAS default for individuals charts):

w_order = Window.partitionBy('batch_id', 'parameter_name') \
                .orderBy('measurement_timestamp')

df_sigma = df.withColumn(
    'moving_range',
    F.abs(F.col('value') - F.lag('value').over(w_order))
).withColumn(
    'sigma_estimate',
    F.avg('moving_range').over(w_order) / 1.128  # d2 constant for n=2
)

Partition alignment

Repartition on (batch_id, parameter_name) before writing Silver and apply ZORDER on timestamp to ensure physical file layout aligns with logical query patterns:

df.repartition('batch_id', 'parameter_name') \
    .write.format('delta') \
    .mode('overwrite') \
    .option('zorderBy', 'batch_id,parameter_name,measurement_timestamp') \
    .saveAsTable('silver_spc_zscores')

Alert de-duplication

In operational deployments, the same rule can fire on consecutive rows within a run. Apply a de-duplication layer before writing to your LIMS alerting table:

df_alerts = (
    df_out
    .filter(F.col('spc_rule') != 'NO_SIGNAL')
    .withColumn('alert_id', F.md5(
        F.concat_ws('|',
            F.col('batch_id'),
            F.col('parameter_name'),
            F.col('spc_rule'),
            F.date_trunc('hour', F.col('rule_fired_at_timestamp'))
        )
    ))
    .dropDuplicates(['alert_id'])
)

The Witness and the Process Stream

Patanjali's Yoga Sutras II.17 identifies the root of suffering as the association between the Seer (Drashtri) and the Seen (Drishya). Advaita Vedanta resolves this through Sakshi — the Witness that observes all phenomena without identification or reaction.

A well-designed SPC monitor is, in a small engineering sense, a Sakshi. The process stream flows — measurements rise, fall, drift, oscillate. The system neither panics nor ignores. Rule 1 fires not because the system is alarmed, but because it has accurately perceived what is. The onset-correct accumulator fires exactly once, at the right moment, on the right signal, without noise.

The transition from SAS to Spark mirrors a deeper shift: from implicit state (RETAIN) to explicit state (aggregate(), mapGroupsWithState()). That explicitness demands clarity — you must name your assumptions, test your edge cases, and document your intentions. That rigour is a form of witness-consciousness in code.

Reliable observation requires correct architecture. Turīya — the fourth state of the Mandukya Upanishad — witnesses the three states (waking, dream, deep sleep) without being any of them. The Gold layer, sitting above Bronze and Silver, witnesses the process without being the process.

— tech4nirvana.com

References

• FDA 21 CFR Part 211 — Current Good Manufacturing Practice for Finished Pharmaceuticals.

• ICH Q10: Pharmaceutical Quality System. 2008.

• ICH Q14: Analytical Procedure Development. 2023.

• Apache Spark Documentation — Window Functions and Higher-Order Functions.

• Databricks Documentation — Delta Lake Time Travel, Unity Catalog, Structured Streaming, mapGroupsWithState.

• Apache Spark — Pandas UDFs with Arrow: https://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html

• Patanjali. Yoga Sutras II.17: Drashtri-drishyayoh samyogo heya-hetuh.

• Shankaracharya. Mandukya Upanishad Bhashya — on Turīya as the fourth (witnessing) state.

There is an ancient Sanskrit phrase that has guided seekers of truth for over a thousand years: "Tat Tvam Asi" — You are That. At its core, Advaita Vedanta, the non-dualist school of Indian philosophy codified by Adi Shankaracharya in the 8th century, teaches that the apparent multiplicity of the world is an illusion. Beneath all diversity lies a single, undivided reality — Brahman.

As a data engineer who has spent over two decades building pipelines, architecting data platforms, and debugging production failures at 2am, I have come to realize something quietly profound: the principles of Advaita Vedanta map onto the challenges of modern data engineering with remarkable precision.

This is not mysticism. This is epistemology — the study of how we know what we know. And data engineering, at its heart, is an epistemological discipline.

Maya: The Illusion of Raw Data

In Advaita Vedanta, Maya (माया) refers to the cosmic illusion — the tendency of the mind to mistake the appearance of things for their ultimate reality. The world we perceive through our senses is real in a practical sense, but it conceals a deeper truth.

In data engineering, raw data is Maya.

A source system presents you with a table of transactions. It looks real. It looks complete. But dig deeper and you find:

• Duplicate records from retry logic

• NULL values where business rules demand non-null

• Timestamps in five different formats across three source systems

• Currency values without denomination codes

• Customer IDs that changed silently after a system migration

The raw data is not lying to you — it is simply presenting its surface reality. The data engineer's job is to pierce the veil of Maya, to look past the apparent truth of the source and ask: what is the actual business reality this data represents?

The Medallion Architecture — Bronze, Silver, Gold — is, in this sense, a structured practice of moving from Maya toward truth. Bronze is raw reality as it arrives. Silver is cleansed, conformed reality. Gold is the curated truth the business actually needs.

Viveka: The Practice of Discrimination

Viveka (विवेक) is one of the four qualifications (Sadhana Chatushtaya) that Shankara prescribed for a serious student of Vedanta. It means discrimination — the ability to distinguish the real from the unreal, the permanent from the impermanent, the essential from the incidental.

In data engineering, Viveka is your data quality framework.

Every day, a data engineer exercises Viveka:

• Is this NULL a missing value or a legitimate unknown?

• Is this spike in the metric a real business event or a pipeline anomaly?

• Is this schema change backward compatible or breaking?

• Should this logic live in the transformation layer or the serving layer?

Without Viveka, data pipelines become swamps of technical debt. Every table gets every column. Every pipeline carries every edge case. The system grows heavy with the unreal mistaken for the real.

The practice of Viveka in data engineering means building systems that know what they are for, and refusing to carry what they are not.

Neti Neti: The Power of Elimination

One of the most powerful methods in Advaita Vedanta is Neti Neti (नेति नेति) — Not this, not this. Rather than trying to define Brahman positively, the seeker systematically eliminates everything that Brahman is not. What remains, when all the unreal has been stripped away, is the truth.

In data engineering, Neti Neti is your schema design and debugging philosophy.

When designing a dimensional model, you ask:

• Is this a fact? Neti — it changes too slowly.

• Is this a dimension? Neti — it has no independent existence without a transaction.

• Is this a measure? Neti — it cannot be aggregated meaningfully.

When debugging a pipeline failure:

• Is it the source system? Neti — the raw data looks clean.

• Is it the transformation logic? Neti — unit tests pass.

• Is it the infrastructure? Iti — yes, the Spark executor ran out of memory due to data skew.

The senior data engineer is not the one who immediately knows the answer. The senior data engineer is the one who knows how to eliminate systematically until the truth reveals itself.

Brahman: The Single Source of Truth

In Advaita Vedanta, Brahman (ब्रह्मन्) is the ultimate reality — the single, undivided, infinite consciousness that underlies all apparent multiplicity. Everything that exists is, in its deepest nature, Brahman.

In data engineering, Brahman is your Single Source of Truth.

Every enterprise data platform is, in a sense, a temple to Brahman. The goal is to create one authoritative, trusted, governed representation of business reality — whether that is:

• A unified customer identity across CRM, billing, and support systems

• A canonical product hierarchy reconciled across ERP and e-commerce

• A single financial ledger that the CFO, auditors, and analysts all agree on

The tragedy of most data platforms is that they multiply Atman instead of realizing Brahman. Every team builds its own mart. Every analyst has their own definition of "active customer." Every dashboard shows a slightly different revenue number.

Unity Catalog in Databricks, data contracts, semantic layers — these are not just technical tools. They are institutional practices of non-duality. They assert: there is one truth, and we will govern access to it, not multiply it.

Upadesha Saram: The Essence of the Teaching

Ramana Maharshi's Upadesha Saram (उपदेश सारम्) distills the entirety of Vedantic practice into 30 verses. Its central teaching is self-inquiry: rather than seeking truth outside, turn attention inward and ask "Who am I?"

For a data engineer, self-inquiry means questioning your own assumptions before building anything:

• Why does this data exist?

• Who will use this output and how?

• What breaks if this is wrong?

• Am I solving the real problem or the stated problem?

The greatest data engineering failures I have witnessed in 22 years were not technical failures. They were failures of inquiry — teams that built what was asked without asking why, that optimized pipelines for throughput without asking whether the data was trusted.

Self-inquiry in data engineering is not navel-gazing. It is the highest form of rigor.

Conclusion: The Engineer as Seeker

Advaita Vedanta does not ask you to abandon the world. It asks you to engage with the world with clarity — to act effectively in the empirical realm while remaining anchored in the understanding of ultimate truth.

This is precisely what great data engineering demands.

Build your pipelines. Design your schemas. Optimize your Spark jobs. But do all of this with Viveka. Pierce the Maya of raw data. Apply Neti Neti to eliminate the inessential. And always pursue Brahman — the single, unified, trusted truth your organization can build decisions upon.

The data platform is not just infrastructure. It is, in its highest aspiration, an instrument of clarity.

Tat Tvam Asi. That is what the data, in its deepest truth, is trying to say.

Karthik Darbha is a Data Engineering & AI Leader with over 22 years of experience in Healthcare, Pharma, Retail, Insurance, and Financial Services. He writes at tech4nirvana.com, exploring the intersection of data architecture and timeless wisdom.