Letting Go of Control: What Advaita Teaches Us About Liquid Clustering

There is a pattern in mature engineering: the more you try to control, the more brittle the system becomes. Liquid Clustering is Delta Lake's answer to over-engineering. Vedanta figured this out millennia ago.

The Bhagavad Gita Problem in Data Engineering

Arjuna's dilemma on the battlefield of Kurukshetra is, at its core, a crisis of control. He wants to manage outcomes — to know, in advance, exactly what will happen if he acts. Krishna's answer is radical: act without attachment to results. Do what the moment demands, not what your anxiety prescribes.

Data engineers face a structurally similar problem when they reach for ZORDER BY. Every time they write to a large table, they feel compelled to run OPTIMIZE + ZORDER BY — to impose order, manually, comprehensively. It feels responsible. It feels controlled. But it is expensive, often redundant, and operationally unsustainable at scale.

Liquid Clustering is the engineering equivalent of nishkama karma — action without the compulsive need to orchestrate every outcome.

Key idea: Liquid Clustering does not optimize everything. It optimizes what needs optimizing. This is not laziness — it is precision grounded in awareness.

What Is Liquid Clustering? (The Technical Grounding)

Liquid Clustering replaces ZORDER BY with an incremental, stateful optimization that tracks file-level clustering health. Available from Delta 3.x (Databricks Runtime 13.3+), it rewrites only data files that have drifted from the clustering target — not the whole table.

In practical terms: if you have a 10 TB sales_transactions table clustered by rep_id and sale_date, and an hourly ingestion job appends 5 GB of new records, a subsequent OPTIMIZE touches only the new files — not all 10 TB. The already-clustered data is left undisturbed.

Maya and the Illusion of Total Order

Advaita Vedanta teaches that Maya — often translated as illusion — is not the claim that the world does not exist, but that we mistake partial appearances for ultimate reality. We see a table with millions of rows and believe that perfect, complete, always-current physical ordering is both achievable and necessary. This is Maya at the engineering layer.

There is a precise geometric reason why this illusion breaks down. ZORDER BY uses a Z-order curve (also called a Morton curve) — a mathematical technique that maps multi-dimensional data onto a single linear sequence while attempting to preserve locality. It works reasonably well in two or three dimensions. But as you add columns, the curve's locality-preserving property degrades rapidly. Points that are close in multi-dimensional space end up far apart on the linear curve. The ordering becomes increasingly arbitrary. You are not achieving global order — you are achieving the appearance of order, at increasing cost.

Liquid Clustering abandons this pretension entirely. It uses a multidimensional clustering approach that does not attempt to collapse all dimensions into a single line. Instead, it tracks clustering health per file, per column set, and acts only where the data has genuinely drifted. It does not chase a Z-curve that was never fully achievable. This is why the 4-column limit that cripples ZORDER BY does not apply in the same way to Liquid Clustering — the underlying geometry is different.

The truth is that queries do not need perfect global order. They need sufficient local clustering — enough that the query planner can skip irrelevant files. Liquid Clustering understands this. It does not chase an impossible ideal; it tracks the real state of each file and acts only where action is warranted.

Enabling Liquid Clustering (SQL):

CREATE TABLE pharma.silver.sales_transactions
  CLUSTER BY (rep_id, sale_date)
AS SELECT * FROM bronze.sales_transactions_raw;

Enabling via PySpark:

from delta.tables import DeltaTable

DeltaTable.createOrReplace(spark) \
  .tableName('pharma.silver.sales_transactions') \
  .addColumn('rep_id', 'STRING') \
  .addColumn('sale_date', 'DATE') \
  .addColumn('product_code', 'STRING') \
  .addColumn('territory', 'STRING') \
  .addColumn('revenue', 'DOUBLE') \
  .clusterBy('rep_id', 'sale_date') \
  .execute()

Alter an existing table:

ALTER TABLE pharma.silver.sales_transactions
  CLUSTER BY (rep_id, sale_date);

The Witness Consciousness of OPTIMIZE

In Advaita, the concept of Sakshi — the witness — describes a mode of awareness that observes without compulsive intervention. The witness knows what is happening without needing to control every outcome. It acts when necessary; it rests when not.

This is precisely how OPTIMIZE behaves on a Liquid Clustering-enabled table. It inspects file statistics. It evaluates clustering health. It writes only what needs rewriting. It is not passive — it is precisely calibrated.

-- OPTIMIZE as Sakshi: acts only where action is needed
OPTIMIZE pharma.silver.sales_transactions;

-- Verify the witness's state
DESCRIBE DETAIL pharma.silver.sales_transactions;

Do not mix OPTIMIZE + ZORDER BY on a table with CLUSTER BY. ZORDER BY overrides the clustering key temporarily — the Sakshi becomes confused. Trust the system.

Choosing Cluster Keys: Neti Neti in Practice

Shankara's method of Neti Neti — "not this, not this" — is a process of elimination that reveals truth by discarding what does not qualify. Choosing cluster keys works the same way.

• Not this: low-cardinality columns like region or therapeutic_area — skip files at too coarse a granularity
• Not this: columns that never appear in WHERE clauses — irrelevant to query planning
• Not this: more than 3–4 columns — clustering effectiveness degrades with dimensionality
• This: high-cardinality identifiers — rep_id, product_code — used in point lookups
• This: date/time columns — sale_date, order_date — used in range scans

Pharma example — sales transactions table:

-- Queries almost always filter by rep_id and sale_date range
CREATE TABLE pharma.silver.sales_transactions
  CLUSTER BY (rep_id, sale_date)
AS SELECT * FROM bronze.sales_transactions_raw;

Turiya: The State Beneath All States

Vedanta describes four states of consciousness: waking, dreaming, deep sleep — and Turiya, the fourth, which is not a state at all but the ground from which the other three arise. It is always present, unchanged, whether you are awake or asleep.

Delta Lake's transaction log is the Turiya of your data platform. Every OPTIMIZE, every write, every schema change is recorded there as a discrete action. But the log itself does not change — it only grows. It witnesses all transformations without being transformed. Liquid Clustering, at its foundation, relies on this immutable log to know what has changed and what has not.

Crucially, the witness does not merely observe — it records measurable state. Each data file's entry in the _delta_log JSON contains a stats column that captures per-column min/max values and null counts for the first 32 columns by default. This is the metadata the query planner uses for file skipping — the ability to eliminate entire Parquet files from a scan without reading them. When your cluster keys are well-chosen and Liquid Clustering is healthy, the engine consults these statistics and skips the irrelevant. The Sakshi has already noted what is present and what is not.

-- Inspect raw file statistics in the transaction log
SELECT
  add.path,
  add.stats
FROM json.`abfss://silver@<storage>.dfs.core.windows.net/sales_transactions/_delta_log/*.json`
WHERE add IS NOT NULL
LIMIT 10;

When you understand your data platform this way — as a system with a permanent witness and measurable, transient transformations — you stop being anxious about every write. The log holds truth. OPTIMIZE acts on it. The table reflects reality at any given point in time.

The Engineer Who Does Not Over-Optimize

The Bhagavad Gita's message to Arjuna was not "do nothing." It was "act from clarity, not from fear." ZORDER BY on a large, frequently-written table is often an act of fear — the anxiety that the data is not ordered enough, that queries will be slow, that something will go wrong.

Liquid Clustering asks you to trust the system. Define your cluster keys thoughtfully. Run OPTIMIZE. Let the engine determine what needs rewriting. Observe the results. Intervene only when the evidence demands it.

This is not passivity. This is precision. And it scales.

na hi jnanena sadrsam pavitram iha vidyate

There is no purifier in this world equal to knowledge. — Bhagavad Gita 4.38

Glossary

Technical Terms

Cardinality — The number of distinct values in a column. High cardinality means many unique values (e.g. rep_id); low cardinality means few (e.g. region = North/South/East/West). High-cardinality columns make better cluster keys because they allow the query engine to skip more files.

DBU (Databricks Unit) — The unit of processing capacity used to measure and bill Databricks workloads. OPTIMIZE operations consume DBUs, so minimising unnecessary reclustering has a direct cost impact.

Delta Lake — An open-source storage layer that brings ACID transactions, schema enforcement, and time travel to data lakes. It sits on top of cloud object storage (e.g. Azure Data Lake Storage) and underpins all Databricks table operations.

Delta Transaction Log (_delta_log) — An append-only JSON log that records every operation performed on a Delta table — writes, deletes, schema changes, and optimizations. It is the authoritative source of truth for the table's current and historical state.

File Skipping — A query optimization technique where the engine reads per-file statistics (min/max values, null counts) from the transaction log and eliminates data files that cannot contain rows matching the query's filter predicates — without reading those files at all.

Liquid Clustering — A Delta Lake optimization feature (DBR 13.3+) that incrementally reclusters only data files that have drifted from the defined clustering target, replacing the need for full-table ZORDER BY runs.

OPTIMIZE — A Databricks SQL command that compacts small files and, when Liquid Clustering is enabled, reclusters files that have drifted. On a clustered table, it is idempotent and safe to run repeatedly.

Parquet — The columnar file format used by Delta Lake to store data on disk. Columnar storage means the engine can read only the columns required by a query, and file skipping works at the Parquet file granularity.

Sargability — A query property describing whether a filter predicate can be resolved using an index or statistics (Search ARGument ABLE). Cluster keys should be chosen from sargable columns — those that appear in WHERE clauses and benefit from file-level min/max statistics.

Z-order Curve (Morton Curve) — A mathematical space-filling curve that maps multi-dimensional data onto a single linear sequence while attempting to preserve locality. Used internally by ZORDER BY. Its locality-preserving property degrades significantly beyond 3–4 dimensions, which is why ZORDER BY becomes less effective with more columns.

ZORDER BY — A Databricks OPTIMIZE sub-command that physically co-locates related data using a Z-order curve. Requires a full table scan on every run and becomes geometrically less effective as the number of clustering columns increases.

Vedantic Terms

Advaita Vedanta — One of the principal schools of Hindu philosophy, associated with Adi Shankaracharya (8th century CE). Advaita means non-dual — the teaching that Brahman (ultimate reality) alone exists, and that the apparent multiplicity of the world arises through Maya.

Maya — Often translated as illusion, but more precisely: the power by which ultimate reality appears as the phenomenal world of multiplicity. Maya does not mean the world is unreal — it means we mistake the appearance for the ground. In this post, it refers to the mistaken belief that perfect, exhaustive physical ordering of data is both achievable and necessary.

Neti Neti — Sanskrit for "not this, not this." A method of inquiry attributed to the Brihadaranyaka Upanishad and developed by Shankara, in which truth is approached by systematically negating everything that does not qualify — rather than by positive assertion. Used here as a framework for eliminating poor cluster key candidates.

Nishkama Karma — Sanskrit for "desireless action" or "action without attachment to results." A central teaching of the Bhagavad Gita (Chapter 3): act fully and precisely, but without anxiety about controlling every outcome. Used here to describe Liquid Clustering's approach — act on what needs acting on, leave the rest undisturbed.

Sakshi — Sanskrit for "witness." In Advaita, the Sakshi is the pure awareness that observes all mental and physical phenomena without being modified by them. It is always present, always aware, never reactive. Used here to describe OPTIMIZE's role: it inspects file statistics and acts with precision, not compulsion.

Turiya — Sanskrit for "the fourth." The fourth state of consciousness in Advaita, beyond waking (jagrat), dreaming (svapna), and deep sleep (sushupti). Turiya is not itself a state but the unchanging ground of awareness from which the other three arise and into which they dissolve. Used here to describe the Delta transaction log — the immutable ground that witnesses all table transformations without itself being transformed.

Karthik Darbha is a Senior Data Engineering & AI Leader with 23 years of professional experience, including 20+ years building enterprise data platforms across Healthcare, Pharma, Retail, Insurance, and Financial Services.