# Metadata Intelligence: Making Your Catalog Work for You

## 1\. The Problem

Every data platform has two estates: the data, and the description of the data. Teams invest heavily in the first and let the second decay.

Open almost any mature Unity Catalog and run a simple count of columns with an empty `comment`. In many enterprise estates the undocumented share is not a rounding error — it is the majority. And the documented minority is not necessarily trustworthy: many of those comments were written once, at table creation, and have not been touched through three schema evolutions since.

The cost of this decay is invisible until it isn't:

*   **Discovery failure.** An analyst searches for "customer lifetime value", finds four candidate tables, and picks the one with the friendliest name. Catalog search indexes table names, column names, and comments — when comments are empty, search degrades to guessing.
    
*   **Semantic drift.** A column called `amt` meant *gross transaction amount* in 2023. After a refactor it now carries *net of fees*. The name stayed; the meaning moved. Nothing in the catalog recorded the change.
    
*   **Sensitive data blind spots.** Nobody can protect what nobody has identified. A free-text `notes` column quietly accumulates phone numbers and account references typed in by call-centre agents. Masking policies keyed on tags cannot fire on a column that was never tagged.
    
*   **Tribal semantics.** The real definitions live in the heads of three senior engineers and a Confluence page last edited two reorganisations ago. When one of them is on leave or moves on, the definitions leave with them — knowledge held only in people is a single point of failure.
    

Manual documentation drives fail for a structural reason, not a motivational one: the rate at which columns are created far exceeds the rate at which humans will write prose about them.

* * *

## 2\. The AI Opportunity

Metadata intelligence is really two different problems that are often conflated. They have different epistemics, and they deserve different machinery.

**Classification asks: *what kind of value is this?*** Is this an email address, a national ID, a phone number? The answer is largely contained in the data itself — patterns, formats, distributions. This is a detection problem.

**Description asks: *what does this column mean in this business?*** Whether `amt` is gross or net is not written in the values. The answer lives in lineage, in transformation logic, in how the column is used downstream. This is an inference problem, and it is where generative models are most useful and most dangerous.

The practical consequence for a Databricks shop in late 2026:

*   **For classification, use the platform before you build.** Unity Catalog's Data Classification became generally available in mid-2026. It uses an LLM-assisted agent to scan enabled catalogs, detect sensitive classes (`class.email_address`, `class.us_ssn`, `class.name` and others, plus custom classes), and record detections in `system.data_classification.results`. Combined with governed tags and ABAC policies, detected columns can be masked automatically. Rebuilding this yourself is rarely justified.
    
*   **For description, the native feature is a starting point, not a pipeline.** Catalog Explorer offers AI-generated comments, but they are requested per object through the UI and are described by Databricks as general descriptions based on the schema. That is useful for one table; it does not scale to a catalog, and schema alone cannot resolve `amt`. The opportunity is a batch pipeline that grounds the model in *evidence beyond the name*: upstream column lineage, profile statistics, and the classification signal itself.
    

Here the Upanishadic idea of **nāma-rūpa** — name and form — is unusually precise. The Chāndogya Upanishad's clay teaching holds that a pot, a jar and a plate are distinctions of name arising from speech; the clay is what is real. A column name is exactly that: a label arising from someone's speech at some point in time. The values, and the transformations that produced them, are the clay. A catalog that stores only names stores only nāma. Metadata intelligence is the discipline of pointing the name back toward what it actually names.

* * *

## 3\. Implementation Sketch

The design below runs as a scheduled Databricks job. It treats Data Classification as an upstream input, generates descriptions only from assembled evidence, and never writes to the catalog without human approval.

```plaintext
information_schema.columns ──► Stage 1: find & rank gaps
system.access.table_lineage ─┘
                                   │
system.access.column_lineage ──► Stage 2: evidence bundle ◄── system.data_classification.results
column profile (custom helper) ┘        │                      (class tags only — never samples)
                                        ▼
                               Stage 3: ai_query (structured output)
                                        │
                                        ▼
                         Delta review table (PENDING) ──► steward approval
                                        │
                                        ▼
                         Stage 4: ALTER ... COMMENT + provenance tag
```

### Stage 1 — Find and rank the gaps

Not every undocumented column matters equally. Rank by downstream fan-out so the pipeline spends its budget where documentation is actually consumed.

```python
from pyspark.sql import functions as F

CATALOG = "finance_prod"
LOOKBACK_DAYS = 90

undocumented = (
    spark.table("system.information_schema.columns")
    .filter(F.col("table_catalog") == CATALOG)
    .filter(F.col("table_schema") != "information_schema")
    .filter(F.col("comment").isNull() | (F.trim("comment") == ""))
    .select(
        F.concat_ws(".", "table_catalog", "table_schema", "table_name").alias("table_fqn"),
        "table_catalog", "table_schema", "table_name", "column_name", "data_type",
    )
)

fanout = (
    spark.table("system.access.table_lineage")
    .filter(F.col("event_time") >= F.date_sub(F.current_date(), LOOKBACK_DAYS))
    .filter(F.col("source_table_full_name").startswith(CATALOG + "."))
    .groupBy(F.col("source_table_full_name").alias("table_fqn"))
    .agg(F.countDistinct("target_table_full_name").alias("downstream_consumers"))
)

candidates = (
    undocumented.join(fanout, "table_fqn", "left")
    .fillna({"downstream_consumers": 0})
    .orderBy(F.desc("downstream_consumers"))
    .limit(2000)                      # bound each run; cost is linear in columns
)
```

System tables are not real-time: lineage and audit events land with some ingestion latency. A column created minutes before the run may have no lineage yet. The pipeline treats this as *no evidence yet*, so it defers the column, not abstains permanently.

### Stage 2 — Assemble the evidence bundle

Three sources of evidence, each chosen to add information the column name lacks:

1.  **Upstream column lineage** — what source columns feed this one.
    
2.  **Aggregate profile** — null fraction and approximate distinct ratio. *Aggregates only; no raw values ever enter the prompt.*
    
3.  **Classification signal** — the class tags already detected for this column.
    

```python
upstream = (
    spark.table("system.access.column_lineage")
    .filter(F.col("event_time") >= F.date_sub(F.current_date(), LOOKBACK_DAYS))
    .filter(F.col("source_column_name").isNotNull())
    .select(
        F.col("target_table_full_name").alias("table_fqn"),
        F.col("target_column_name").alias("column_name"),
        F.concat_ws(".", "source_table_full_name", "source_column_name").alias("src"),
    )
    .groupBy("table_fqn", "column_name")
    .agg(F.slice(F.array_sort(F.collect_set("src")), 1, 5).alias("upstream_columns"))
)

# Deliberately select class_tag and confidence ONLY.
# The results table also has a `samples` column containing raw matched values —
# it must never be joined into an LLM prompt.
classes = (
    spark.table("system.data_classification.results")
    .filter(F.col("catalog_name") == CATALOG)
    .select(
        F.concat_ws(".", "catalog_name", "schema_name", "table_name").alias("table_fqn"),
        "column_name", "class_tag", "confidence",
    )
    .groupBy("table_fqn", "column_name")
    .agg(F.collect_set(F.concat_ws(":", "class_tag", "confidence")).alias("detected_classes"))
)

# compute_column_profile() is a CUSTOM HELPER, not a native PySpark/Databricks API.
# It runs approx_count_distinct and a null count on a TABLESAMPLE of each table
# and returns (table_fqn, column_name, null_fraction, distinct_ratio) — aggregates only.
profiles = compute_column_profile(candidates, sample_pct=1)

bundle = (
    candidates
    .join(upstream, ["table_fqn", "column_name"], "left")
    .join(classes,  ["table_fqn", "column_name"], "left")
    .join(profiles, ["table_fqn", "column_name"], "left")
)
```

### Stage 3 — Constrained generation with `ai_query`

The prompt does three things: it restricts the model to the supplied evidence, forces it to declare which evidence it used, and gives it an explicit, respectable way to say *I don't know*.

```python
ENDPOINT = "<your-approved-model-serving-endpoint>"  # endpoint names change; use your governed endpoint

response_format = """{
  "type": "json_schema",
  "json_schema": {
    "name": "column_description",
    "schema": {
      "type": "object",
      "properties": {
        "description":         {"type": "string"},
        "evidence_used":       {"type": "array", "items": {"type": "string"}},
        "confidence":          {"type": "string", "enum": ["HIGH", "MEDIUM", "LOW"]},
        "needs_human":         {"type": "boolean"},
        "suspected_sensitive": {"type": "boolean"}
      },
      "required": ["description", "evidence_used", "confidence", "needs_human", "suspected_sensitive"]
    },
    "strict": true
  }
}"""

INSTRUCTIONS = (
    "You document columns in a data catalog. Use ONLY the evidence provided. "
    "Write one or two plain sentences describing what the column contains. "
    "Do not invent business rules, units, currencies or calculation logic not present in the evidence. "
    "If the name is ambiguous and the evidence does not resolve it, set needs_human=true and say what is ambiguous. "
    "List every evidence field you relied on in evidence_used."
)

proposals = bundle.withColumn(
    "llm",
    F.expr(f"""
      ai_query(
        '{ENDPOINT}',
        concat(
          '{INSTRUCTIONS}',
          '\\nTABLE: ', table_fqn,
          '\\nCOLUMN: ', column_name, ' (', data_type, ')',
          '\\nUPSTREAM_COLUMNS: ', coalesce(array_join(upstream_columns, ', '), 'none recorded'),
          '\\nDETECTED_CLASSES: ', coalesce(array_join(detected_classes, ', '), 'none'),
          '\\nNULL_FRACTION: ', coalesce(cast(round(null_fraction, 3) as string), 'unknown'),
          '\\nDISTINCT_RATIO: ', coalesce(cast(round(distinct_ratio, 3) as string), 'unknown')
        ),
        responseFormat => '{response_format}',
        failOnError => false
      )
    """),
)

parsed_schema = ("description STRING, evidence_used ARRAY<STRING>, confidence STRING, "
                 "needs_human BOOLEAN, suspected_sensitive BOOLEAN")

(proposals
 .withColumn("p", F.from_json(F.col("llm.response"), parsed_schema))
 .withColumn("ai_error", F.col("llm.errorMessage"))
 # Cross-check: two independent signals disagree → route to a steward, regardless of LLM confidence
 .withColumn("classification_conflict",
             F.col("p.suspected_sensitive") != F.col("detected_classes").isNotNull())
 .withColumn("status", F.lit("PENDING"))
 .withColumn("proposed_at", F.current_timestamp())
 .drop("llm")
 .write.mode("append")
 .saveAsTable("governance.metadata.description_proposals"))
```

> **Verify before copy-paste:** `ai_query` options and the exact shape of its return value under `failOnError => false` have evolved across Databricks Runtime releases. Confirm against current docs for your runtime; treat this block as an architectural sketch.

The `classification_conflict` flag is the most valuable column in the table. When the description model suspects sensitivity that Data Classification did not detect — or vice versa — you have two independent systems disagreeing about the same column. That disagreement is precisely where human attention earns its cost.

### Stage 4 — Apply only what a human approved

```python
# sql_string_literal() is a CUSTOM HELPER that escapes quotes/backslashes
# and returns a safe single-quoted SQL literal. It is not a Spark built-in.
approved = (spark.table("governance.metadata.description_proposals")
            .filter("status = 'APPROVED' AND applied_at IS NULL")
            .collect())

for r in approved:
    tbl, col = r.table_fqn, r.column_name
    spark.sql(f"ALTER TABLE {tbl} ALTER COLUMN `{col}` COMMENT {sql_string_literal(r.final_description)}")
    spark.sql(f"ALTER TABLE {tbl} ALTER COLUMN `{col}` SET TAGS ('doc_provenance' = 'ai_drafted_human_approved')")
    # then stamp applied_at on the proposal row (MERGE/UPDATE on the review table) — omitted for brevity
```

At volume, the constraint is not driver memory — approved rows are small. The constraint is that every `ALTER` is a sequential catalog operation and a separate Delta metadata commit. Three habits keep this manageable:

*   Group approvals by table.
    
*   Cap the number of tables per run.
    
*   Use `toLocalIterator()` instead of `collect()` if the approved backlog is genuinely large.
    

DDL is issued from the driver regardless; it cannot be pushed to executors.

Two design choices matter here. The `final_description` field is the steward's edited text, not the raw model output. And the provenance tag means anyone reading the catalog can distinguish *human-authored*, *AI-drafted and human-approved*, and (if you ever allow it) *AI-only* — which is the minimum honesty a catalog owes its readers.

For the PII side, the equivalent workflow is native: enable Data Classification on the catalog, review detections in the results UI, enable auto-tagging per class only after reviewing initial results, and attach ABAC masking policies to the `class.*` tags.

* * *

## 4\. Limitations & Risks

**Plausible-but-wrong descriptions (hallucination).** A wrong description is worse than an empty one, because an empty comment signals *unknown* while a fluent wrong comment signals *known*. The `amt` column is the canonical failure: a model will confidently call it "transaction amount", which is true enough to pass review and wrong enough to corrupt a revenue report.

Śaṅkara opens his Brahma Sūtra commentary with a definition of **adhyāsa** — superimposition — as the appearance, in one place, of something previously seen elsewhere, in the manner of memory. That is a near-exact description of how a language model writes a column comment. It has seen ten thousand columns called `amt` in its training data, and it projects that remembered meaning onto *your* column. The rope is taken for a snake not because the observer is careless, but because the snake was genuinely seen before — somewhere else.

**Stale truth (observability of the AI layer).** A description that was correct at approval becomes wrong after the next refactor. Without monitoring, the catalog regains its old disease with a new veneer of authority. And without metrics — acceptance rate, edit distance between proposal and approved text, conflict rate — you cannot tell whether the generator is improving or decaying.

**Sensitive data in the metadata path (PII exposure).** Three distinct leaks to watch:

*   `system.data_classification.results` stores up to five raw sample values per detection. Databricks restricts it to account admins by default; widening that grant to "make the pipeline work" turns your PII index into a PII sample store.
    
*   Column profiles can leak values if they include min/max or top-k. Aggregates such as null fraction and distinct ratio do not.
    
*   If the serving endpoint routes to an external model provider, every column name, table name and upstream path leaves your boundary. In regulated BFSI estates, table names alone can be sensitive (`fraud_investigation_subjects`).
    

**Absence of a tag is not absence of PII (false confidence).** Data Classification reports `HIGH` or `LOW` confidence on what it detects; it says nothing about what it missed. Free-text fields, embedded JSON, and organisation-specific identifier formats are the classic false negatives. "Zero detections" is a statement about the detector, not about the data.

**Skill gap.** Writing an evidence-constrained prompt, designing a review rubric, and measuring acceptance quality are ML-evaluation skills. Most data engineering teams have not had to build them, and catalog work tends to land with whoever has spare capacity.

**Cost at scale and operational side effects.** Thousands of columns, re-run nightly, add up — in model tokens, in the profiling compute, and in Data Classification's own scanning cost (reported in `system.billing.usage`). Separately, Databricks documents that saving a comment issues an `ALTER` statement that can disrupt running pipelines and jobs. A comment change is a Delta metadata commit. A writer committing concurrently to the same table can fail with `MetadataChangedException`, a subclass of `ConcurrentModificationException`. A bulk-apply loop at 2 PM on a business day is an incident waiting to happen.

* * *

## 5\. How to Overcome

*   **Ground every description in non-name evidence.** If the bundle contains nothing beyond the column name and type, skip generation and route straight to a human. Name-only generation is pure adhyāsa.
    
*   **Force abstention to be cheap.** `needs_human=true` must be a first-class outcome, tracked and rewarded — not a failure mode. A pipeline that documents 60% of columns correctly is far more valuable than one that documents 100% with 15% wrong.
    
*   **Re-trigger on schema change, not on calendar.** Watch for column type changes, upstream lineage changes, and table-level `ALTER` events in `system.access.audit`; mark affected approved descriptions as `NEEDS_REVALIDATION` rather than leaving them silently stale.
    
*   **Instrument the generator.** Track acceptance rate, mean edit distance, abstention rate and `classification_conflict` rate per schema, per model version. A falling acceptance rate is your earliest signal of prompt drift or a model swap gone wrong.
    
*   **Treat the classification results table as sensitive.** Never select `samples` in any pipeline; keep grants at the default unless a named, reviewed use case requires otherwise; prefer a view that exposes only `class_tag` and `confidence`.
    
*   **Keep inference inside the boundary.** Use a Databricks-hosted or governed serving endpoint for metadata work, and confirm with your security team where that endpoint actually processes data.
    
*   **Test the detector, not just the data.** Seed a non-production catalog with synthetic PII in deliberately awkward shapes (free text, JSON, custom ID formats) and measure what Data Classification finds. Add custom classes where your organisation's identifiers are systematically missed.
    
*   **Budget and schedule the writes.** Cap columns per run, process only new or changed columns, and apply approved comments in a maintenance window — not inside business-hours pipeline schedules.
    

* * *

## 6\. The Takeaway

A catalog is not documentation; it is a claim about reality. Every comment asserts *this is what this column means*, and every tag asserts *this is what this column contains*. AI does not change the nature of those claims — it changes their production rate. That makes the gate, not the generator, the important part of the system.

The pattern that works is the same one Article 4 arrived at for root cause analysis: deterministic machinery assembles the evidence (lineage, profiles, classification), the model drafts within that evidence, and a human witnesses and approves. Use the platform's native classification for *what kind of value*; build an evidence-grounded pipeline for *what it means*; and let the disagreements between the two direct your stewards' limited attention.

Nāma-rūpa is not a flaw to be eliminated — names are how we work. The discipline is remembering that the name is not the clay, and building a catalog that keeps pointing back to what the data actually is. A model that superimposes remembered meanings is useful precisely to the degree that something in the system is still checking whether this rope is a rope.

* * *

*Next in the series — Article 6:* ***Cost & Performance Optimization with AI Assist***\*. Using system tables, query history and an LLM advisor to find the expensive patterns in your Databricks estate — and why an optimization recommendation needs the same evidence discipline as a root cause.\*

* * *

*Karthik Darbha is a Senior Data Engineering & AI Leader with 23 years of professional experience, including 20+ years building enterprise data platforms across Healthcare, Pharma, Retail, Insurance, and Financial Services. He writes about data engineering, program management, and the intersection of technology and philosophy at* [*tech4nirvana.com*](https://tech4nirvana.com/)*.*