Building audit-ready regulatory reporting on Databricks with Unity Catalog
OCT. 8, 2026
6 Min Read
Regulatory reporting becomes audit-ready on Databricks when Unity Catalog and Delta Lake record lineage, access and change history as pipelines run, so the evidence exists before anyone asks for it.
Most banks still prove their numbers the hard way. A quarter-end scramble traces Basel capital figures, CCAR projections and AML alert counts back to source systems through spreadsheets and tribal knowledge, and supervisors keep finding that reconstruction wanting.
Of the 31 global systemically important banks the Basel Committee assessed, only 2 were fully compliant with its risk data aggregation and reporting principles. The durable fix is capturing lineage as pipelines run and treating the platform's own metadata as your audit trail.
Key Takeaways
- 1. Lineage recorded by the platform as each transformation runs gives regulators stronger evidence than lineage reconstructed at quarter end.
- 2. Unity Catalog column lineage, access policies, pipeline expectations and audit tables only produce a full chain of custody when every feed is registered and referenced by table name.
- 3. Default retention on lineage, audit and table history falls short of multi-year lookbacks, so banks need scheduled copies and dated submission snapshots.
What Unity Catalog gives a regulatory reporting team
Unity Catalog gives your reporting team one governance layer for every table, view, model and dashboard that feeds a submission. It controls who can read each dataset, logs access events to an audit system table and records lineage automatically for queries that run on Databricks, so the evidence and the data share one source.
Consider a capital adequacy team producing FR Y-9C schedules and Pillar 3 disclosures from the same exposure tables. Once those tables are registered under a three-level name such as regulatory.credit_risk.exposures, each object carries its access grants, its classification tags and its upstream history. An examiner asking who changed risk-weighted asset inputs last quarter gets a query result from table history and the audit log instead of an email chain.
Reporting failures rarely start in the reporting layer. They start when a source feed shifts and nobody downstream knows. A catalog won't stop upstream changes, but it makes their impact visible to the people who sign the submission.
"Supervisors give more weight to evidence produced as a byproduct of daily operations than to documentation assembled for the review itself."
Why lineage captured at runtime holds up better under audit
Lineage captured at runtime holds up because the platform records it as each transformation runs, while reconciled lineage is rebuilt afterward by people who need it to tie out. Supervisors give more weight to evidence produced as a byproduct of daily operations than to documentation assembled for the review itself.
Picture an AML team asked why suspicious activity alert volumes dropped sharply between two quarters. With runtime lineage, the analyst opens the alert table's lineage graph, sees that a transaction-screening job began reading from a new card-payments table mid-quarter and confirms the change against the dated input snapshots kept for each submission in an afternoon. Without it, the same question becomes a two-week hunt through job schedulers and old notebooks. The tradeoff is upfront discipline. Pipelines have to run on governed compute, and teams lose the freedom to script one-off extracts on unmanaged clusters. You pay that cost once, while the reconciliation cost returns every quarter.
How column-level lineage answers a regulator's traceability question
Column-level lineage answers the question regulators care about most, which is where a reported number came from. Unity Catalog traces each column in a reporting table back through the notebooks, jobs and pipelines that produced it, and it aggregates that history across every workspace on the same metastore.
Take a single CCAR line item such as projected net charge-offs for commercial real estate. A reviewer can follow that column from the final submission table to the loss model output, then to loan-level balances, then to the core banking extract that landed in a bronze table. Each hop shows the job or notebook that wrote it, which is the chain of custody an examiner wants to see.
Settle two technical conditions before you promise this to auditors. Databricks captures column lineage only when both the source and the target are referenced by table name, so pipelines that read raw file paths lose column detail. Renamed tables and columns also lose their lineage, so a casual rename mid-quarter can break the trail you'll need at year end.
Treat naming as a governed choice. Register landing data as tables before any transformation, freeze reporting schema names during submission windows and route renames through change control so the trail stays intact.
Which controls turn a Databricks pipeline into audit evidence
Six platform controls turn an ordinary Databricks pipeline into audit evidence. Each one records something an examiner will ask about, from who touched the data to which checks it passed, and each writes that record into tables you can query. These are the controls worth configuring first.
- Unity Catalog privileges granted at catalog, schema or table level, which set who can read or modify each reporting dataset
- Attribute-based access control (ABAC) policies on governed tags, which table owners can’t override when set at catalog or schema level
- Row filters and column masks that restrict sensitive customer fields at query time
- Pipeline expectations that warn, drop or fail records breaking a data quality rule, with results logged to the pipeline event log
- Delta table history, which lists every write with its operation, user and timestamp
- The system.access.audit table, which records access and permission events for the workspaces in a region
A liquidity team filing the FR 2052a shows how these fit. An expectation fails the run when a counterparty identifier is null, so a bad file never reaches the report. The audit table shows which analyst reran the job, and the table history shows the corrected version that was published.
None of these controls is exotic. The hard part is configuring them consistently across pipelines owned by different teams. That's the governance work Lumenalta®does with Unity Catalog, where its teams lead architecture and delivery design while the bank keeps the keys, so the controls stay client-controlled and portable.
How long you need to keep lineage and audit history
Keep lineage and audit history for as long as your longest regulatory lookback, which for most bank submissions runs well past a year. Databricks system tables for lineage and audit events hold 365 days by default, and Delta table history defaults to 30 days, so the defaults won't cover a multi-year examination on their own.
A bank asked to explain a capital figure filed six quarters earlier will find the system tables have already rolled past it. System tables are read-only and roll off after 365 days, so the standard pattern is a scheduled job that copies lineage and audit rows into governed tables you own, plus a dated snapshot of each submission's inputs. Table history isn't designed as a long-term backup, and raising retention adds storage cost, so snapshots beat stretching time travel. Old files are not protected just because nobody runs VACUUM, since predictive optimization runs it automatically on Unity Catalog managed tables and is on by default for most accounts.
| Evidence you'll be asked for | Where Databricks keeps it | Default retention | What to do for a multi-year lookback |
|---|---|---|---|
| Table-level lineage | The system.access.table_lineage system table | 365 days on a rolling basis | Copy rows into a governed history table on a schedule |
| Column-level lineage | The system.access.column_lineage system table | 365 days on a rolling basis | Copy it in the same scheduled job |
| Access and permission events | The system.access.audit system table | 365 days | Stream events into a retained audit table you own |
| Who changed a reporting table and when | The Delta transaction log behind each table | 30 days unless you raise it | Raise log retention on reporting tables only |
| The exact data behind a filed figure | Older Delta data files | 7 days before VACUUM can remove them | Write each submission's inputs to a dated snapshot table |
Common gaps that still fail a regulatory data review
The gaps that still fail reviews are organizational more than technical. Banks deploy a catalog but leave critical feeds outside it, name data owners who lack the access to act and monitor pipeline health without linking quality issues to the reports they touch.
Regulators have put a price on that last gap. One large US bank drew a $75 million civil money penalty in 2024, with the OCC citing a lack of processes to monitor how data quality concerns affect regulatory reporting.
Lineage closes that gap when you read it downstream. When an expectation drops records from a loan-balance table, an impact query against the lineage tables lists every report that reads from it, so the reporting lead knows an FR Y-14 schedule is exposed before it's filed.
Shadow adjustments are the other recurring miss. A treasury analyst's spreadsheet that tweaks a figure after the governed pipeline finishes breaks lineage at the last step. Move manual adjustments into a governed table with a reason code and an approver, and the exception becomes part of the evidence.
What audit readiness looks like after the first submission cycle
After the first submission cycle, audit readiness looks quiet. Examiner questions get answered from lineage and audit queries within hours, quarter-end reconciliation shrinks to real exceptions and your team spends its time judging the numbers instead of proving their origin.
"Move manual adjustments into a governed table with a reason code and an approver, and the exception becomes part of the evidence."
That quiet comes from dull choices repeated every cycle. None of it is glamorous, and each choice compounds. The second cycle is easier than the first, and by the fourth, the evidence is simply there when someone asks.
Banks that reach this point treat Unity Catalog as part of their reporting control framework. Lumenalta® approaches Databricks governance the same way, building Unity Catalog controls into delivery design from the first pipeline so they hold as reporting scope grows. You're left with a reporting process where proof is a byproduct of doing the work correctly.
Table of contents
- What Unity Catalog gives a regulatory reporting team
- Why lineage captured at runtime holds up better under audit
- How column-level lineage answers a regulator's traceability question
- Which controls turn a Databricks pipeline into audit evidence
- How long you need to keep lineage and audit history
- Common gaps that still fail a regulatory data review
- What audit readiness looks like after the first submission cycle
