Using Databricks system tables for billing observability and platform FinOps
OCT. 28, 2026
6 Min Read
Databricks system tables give platform teams a queryable, account-wide record of what every workload costs, and that record is the strongest foundation you have for platform FinOps.
Most teams still reconcile DBU spend in monthly spreadsheet exports, long after the expensive job has finished. Nearly every practitioner in the State of FinOps 2026 survey- 98% of 1,192 respondents- now manages AI spend, up from 31% two years earlier. A monthly export can't keep pace with that volume of new workloads.
System tables close the gap because they sit inside Unity Catalog as governed tables you query with SQL, so cost observability becomes a data product your team owns. The payoff comes from the queries and dashboards you build on top, and that's where most Databricks cost programs stop short.
Key Takeaways
- 1. Databricks system tables turn DBU spend into governed, queryable data, so platform teams can explain cost changes the next morning instead of at month end.
- 2. A reliable cost model joins billing usage to list prices on the valid price window and labels results as list-price estimates finance can reconcile.
- 3. Tag coverage and billing corrections decide attribution accuracy, so the untagged share belongs on the dashboard as a tracked metric.
What Databricks system tables reveal about platform spend
Databricks system tables are read-only tables in the system catalog that record account activity, including billable usage, SKU prices, compute configuration, job runs and SQL query history. For FinOps, they answer what an invoice can't, which is the workload that consumed the DBUs, who ran it and how its cost is trending.
Consider a platform lead asked why spend in one workspace jumped last month. The invoice shows a total, while the system.billing.usage table shows each usage record with its SKU, workspace, custom tags and, for jobs on job compute, the job ID that produced it. Grouping those records by day and product points to the change within minutes, often a new scheduled job or an idle all-purpose cluster. Records are typically available within 12 hours, which turns a monthly surprise into a next-morning conversation with the owning team.
Which system schemas carry the most FinOps signal
Billing, compute, lakeflow and query schemas carry most of the FinOps signal. Billing tells you what was consumed and at what list price, compute and lakeflow tell you which clusters and jobs did the consuming, and query history explains SQL warehouse spend. Start with billing and add the others as your attribution questions get sharper.
These five tables cover most first-year FinOps work.
- system.billing.usage records every billable usage event with its SKU, quantity, workspace, tags and workload identifiers.
- system.billing.list_prices keeps a historical log of SKU list prices, so you can convert DBUs into dollars for any period.
- system.compute.node_timeline captures minute-level CPU and memory utilization for nodes on classic clusters.
- system.lakeflow.jobs connects job IDs to the job names and owners your stakeholders recognize.
- system.query.history captures statements on SQL warehouses and serverless compute, with duration, bytes read and the user who ran them.
Access comes first. Only users who hold both the account admin and metastore admin roles see system tables by default, so grant USE CATALOG, USE SCHEMA and SELECT to a FinOps group and analysts can query spend without admin rights.
Retention shapes the design. Most system tables keep 365 days of history, and configurable retention, a Beta feature, raises that to 395 days. Year-over-year comparisons need that setting or your own archive table.
How billing tables become a trustworthy cost model
A trustworthy cost model joins system.billing.usage to system.billing.list_prices on the SKU name and on the price window that covers each usage record. Multiplying usage quantity by the effective list price gives an estimated dollar figure per record, which you then roll up by workspace, product, tag or job.
The time window is where hand-built models go wrong. The prices table adds a row each time a SKU price changes, so joining on SKU alone counts a record twice whenever a price changed mid-period. Matching usage against the price valid at that moment keeps last quarter's numbers stable when you rerun them next year.
State one limit upfront. The prices table reflects published list rates, so it won't show negotiated discounts. Most platform teams publish list-price estimates for allocation and trend work, then apply a contract discount factor separately so finance can reconcile to the invoice.
"Matching usage against the price valid at that moment keeps last quarter's numbers stable when you rerun them next year."
Queries that attribute DBU spend to the workloads behind it
Attribution queries group usage records by the identifiers stored with each record. Job ID, warehouse ID and cluster ID tie each DBU to a workload, and the run_as field names the identity that ran it. Joining those IDs to the lakeflow and compute tables adds the job names and owners your stakeholders recognize.
Picture a monthly report of the 20 most expensive jobs. The query filters usage to the jobs billing origin product, sums quantity times price by job ID, then joins the jobs table for names. That table keeps a new row for every change, so the query takes the latest row per workspace and job ID, or each renamed job appears twice.
Coverage has gaps. Job IDs are populated only for jobs on job compute and serverless compute, so work scheduled on all-purpose clusters can't be pinned to a single job with precision. On all-purpose compute, run_as identifies whoever created the cluster, so a join to system.compute.clusters gives a cleaner owner field.
Shared SQL warehouses need query history instead, which records each statement's duration, bytes read and author so you can rank the heaviest queries.
| FinOps question | Where the answer comes from | What the platform team gets |
|---|---|---|
| Which products drove this month's increase? | Usage records grouped by billing origin product and day | A trend line that isolates the product and date behind a spike |
| What does each scheduled job cost? | Usage joined to list prices and the latest job record | A ranked list of jobs with names and estimated dollars |
| Who owns oversized classic clusters? | Usage joined to cluster configuration and node utilization | Named owners for compute that runs with low CPU use |
| Which teams consume shared SQL warehouses? | Query history grouped by user and warehouse | A defensible basis for splitting warehouse cost across teams |
| How much spend has no owner? | Usage records with empty custom tags | An untagged share you can push down month over month |
Building cost dashboards finance partners will actually trust
Finance trusts a cost dashboard when its numbers reconcile, its definitions stay fixed and every figure traces back to a query anyone can rerun. Databricks recommends AI/BI dashboards on system table billing data, and account admins can import a prebuilt usage dashboard from the account console as a starting point.
Treat the import as scaffolding. Its default views break usage down by product, SKU and custom tag, and version 2.0, in Preview, adds cost forecasting. Platform teams then add cost per business unit, cost per pipeline run and the untagged share, and pair them with budgets that email owners at up to four monthly thresholds. Lumenalta®'s data platform observability practice builds FinOps and platform health dashboards on system tables, keeping queries and metric definitions in version control so a definition change gets reviewed like code. Reporting lines support this placement, since 78% of FinOps teams now report to the CTO or CIO, the same leaders who own the platform roadmap.
"Finance trusts a cost dashboard when its numbers reconcile, its definitions stay fixed and every figure traces back to a query anyone can rerun."
Where missing tags distort Databricks cost attribution
Untagged usage is the largest source of error in Databricks cost attribution. Usage records carry the custom tags applied to the compute behind them, so any workload launched without team or cost center tags lands in an unallocated bucket that someone argues about at month-end.
A familiar pattern shows up in the first untagged spend report. Classic clusters created through compute policies carry tags, while ad hoc serverless notebooks carry none. Serverless usage policies, in Public Preview, close part of that hole by applying tags to serverless notebooks, jobs, pipelines and serving endpoints for assigned users. They don't cover classic compute, which still relies on compute policies.
Billing corrections are the second distortion. The usage table records a correction as a retraction that cancels the original and a restatement with the corrected quantity. Queries that keep only original records will overstate spend whenever a correction posts, so sum across all record types and drop groups that net to zero.
Making system tables the operating rhythm for platform FinOps
System tables pay off when they set a weekly rhythm. Platform and FinOps leads review the same governed queries each week, owners see their costs before finance asks, and tag coverage becomes a tracked metric. The tables already hold the record, and discipline in how you query, publish and act on it decides the result.
Teams that get this right resist building everything at once. They start with list-price estimates by product, add job and warehouse attribution, then publish the untagged share and push it down monthly. Each layer earns trust before the next arrives, and finance stops asking for the spreadsheet. That record matters more as agents start running workloads. Databricks describes the platform as a governed system of record for agents, and system tables are where that record shows what each workload cost and who ran it.
Lumenalta® approaches platform FinOps this way, designing the observability layer on Databricks while the client owns the keys, so queries, dashboards and tag rules stay client-controlled and portable. Over a few quarters, that practice gives you a cost record your engineers trust as much as your controller does.
Table of contents
- What Databricks system tables reveal about platform spend
- Which system schemas carry the most FinOps signal
- How billing tables become a trustworthy cost model
- Queries that attribute DBU spend to the workloads behind it
- Building cost dashboards finance partners will actually trust
- Where missing tags distort Databricks cost attribution
- Making system tables the operating rhythm for platform FinOps
See how system table cost observability lowers cost and improves data agility.