Top 5 System Table Queries for Understanding Your Databricks Costs

Databricks system tables provide a wealth of information into how you use Databricks and what it costs. Focusing in on cost, the `system.billing.usage` table provides a globally aggregated view on costs for your entire Databricks account and it along with the `system.billing.list_prices` can give you deep insight into where spend is being allocated. Databricks offers a prebuilt Usage Dashboard that provides an excellent starting point for understanding costs but understanding the underlying tables and building queries on them allows you take your insights to the next level, especially when leveraging the visualization capabilities of Databricks.

Below are five queries to use as a starting point in better understanding your spend. We start with a simple daily breakdown of spend by product, then progress to understanding spend through the lens of specific SQL Warehouses and Tags, and finally identify days with anomalous spend as well as forecasting future spend with AI. All of the queries provide insight via the returned results but are even more effective when used as datasets in an AI/BI dashboard where the data can be visualized and further interrogated with Genie.

Our first query is simply just a breakdown of spend by day and by product. We join the usage table (our fact) with our pricing table (an SCD Type 2 dimension) to get our spend over time at list pricing. As mentioned above, you could substitute the provided list price table with a custom table that incorporates any relevant discounts to see exact costs.