Logistics leader supercharges data processing with Azure Databricks, cutting costs by 25%

Our client, one of the largest less-than-truckload (LTL) carriers in North America—specializing in TL and LTL linehaul, intermodal drayage, pool distribution, and temperature-controlled logistics— needed to modernize a legacy data infrastructure that was limiting visibility, margin performance, and scalability across the enterprise.

Prior to implementing Databricks, the organization faced significant operational constraints. Their entire data estate ran on AS/400 DB2 servers and Oracle databases requiring VPN access, with no centralized cloud data platform for analytics or AI. Siloed legacy systems limited cross-business visibility and slowed reporting—multiple teams relied on a brittle process of running queries against on-prem databases, exporting results to Excel, manually reformatting data, and emailing spreadsheets to stakeholders, creating delays, inconsistencies, and duplicated effort.

Their pricing operation was entirely manual—a team of approximately 15 people pricing routes in Excel without competitive intelligence, historical pattern analysis, or automated optimization, leading to consistent underpricing and significant margin erosion at scale. Customers had zero digital visibility into shipments; checking the status of a load required calling customer service directly. There was no self-service tracking, quoting, booking, or invoicing capability.

Latency issues and concurrent workloads reduced data processing speed. On-premises databases created scalability barriers during high-volume periods, and growing data volumes from fleet operations, terminal check-ins, and acquisitions overwhelmed the legacy environment. Integrating data from newly acquired companies required lengthy manual migration projects, slowing time-to-value from M&A activity. Without a unified, governed data platform, the company had no pathway to adopt machine learning, streaming analytics, or AI-driven operational intelligence at scale.

Lumenalta partnered with the company to design, build, and deploy a full Azure Databricks Lakehouse Platform as the unified foundation for data engineering, governance, analytics, and operational applications.

All data ingestion and transformation pipelines were standardized on Delta Lake, delivering ACID transactions, schema enforcement, and time travel—eliminating data silos across legacy systems and creating a governed single source of truth.

All ingestion and transformation pipelines were developed and orchestrated through Databricks Notebooks, providing standardized, repeatable pipeline patterns across the organization. Data flows through a structured Medallion Architecture (Bronze/Silver/Gold): raw data lands in the Bronze layer via Autoloader, is cleansed and deduplicated in the Silver layer through Delta Stream processing, and is shaped into consumption-ready datasets in the Gold layer with applied business rules. This standardized, repeatable pipeline pattern ensures maintainability and accelerates onboarding of new data sources.

Unity Catalog was implemented as the enterprise governance layer, providing fine-grained role-based access controls, full data lineage tracking, and centralized auditing across all workspaces—delivering the security and compliance posture the organization previously lacked entirely.

Databricks SQL Serverless enables analysts and business users to access curated Delta tables directly for ad hoc analysis, dashboards, and self-service reporting without engineering dependencies or manual Excel workflows.

A dedicated Speed Layer processes real-time operational data through Azure EventHubs and Databricks Structured Streaming, powering live manifest dashboards that give operations teams real-time visibility into truck movements, terminal check-ins, and load status across the network.

The pricing operation was transformed from a 15-person manual process into a single-click automated system. Data from Oracle and DB2 source systems is ingested through the Databricks Lakehouse, optimized through Azure Pipelines, and delivered to a PostgreSQL application layer that incorporates competitive data, historical pricing patterns, weight, destination, and shipping type into automated pricing calculations.

Delta Sharing enables secure, governed sharing of curated datasets across business units and with external partners without data duplication or additional infrastructure.

The curated Gold layer is replicated to PostgreSQL for application-ready access, powering a customer self-service portal (real-time shipment tracking, quoting, booking, and invoicing) and driver-facing tools—replacing the previous phone-based customer service model with a 360° unified customer platform across five business lines.

Lumenalta implemented a lift-and-shift approach to accelerate delivery, establishing a secure network bridge between on-premises systems and the Azure cloud. Wrapper services and next-generation APIs allowed existing systems to continue running seamlessly while the new Databricks infrastructure scaled in parallel.

The company's adoption of the Azure Databricks Lakehouse Platform fundamentally transformed their data operations, customer experience, and margin performance. The platform reached MVP in 6 months from kickoff and continues to deliver compounding value:

6 Months to MVP launch: From zero cloud data platform to a fully operational Databricks Lakehouse processing production workloads— including governance, streaming, and application delivery—in six months.

25% Lower compute costs year-over-year: They achieved a 25% reduction in DBU usage costs year-over-year through optimized pipeline patterns and efficient resource utilization, demonstrating increasing value per dollar spent on Databricks.

85% Reduction in hardware costs: Infrastructure costs dropped from $500 to $75 per unit, significantly lowering rollout expenses across the network.

Billions of rows processed daily: The lakehouse handles billions of rows daily across batch and streaming workloads without performance degradation, powered by 5TB of analytics and 500 pipelines supporting high-volume data processing.

Millions in weekly revenue through digital portals: The customer and driver portals, powered by Databricks Lakehouse data, now drive millions in weekly revenue, replacing the previous manual, phone-based customer service model with self-service digital experiences.

2.5 Million shipments tracked annually in real time: The streaming speed layer delivers live manifest dashboards showing truck movements, terminal operations, and load status in real time— capabilities that previously did not exist in any form.

360° Unified customer platform across five business lines: Enabling cross-sell and upsell opportunities alongside dramatically improved customer experience.

4x User adoption: Active platform users grew from 2,000 to 8,000, reflecting enterprise-wide adoption of the new Databricks-powered analytics environment.

Pricing operation transformed (15 people → single click): The automated pricing engine eliminated manual Excel-based pricing, enabling data-driven rate optimization that directly addresses the margin erosion from years of underpricing. This single capability directly impacts revenue and profitability at scale.

Instant acquisition data integration: Every time the company acquires a new company, that company's data can be instantly integrated into the governed lakehouse, eliminating lengthy manual migration projects that previously delayed M&A time-to-value.

Zero to six Databricks products in production: They went from no Databricks products to six in active production use: Azure Databricks (Spark runtime), Databricks Notebooks, Delta Lake, Unity Catalog, Databricks SQL Serverless, and Delta Sharing—with natural expansion pathways into Mosaic AI for predictive analytics, demand forecasting, and fleet optimization.