North American asset manager streamlines data management with a modern lakehouse on Databricks and AWS

This privately held investment management firm specializes in a comprehensive suite of investment products, risk management solutions, and advisory services. Managing funds for investors and institutions worldwide, the company employs associates globally.

Before adopting Databricks, the organization faced significant data-management limitations that constrained agility and slowed innovation. Multiple teams relied on disparate tools—including pandas notebooks, on-prem compute, and ad-hoc scripts—which resulted in duplicated efforts and inconsistent workflows. Legacy systems struggled to handle increasing data volumes across environments such as S3, PostgreSQL, Snowflake, and other legacy databases.

Development teams lacked a centralized platform for experimentation, spending more time managing infrastructure than producing insight. Analysts, data scientists, and engineers operated in separate environments without a unified workspace, delaying cross-functional collaboration and reducing productivity. The on-prem and cloud-based hybrid setup also increased maintenance costs and deployment complexity, making it difficult to scale analytics or introduce advanced capabilities such as streaming, machine learning, or AI at an enterprise level.

The firm needed a unified data foundation capable of supporting real-time analytics, AI-driven insights, and robust governance for its investment and research ecosystem.

The firm selected Lumenalta to design a phased modernization strategy that would migrate legacy pipelines and analytics workloads to the Databricks Lakehouse Platform on AWS. The engagement began in August 2024 with an initial proof of concept and has since expanded into an ongoing enterprise adoption effort.

This modernization initiative aimed to unify disparate data pipelines within a single Lakehouse architecture, streamline analytics workflows using Databricks SQL and Delta Live Tables, and strengthen data governance and compliance through Unity Catalog and automated testing frameworks. The approach balanced innovation with operational stability, ensuring that migration efforts did not disrupt existing business processes while laying the groundwork for long-term scalability.

Phase 2 – Enterprise Rollout of two lines of business (LOBs):
Following POC success, two key lines of business were migrated:

The adoption of the Databricks Lakehouse Platform has streamlined operations, enhanced collaboration, and established a scalable foundation for AI-enabled investment insights. By consolidating pipelines from R scripts, SQL Server ETL, AWS Glue jobs, and Jupyter notebooks into one Lakehouse environment, teams can now manage data more efficiently and accelerate reporting cycles. The unified workspace enables analysts, engineers, and data scientists to collaborate in real time, improving speed and consistency of analysis across the enterprise.

The platform has also simplified infrastructure and reduced maintenance overhead through cloud-native scalability, while Databricks SQL and Delta Live Tables have delivered faster, more reliable analytics. Unity Catalog has strengthened governance, ensuring centralized control over permissions, lineage, and auditing—an essential capability for meeting strict regulatory requirements.

This modernization initiative has positioned the firm to expand into advanced analytics and AI, with early exploration of Mosaic AI tools preparing the organization for future use cases in predictive modeling and portfolio optimization. The partnership continues to evolve as the company scales its Lakehouse capabilities across the enterprise.