Senior Databricks Engineer / Ingestion Lead

About the Role

We partner with global enterprises to design and build large-scale data platforms that power the products and operations at the center of their business. This engagement is a Databricks lakehouse foundation for a US defense manufacturer, and we are looking for a Senior Databricks Engineer to own the ingestion layer.

The sources are the job. Change data capture out of a PLM system. JDBC ingestion against an ERP where session-level initialization parameters decide whether row-level security hands you the right rows or quietly the wrong ones. Bill-of-materials reconciliation across three systems of record that each describe the same part differently, where someone has to decide which is right.

This is a hands-on senior role with real ownership. Two people on the engagement means ingestion is yours end-to-end: you make the pattern calls per source and sit in the client discovery sessions rather than receiving tickets from them, working alongside the engagement architect who owns platform direction and governance.

The engagement opens with a discovery and design phase, where you go deep on each source system and design the right ingestion pattern for it. It then moves into the build, executing what you designed.

Actively hiring

We are hiring for a current opening on an active client project. This is a specific, presently open role. We review applications on a rolling basis and aim to move qualified candidates through our process promptly.

What You'll Do

• Run source discovery across each system in scope, documenting data structures, change patterns, extraction constraints, data quality, and current-state integration topology.
• Assess extraction options per source, including change data capture feasibility from PLM, JDBC access to Oracle E-Business Suite under its row-level security model, SQL Server change patterns, and file-based ingestion from S3.
• Design the ingestion pattern for each source (Lakeflow Connect, custom JDBC, Auto Loader, or federation) with the tradeoffs written down and defensible.
• Model bill-of-materials data across PLM, ERP, and MBSE sources, keyed on part number plus revision, including how conflicting representations of the same part are reconciled into the bronze and silver layers.
• Build the ingestion layer: Lakeflow Connect pipelines including change data capture from Teamcenter PLM, custom JDBC connectors against Oracle E-Business Suite with session-level initialization parameter handling, Auto Loader for incremental S3 ingestion, and SQL Server ingestion patterns.
• Handle schema evolution deliberately, with automatic handling of new columns, rescued data columns for unexpected shapes, and controlled failure on renamed columns rather than silent data loss.
• Operate nightly batch patterns with idempotent reruns and backfill support, quarantining bad source data rather than dropping it.
• Apply strong engineering practices including testing, CI/CD, and version control, and write expectations and quality checks into pipelines so failures are attributable to a source and a rule.
• Document for handoff, producing each source's ingestion contract and runbook so the client team can own it.

What We're Looking For

• 7+ years of production data engineering on Databricks, with strong Python, PySpark, and SQL. You are fluent in Delta Lake, Auto Loader, and declarative pipeline development.
• Hands-on experience with Lakeflow Connect or equivalent managed change data capture ingestion.
• Direct experience building JDBC-based ingestion against Oracle, ideally Oracle E-Business Suite, and familiarity with its security model, session context, and multi-org data structures.
• Experience with SQL Server as an ingestion source.
• Working fluency in manufacturing PLM and MBSE systems: Teamcenter, Cameo, and Siemens Capital. You do not need to be an administrator, but you need to understand their data models well enough to design extraction correctly.
• Experience modeling bill-of-materials data, including revision handling, effectivity, and multi-source reconciliation.
• Solid experience with Git-based workflows, CI/CD pipelines, and testing frameworks such as PyTest.
• Strong AWS experience, particularly S3 and IAM roles.
• Databricks Data Engineer Professional certification.
• Experience in a defense, aerospace, or otherwise regulated data environment.
• Ability to do discovery work: sit with a source system owner, ask the right questions, and document accurately what you find.
• A track record of building pipelines that survive contact with messy real-world sources and that other engineers can maintain.
• US citizenship and ability to pass a background check.
• Fluent English, both written and spoken, with strong collaboration skills in distributed teams.

Nice to Have

• Prior Teamcenter or other PLM database extraction experience, including direct schema knowledge.
• Familiarity with Unity Catalog governance from the producer side: tagging, documentation, and lineage-friendly pipeline design.
• Experience with Infrastructure as Code using Terraform, and with Databricks Asset Bundles.
• Experience integrating AI-native tooling into engineering workflows.

Why Lumenalta is an amazing place to work

At Lumenalta, you can expect that you will:

• Be 100% dedicated to one project at a time so that you can innovate and grow.
• Be a part of a team of talented and friendly senior-level developers.
• Work on projects that allow you to use leading tech. We have built for Bloomberg, Target, and Two Sigma, and data platform work for Redwood Logistics and Arbol.

For Databricks practitioners specifically, we are a Databricks partner with Champions and MVPs on our team, and that carries three concrete things:

• Delivery Partner Program qualification. Databricks co-delivers its Professional Services engagements through a short list of approved partner organizations, and practitioners have to be individually qualified before they can work on them. The qualification is granted once at the individual level and does not expire, so it stays with you. Partner organizations are the route in, so the qualified pool stays small. That scarcity is most of the value, and we sponsor it.
• Champion path. Databricks Champion is an individual recognition with its own badge, a dedicated Databricks solutions architect as your mentor, access to product teams, confidential roadmap briefings, and an invitation to the internal Tech Summit. Nomination has to come from a partner organization, which puts it out of reach for independents and most contractors. We nominate actively and regularly, and engineers are nominated here as readily as architects: someone with hard-won extraction knowledge in systems most people never touch has more to contribute to that community than most.
• Certifications covered. We offer the exam fee for any Databricks certification, as we are official partners.

Our Process

A screening call, a technical interview, and a HackerRank assessment.

The technical interview is a conversation rather than a whiteboard exam. We will ask you to walk through an ingestion pipeline you built against a difficult source, how you handled schema drift and bad data, and how you would approach discovery and change data capture design for a source system nobody has documented. Relevant project references and specific source system experience carry more weight with us than certifications.

Location

This is a fully remote position. This position supports a U.S. Government contract that requires all personnel to possess U.S. Citizenship. Availability to work overlapping Pacific, Central, or Eastern U.S. time zones.

Application Deadline

Applications will be accepted until October 4th, 2026. Candidates can expect feedback by October 12th, 2026.