About the job
Lead Principal Azure Data Engineer (Databricks)
Location: London (Hybrid) | Practice Area: Technology & Engineering | Type: Permanent
Shape the Future of Financial Services with Data Innovation
The Role
As a Lead Principal Azure Data Engineer specializing in Databricks at Capco, you will spearhead the architectural design and implementation of cutting-edge enterprise data solutions within the Azure ecosystem. Your expertise will drive the creation of both streaming and batch data pipelines, enabling our clients to advance their data capabilities. Collaborating with diverse teams, you will provide strategic guidance on best practices and technical approaches, ensuring adherence to the highest quality and security standards throughout all deployments.
What You’ll Do
Lead comprehensive delivery of Databricks-driven solutions within Azure environments.
Design secure, scalable data pipelines utilizing DeltaLake, Spark Structured Streaming, and Unity Catalog.
Establish and uphold engineering best practices across the data lifecycle.
Collaborate with clients to design solution strategies and robust data governance frameworks.
Mentor engineering teams and actively participate in internal capability enhancement programs.
What We’re Looking For
Demonstrated proficiency in the Databricks platform, including Unity Catalog, DeltaLake, and orchestration.
Advanced skills in Python, PySpark, and distributed data processing frameworks.
Extensive experience developing CI/CD pipelines with tools such as Azure DevOps, Jenkins, and GitHub Actions.
Thorough understanding of data lakehouse concepts, data modeling, and GDPR-compliant design methodologies.
Proven track record in creating robust, production-grade data pipelines from ingestion to serving.
Bonus Points For
Strong client-facing and commercial acumen to support pre-sales and RFP engagements.
Experience in coaching and mentoring engineering teams.
Development skills in Scala or Java.
Familiarity with handling PII and sensitive data in compliance with regulations.

