Senior Azure Data Engineer (Databricks)

Lead Principal Azure Data Engineer (Databricks)Location: London (Hybrid) | Practice Area: Technology & Engineering | Type: PermanentShape the Future of Financial Services with Data InnovationThe RoleAs a Lead Principal Azure Data Engineer specializing in Databricks at Capco, you will spearhead the architectural design and implementation of cutting-edge enterprise data solutions within the Azure ecosystem. Your expertise will drive the creation of both streaming and batch data pipelines, enabling our clients to advance their data capabilities. Collaborating with diverse teams, you will provide strategic guidance on best practices and technical approaches, ensuring adherence to the highest quality and security standards throughout all deployments.What You’ll DoLead comprehensive delivery of Databricks-driven solutions within Azure environments.Design secure, scalable data pipelines utilizing DeltaLake, Spark Structured Streaming, and Unity Catalog.Establish and uphold engineering best practices across the data lifecycle.Collaborate with clients to design solution strategies and robust data governance frameworks.Mentor engineering teams and actively participate in internal capability enhancement programs.What We’re Looking ForDemonstrated proficiency in the Databricks platform, including Unity Catalog, DeltaLake, and orchestration.Advanced skills in Python, PySpark, and distributed data processing frameworks.Extensive experience developing CI/CD pipelines with tools such as Azure DevOps, Jenkins, and GitHub Actions.Thorough understanding of data lakehouse concepts, data modeling, and GDPR-compliant design methodologies.Proven track record in creating robust, production-grade data pipelines from ingestion to serving.Bonus Points ForStrong client-facing and commercial acumen to support pre-sales and RFP engagements.Experience in coaching and mentoring engineering teams.Development skills in Scala or Java.Familiarity with handling PII and sensitive data in compliance with regulations.

AI Engineer - Forward Deployed Engineer (All Levels) Req ID: CSQ426R186 Our Mission The AI Forward Deployed Engineering (AI FDE) team at Databricks is a specialized, customer-centric group focused on delivering exceptional AI solutions. We engage with our clients to develop and implement innovative AI applications that are cutting-edge and tailored to their unique needs. Our collaborative environment allows us to work closely with engineering, product management, and developer relations, as well as support our internal subject matter experts. We seek passionate individuals who thrive on customer interaction, teamwork, and a desire to stay ahead of the latest trends in Generative AI, LLMOps, and machine learning. Your Impact Design and develop groundbreaking Generative AI solutions utilizing the latest advancements from our Mosaic AI research to address client challenges. Lead the deployment of consumer-facing and internal Generative AI applications into production. Act as a trusted technical consultant to clients across various industries. Represent Databricks as a thought leader at industry events such as the Data + AI Summit. Work collaboratively with product and engineering teams to shape strategic priorities and influence the product roadmap. What We Value in You Proven experience in developing Generative AI applications, including retrieval-augmented generation (RAG), multi-agent systems, Text2SQL, and fine-tuning, with tools such as HuggingFace, LangChain, and DSPy. Strong background in deploying production-grade Generative AI applications, including evaluation and optimization techniques. Several years of hands-on experience in data science, utilizing common tools such as pandas, scikit-learn, and PyTorch. Experience in building scalable machine learning deployments on cloud platforms like AWS, Azure, or GCP. A graduate degree in a quantitative field (Computer Science, Engineering, Statistics, Operations Research, etc.) or equivalent practical experience. Ability to communicate and teach technical concepts effectively to both technical and non-technical audiences. A genuine passion for collaboration, continuous learning, and leveraging AI to drive business success. [Preferred] Familiarity with the Databricks Intelligence Platform and Apache Spark™ for processing large-scale distributed datasets.

At Databricks, we're committed to empowering data teams to tackle some of the world's most challenging issues, including security threat detection and cancer drug development. Our mission is to create and manage the premier data and AI infrastructure platform so our customers can dedicate their efforts to the high-value challenges crucial to their missions. Our engineering teams develop cutting-edge products that meet significant real-world needs. We operate one of the largest-scale software platforms, comprising millions of virtual machines that generate terabytes of logs and process exabytes of data daily. Given our scale, we frequently encounter hardware, network, and operating system faults, and our software must effectively shield our customers from these challenges. The Delta DML team is responsible for the core write-path operations for Delta Lake, the open-source storage layer underlying the Databricks Lakehouse. Our goal is to deliver industry-leading performance and a seamless user experience at massive scale, with most data written in Databricks passing through our platform. We innovate in performance through features like Low Shuffle Merge and Deletion Vectors, and actively contribute to open-source initiatives that unify Delta and Iceberg formats. We are on the lookout for a highly skilled and experienced Senior Staff Software Engineer to join our backend team. In this pivotal role, you will design, develop, and maintain robust backend systems that power Databricks workspaces. You will work on building the next-generation platform for serving workspace assets, ensuring high queries per second (QPS), low latency, reliability, and performance while proactively addressing future growth challenges. As a senior member of the team, your responsibilities will also include providing technical leadership, mentorship, and guidance to junior engineers, thereby enhancing the overall coding practices and system designs of the team.

Join our innovative data engineering team as a Senior Data Engineer! We are looking for a talented professional to design, develop, and optimize data pipelines and workflows utilizing Databricks and AWS. You will play a pivotal role in enhancing our data processing capabilities while ensuring data quality, security, and compliance.Key Responsibilities:Provide L2 and L3 support within Data Engineering to troubleshoot and resolve production pipeline issues.Create and sustain high-performance data pipelines using Databricks.Collaborate with architects and analysts to construct data models and transformations.Ensure data quality, security, and compliance in Databricks environments.Implement monitoring and logging strategies for data pipeline reliability and performance.Deploy and manage code through CI/CD workflows.Efficiently troubleshoot and resolve complex data processing challenges.

REQ ID: FEQ227R147Location: London, UKRecruiter: Dina HussainAt Databricks, our fundamental values shape our operations, fostering a culture of proactivity and a client-focused approach. We are dedicated to creating a cohesive platform that democratizes access to data and AI, empowering our customers to make informed decisions that drive their businesses forward.In the UK&I Digital Natives Hunting territory, we operate at the pace of the world’s most innovative companies. We don’t merely sell software; we collaborate with technical founders, CTOs, and VPs of Engineering at rapidly growing startups to shape the future of AI. You will play a pivotal role in this mission, leveraging your executive-level technical credibility and a 'Hunter' mentality to gain mindshare and secure new business opportunities. Join us in transforming the way people interact with data and contribute to building a better world!Reporting to Manager, Field EngineeringYour Impact:Executive Evangelism & Thought Leadership: Serve as a Technical Advisor, establishing peer-to-peer relationships with CTOs, VPs of Engineering, and technical founders at high-growth DNBs. You will secure technical mindshare in initial meetings by articulating a vision for Data + AI that resonates with their rapid innovation cycles.Drive New Logo Motion: Execute the high-velocity 'Hunter DNA'—prioritizing Time to Value, qualifying out rigorously when necessary, and concentrating efforts on challenging, high-prestige accounts that position Databricks as the benchmark for the UK startup ecosystem.Ecosystem Building: Foster awareness within the UK startup and VC ecosystem. You will represent Databricks at key events, cultivating strategic relationships with VCs, accelerators, and incubators to identify and influence the next generation of 'unicorns.'Strategic Technical Discovery: Transition from traditional 'hands-on' SA work to focus on high-level architecture and strategic road-mapping. You will connect complex business requirements with technical solutions, ensuring that the Databricks Intelligence platform is recognized as a strategic enabler of a startup's core product.Regional Mentorship: Act as a role model and mentor for the broader Field Engineering team, especially guiding Solutions Architects on effectively engaging with CxOs and navigating the unique challenges of the digital-native segment.

Join Databricks as an Engineering Manager specializing in Storage. In this pivotal role, you will lead a talented team of engineers to develop and enhance our innovative storage solutions that empower data-driven organizations. Your leadership will not only drive technical excellence but also foster a culture of collaboration and continuous improvement.

Join Databricks as a Data Platform Solutions Architect in our professional services team, focusing on emerging enterprises and DNB. In this pivotal role, you will collaborate with clients to design and implement innovative data solutions that drive business transformation. Your expertise will empower organizations to leverage the full potential of the Databricks platform, enhancing their data analytics capabilities.

At Databricks, we are driven by a passion for empowering data teams to tackle some of the world's most challenging problems—whether it's revolutionizing transportation or accelerating medical advancements. Our mission is to build and operate the premier data and AI infrastructure platform, enabling our clients to harness deep data insights to enhance their businesses.Founded by engineers and deeply committed to our customers, we eagerly embrace every opportunity to confront technical challenges. From designing next-gen UI/UX for data interfaces to scaling our services and infrastructure across millions of virtual machines, we are just getting started.As part of our ongoing commitment, we are embarking on a multi-year journey to develop the best Lakehouse Platform. While we are building on a solid foundation, our aim is to create significantly improved products. This involves re-evaluating every component to deliver our customers the fastest, easiest to use, and most secure data platform for all their data workloads.As a Software Engineer, you will play a vital role as a founding member of our London site and the core team in our journey towards achieving the Lakehouse vision. You will be engaged in the complete development lifecycle and will embody the core values that define Databricks.Your Impact:Our backend teams tackle diverse challenges across essential service platforms. You may work on:Complex problems that encompass product and infrastructure components, including distributed systems, scalable service architecture, monitoring, workflow orchestration, and enhancing developer experience.Developing reliable, secure, and high-performance services and client libraries for managing vast amounts of data on cloud storage backends such as AWS S3, GCS, and Azure Blob Store.Creating product features that empower our customers to seamlessly store and access their data.Addressing reliability issues related to Lakebase.Proactively identifying downtime causes and systematically eliminating root issues.Assisting the organization in defining SLIs, achieving SLOs, and driving long-term reliability improvements.

Begin your journey here. Advance your career with us.AECOM seeks an experienced Principal Civil Engineer to lead technical initiatives across a variety of projects for data center clients. These projects encompass site due diligence assessments, master planning, detailed design, and construction support. This role is pivotal in fostering business growth, enhancing client relations, and advancing AECOM's civil engineering capabilities within the data center sector.As part of our Civil Engineering Business Unit, you will collaborate with a diverse team of professionals, including ground engineers, structural engineers, mechanical engineers, electrical engineers, architects, project managers, environmental specialists, and surveyors, both locally and globally, to support our extensive portfolio of data center projects.In the Data Centres Market Sector, you will represent AECOM as a knowledgeable ambassador, delivering expert civil engineering counsel throughout EMEA and in conjunction with global teams. Your contributions will ensure the successful execution of data center projects for major technology clients, emphasizing technical excellence, resilience, and cross-disciplinary integration while reinforcing AECOM's standing in the Hyperscale and Co-location data center market.Your responsibilities will include:Providing strategic civil engineering leadership and technical governance for data center projects, ensuring adherence to AECOM standards and regulatory requirements.Leading, reviewing, and approving civil engineering infrastructure designs and specifications, creating viable solutions for site development, earthworks, drainage, pavements, access, utility infrastructure, and civil engineering works related to data centers.Coordinating with fellow civil engineers and technical experts to ensure the successful delivery of our data center project portfolio, collaborating with ground engineers, electrical engineers, mechanical engineers, structural engineers, project managers, architects, and environmental consultants across the UK and internationally.Enhancing client engagement, driving business development, strategic asset planning, and sector thought leadership, while mentoring and developing civil engineers to improve technical capabilities and delivery excellence within the Data Centres sector.Join us and thrive:At AECOM, we value our employees and offer a variety of core and personalized benefits tailored to support your future and well-being. Enjoy career development opportunities, a flexible hybrid working model to achieve work-life balance, access to technical practice networks, AECOM University, and opportunities for volunteering.

Genomics England seeks a Principal Data Engineer based in London. This senior role centers on building and maintaining the data systems that power the company’s work in healthcare and genomics. The position involves designing and implementing solutions that turn genomic data into insights for better patient outcomes. Key responsibilities Design and develop infrastructure to manage large-scale genomic data projects. Collaborate with teams across Genomics England to identify data needs and deliver tailored solutions. Maintain data integrity and ensure accessibility throughout all systems. Influence and shape the company’s data engineering practices and standards. Requirements Significant experience in data engineering, preferably within complex or scientific settings. Proven ability to design, build, and optimize data systems. Comfort working alongside colleagues from a range of disciplines. Strong commitment to data quality and security.

As a Senior Forward Deployed Engineer at Databricks, you will play a critical role in deploying our cutting-edge data analytics solutions to our clients. You will work closely with customers, providing technical expertise and support to ensure successful implementation and utilization of our platform. Your role will involve collaborating with various teams to address complex customer challenges, enhance user experience, and drive product adoption.

Are you ready to shape the future of data solutions? Join Databricks as a Data Platform Solutions Architect within our Professional Services team. In this role, you will leverage your expertise to help clients harness the power of data, driving innovation and efficiency in their businesses.As a key contributor, you will collaborate with cross-functional teams to design and implement robust data platforms that meet diverse business needs. Your technical acumen and strategic insight will play a vital role in delivering exceptional service and ensuring client satisfaction.

Role Overview AECOM is hiring a Principal Mechanical Engineer with a focus on data centres in London. This senior position shapes the design and operation of advanced data centre facilities. What You Will Do Lead technical projects for data centre mechanical systems Guide and mentor junior engineers Work closely with multidisciplinary teams to deliver efficient, sustainable solutions Drive high standards for performance and reliability in data centre design

Role overview Nagarro seeks a Principal Engineer - Senior Project Manager in Data Engineering for its London office. This position focuses on leading complex data engineering initiatives from initial planning through delivery. The role blends hands-on technical leadership with project management, guiding a team to create scalable data solutions that support changing business requirements. What you will do Lead and manage a team of data engineers on large-scale projects Oversee the design and implementation of scalable data architectures Translate business requirements into practical, innovative data solutions Drive data strategy and ensure alignment with organizational goals Mentor and inspire team members, encouraging technical growth and collaboration Requirements Extensive experience in data engineering and project management Strong technical background in designing and implementing data solutions Proven ability to lead, inspire, and develop engineering teams Skilled at turning business needs into effective data strategies Nagarro values candidates who combine technical expertise with a collaborative leadership approach.

Join Appvia as a Principal Azure Platform Engineer and help shape the future of cloud technology!Location: London, UK (Hybrid - typically 1-2 days per month in office)Contract: PermanentCompensation: £90,000 - 120,000 + 10% Bonus + 10% Shares + Comprehensive BenefitsAbout UsAt Appvia, we empower businesses to advance on their cloud and DevOps journey. As a top-tier provider, we leverage innovative technologies and services to facilitate seamless cloud adoption for our clients.Our goal is to enable organizations to efficiently deploy applications in the cloud. We are dedicated to constructing a robust cloud infrastructure that simplifies management and updates for platform engineering teams while providing developers with the flexibility to deploy their applications effortlessly. Our commitment to delivering value drives our passion for our clients' success.About the RoleAs the Principal Azure Platform Engineer, you will play a vital role in guiding our clients towards achieving cloud and DevOps excellence. Your expertise will be instrumental in designing and implementing scalable, resilient, and secure platforms, while aligning strategic initiatives with customer goals.This position presents an exciting opportunity to lead a team of cloud consultants, championing innovation in cloud adoption strategies and contributing to the development of Appvia's internal solutions team during a transformative phase for our organization.Interview Process - 4 StagesInitial discussion with our Talent Acquisition Manager.Technical Assessment in collaboration with the Engineering team.In-depth Technical Interview with Practice Leadership.Final Competency-focused Interview with Engineering Leadership.Important NoteTo be considered, candidates must either hold active SC or be eligible for SC clearance:Must possess a British Passport or ILR.Must have lived in the UK for at least the last 5 years.No criminal records.No financial irregularities.QualificationsAbout YouYou are passionate about emerging technologies and automation, with a proven track record in leading an Engineering Team adhering to Agile and DevOps best practices. You should be able to discuss your:Expertise in continuous integration and deployment practices.Ability to collaborate effectively with individuals at all technical levels.Understanding of industry best practices.Project Management experience.Exceptional interpersonal skills, including the ability to inspire a team towards shared objectives.ResponsibilitiesCollaborate with prospective and current clients to streamline cloud adoption.

REQ ID: FEQ127R254 Location: Hybrid – London Join our dynamic Professional Services team at Databricks, where we are actively seeking talented individuals for various roles, including Senior Solutions Consultant, Resident Solutions Architect, or Senior Resident Solutions Architect. This position is ideal for those with a strong background in big data solutions, analytics architecture, or technical consulting. As a Big Data Solutions Architect (Internally known as Resident Solutions Architect), you will collaborate with clients to address their big data challenges using the Databricks platform. This role involves overseeing data engineering, data science, and cloud technology projects, ensuring seamless integration with client systems, delivering training, and executing technical tasks that maximize the value derived from their data. Our RSAs are billable professionals who excel in project execution while providing exceptional customer service. You will report directly to the regional Manager/Lead. Your Impact: Lead impactful technical projects, including the design and construction of reference architectures, and the productionalization of customer use cases. Collaborate with engagement managers to define the scope of diverse professional services based on client input. Guide strategic clients through the implementation of transformative big data projects, including third-party migrations, from initial design to deployment of leading big data and AI applications. Consult on architectural design and assist in bootstrapping or implementing customer projects, fostering a solid understanding and adoption of Databricks solutions. Provide elevated support for customer operational challenges. Work closely with the Databricks technical team, Project Manager, Architect, and client teams to ensure all technical components meet customer requirements. Engage with Engineering and Databricks Customer Support to offer product feedback and facilitate rapid resolution of engagement-specific issues. What We Seek: Extensive experience in data engineering, data platforms, and analytics. Proficient in coding with Python or Scala. Familiarity with at least two major cloud ecosystems (AWS, Azure, GCP) with deep expertise in one. Strong communication skills and a passion for customer success.

At Databricks, we are dedicated to empowering data teams to tackle the world's most pressing challenges — from transforming transportation to accelerating medical innovations. Our mission is to create and maintain the premier data and AI infrastructure platform, enabling our clients to leverage profound data insights to enhance their operations. Established by engineers with a customer-centric approach, we enthusiastically seize every opportunity to address technical hurdles, whether that involves designing next-generation UI/UX for data interaction or scaling our services across millions of virtual machines. We're just getting started. The Lakebase Platform Reliability team operates across a diverse array of stacks, systems, and stakeholders. This includes AI-driven tools and workflows for customer management, real-time incident observability, and systems that support compliance through monitoring and auditing, as well as customer-facing operational APIs and maintenance workflows. As part of this team, you'll play a vital role in our overarching platform mission: to develop robust resource management infrastructure, dependable distributed services, and internal tools that empower Databricks engineers to operate seamlessly across various cloud environments.

Req: FEQ127R163 Location: London Skills: Expertise in Data Science, Machine Learning, Artificial Intelligence, Large Language Models, Generative AI, and Big Data technologies like Spark As a Specialist Solutions Architect specializing in Machine Learning Engineering, you will serve as a pivotal technical resource for Databricks customers and the Field Engineering team. Collaborating closely with Solution Architects, you will assist customers in designing and implementing production-grade machine learning applications on the Databricks platform, ensuring alignment with the evolving Databricks Data Intelligence Platform roadmap. You will continually advance your technical acumen by leveraging cutting-edge technologies in Generative AI, LLMOps, and Machine Learning, while also enhancing your influence through mentorship and establishing yourself as an authority in ML. Your Impact: Design and architect production-ready ML workloads for customers on our unified platform, which includes comprehensive ML pipelines, optimization for training and inference, integration with cloud-native services, and MLOps practices. Deliver advanced technical support to Solution Engineers during the technical sales process, covering aspects from feature engineering and model training to tracking, serving, and monitoring—all within a single integrated platform, and participating actively in the Databricks ML Subject Matter Expert community. Work collaboratively with product and engineering teams to voice customer needs, set priorities, and shape the product development roadmap, facilitating the adoption of Databricks’ ML solutions. Enhance customer data science capabilities and apply best practices in MLOps to effectively deploy these capabilities across various domains. Act as a trusted technical advisor for customers developing Generative AI solutions, including Retrieval-Augmented Generation (RAG) architectures, querying structured data using natural language, content generation, and monitoring solutions. Qualifications: Minimum of 5 years of experience in a customer-facing technical role, with a focus on pre-sales or post-sales support across diverse industry sectors. 5+ years of hands-on machine learning experience in at least one of the following roles: ML Engineer: Developing production-grade cloud infrastructure (AWS/Azure/GCP) to support the deployment of machine learning applications, including drift monitoring. Data Scientist: Proficient in modern natural language processing techniques, including vector databases, fine-tuning of large language models (LLMs), and deploying LLMs using tools such as HuggingFace, Langchain, and OpenAI. Graduate degree in a relevant field is preferred.

About BMLL Technologies:BMLL Technologies stands at the forefront of financial data analytics, providing unparalleled access to harmonised Level 3, 2, and 1 historical data across global equities, ETFs, futures, and US equity options. With our cutting-edge analytics, market participants can gain immediate insights into granular T+1 order book data, facilitating accelerated research, refined trading strategies, and a deeper understanding of market dynamics.In 2025, we were proud to be acquired by Nordic Capital, alongside minority stakeholder Optiver, underscoring our commitment to catalyzing significant growth in the next phase of our journey.We foster an inclusive, collaborative culture and offer a hybrid working environment that includes regular office days in London, engaging weekly team lunches, and a variety of social activities outside of working hours.To learn more, please visit our website or connect with us on X (@bmlltech) and LinkedIn (@BMLL).About the Role:As a leading provider of financial data, BMLL transforms raw market data from financial exchanges into an accessible and standardised format that caters to various customer use cases. Our extensive repository encompasses over 10 years of in-depth market data from approximately 100 venues, comprising around 1.5 billion HDF5 files (approximately 1.5 PB) stored on S3 and organised within Postgres, with partitions based on date, venue, and instrument.While our current structure has served us well, we now face challenges due to the sheer volume of small files and the separate storage of the Postgres catalogue, which limits our ability to evolve, back up, recover, and share data effectively. Our strategic initiative involves migrating this processed layer to Delta Lake, both for data and catalogue purposes. Additionally, we are transitioning our Data Products from Snowflake to Iceberg-compatible Delta Tables to optimise performance, reduce costs, and enhance manageability.We seek an accomplished Databricks Lead Engineer and Architect to spearhead the Delta Lake and Delta Table re-engineering projects. This hands-on role demands direct involvement in the project.The primary objective extends beyond mere format migration; it encompasses addressing critical architectural goals for high-cardinality data, including:Establishing a Delta Lake architecture for large-scale operations.Modelling and partitioning L2/L3 order book data within Delta Lake.Implementing metadata management, data compaction, and versioning strategies.Designing a system that accommodates multiple downstream delivery models.Creating a robust backup strategy at scale, aiming for a 1-day RTO.This opportunity invites an experienced engineer to join a forward-thinking FinTech company and play a pivotal role in a significant re-engineering project.