companyBMLL Technologies logo

Lead Databricks Engineer (Contract)

BMLL TechnologiesLondon, England, United Kingdom
Hybrid Contract

Clicking Apply Now takes you to AutoApply where you can tailor your resume and apply.


Unlock Your Potential

Generate Job-Optimized Resume

One Click And Our AI Optimizes Your Resume to Match The Job Description.

Is Your Resume Optimized For This Role?

Find Out If You're Highlighting The Right Skills And Fix What's Missing

Experience Level

Manager

Qualifications

Essential Qualifications:Proven industry experience with Databricks, including:Delta LakeUnity CatalogDelta TablesDelta UniForm (Universal Format)Hands-on experience with Apache Iceberg and data architecture principles. Strong problem-solving skills and the ability to work collaboratively in a team-oriented environment.

About the job

About BMLL Technologies:

BMLL Technologies stands at the forefront of financial data analytics, providing unparalleled access to harmonised Level 3, 2, and 1 historical data across global equities, ETFs, futures, and US equity options. With our cutting-edge analytics, market participants can gain immediate insights into granular T+1 order book data, facilitating accelerated research, refined trading strategies, and a deeper understanding of market dynamics.

In 2025, we were proud to be acquired by Nordic Capital, alongside minority stakeholder Optiver, underscoring our commitment to catalyzing significant growth in the next phase of our journey.

We foster an inclusive, collaborative culture and offer a hybrid working environment that includes regular office days in London, engaging weekly team lunches, and a variety of social activities outside of working hours.

To learn more, please visit our website or connect with us on X (@bmlltech) and LinkedIn (@BMLL).

About the Role:

As a leading provider of financial data, BMLL transforms raw market data from financial exchanges into an accessible and standardised format that caters to various customer use cases. Our extensive repository encompasses over 10 years of in-depth market data from approximately 100 venues, comprising around 1.5 billion HDF5 files (approximately 1.5 PB) stored on S3 and organised within Postgres, with partitions based on date, venue, and instrument.

While our current structure has served us well, we now face challenges due to the sheer volume of small files and the separate storage of the Postgres catalogue, which limits our ability to evolve, back up, recover, and share data effectively. Our strategic initiative involves migrating this processed layer to Delta Lake, both for data and catalogue purposes. Additionally, we are transitioning our Data Products from Snowflake to Iceberg-compatible Delta Tables to optimise performance, reduce costs, and enhance manageability.

We seek an accomplished Databricks Lead Engineer and Architect to spearhead the Delta Lake and Delta Table re-engineering projects. This hands-on role demands direct involvement in the project.

The primary objective extends beyond mere format migration; it encompasses addressing critical architectural goals for high-cardinality data, including:

  • Establishing a Delta Lake architecture for large-scale operations.
  • Modelling and partitioning L2/L3 order book data within Delta Lake.
  • Implementing metadata management, data compaction, and versioning strategies.
  • Designing a system that accommodates multiple downstream delivery models.
  • Creating a robust backup strategy at scale, aiming for a 1-day RTO.

This opportunity invites an experienced engineer to join a forward-thinking FinTech company and play a pivotal role in a significant re-engineering project.

About BMLL Technologies

BMLL Technologies is a premier independent provider of harmonised historical data and analytics in the financial sector. We empower market participants with actionable insights, enhancing their trading strategies and market comprehension.

Similar jobs

Tailoring 0 resumes

We'll move completed jobs to Ready to Apply automatically.