About the job
P-1348
Join Databricks, where our mission is to empower data teams to tackle the world's most challenging problems, from detecting security threats to advancing cancer drug development. We build and maintain the premier data and AI infrastructure platform, enabling our customers to focus on their critical missions. Our engineering teams are dedicated to creating innovative technical products that address significant needs while pushing the limits of data and AI technology. We operate with the resilience, security, and scalability necessary to ensure our customers succeed on our platform.
Our platform operates at an unparalleled scale, comprising millions of virtual machines and generating terabytes of logs, processing exabytes of data daily. We encounter various cloud hardware, network, and operating system faults, and our software must adeptly protect our customers from these challenges.
As a Staff Software Engineer on the Data Platform team, you will contribute to the development of the Data Intelligence Platform at Databricks, automating decision-making processes across the organization. Collaborating with Product Teams, Data Science, Applied AI, and more, you will create tools for logging, orchestration, data transformation, metric storage, governance platforms, and data consumption layers. Leveraging cutting-edge Databricks products and tools within the data ecosystem, your team will serve as a significant in-house customer, providing insights that shape our product's future.
Your Impact:
- Design and manage the Databricks metrics store, facilitating shared access to detailed metrics across business units and engineering teams with high quality and performance.
- Develop the cross-company Data Intelligence Platform, encompassing all business and product metrics necessary for running Databricks, balancing data protection with ease of sharing as we transition to a public entity.
- Create tools and infrastructure for efficiently managing Databricks operations at scale across multiple clouds and geographies, including CI/CD processes, testing frameworks for pipelines and data quality, and infrastructure-as-code tools.
- Establish the foundational ETL framework utilized by all company-developed pipelines.
- Collaborate with engineering teams to enhance...

