Senior Engineering Manager - Photon Team at Databricks | San Francisco

The Applied AI team at Databricks is dedicated to pioneering advancements in GenAI-driven products. In recent years, we have successfully launched notable innovations such as the Databricks Assistant, AI/BI Genie, and Agent Bricks. These products are utilized by hundreds of thousands of Databricks users daily. We are addressing complex challenges such as code suggestions, error detection and correction, text-to-SQL generation, automatic pipeline creation, and knowledge QA. As our GenAI products continue to advance, we are on the lookout for multiple GenAI Engineers, ranging from junior to senior levels, to spearhead the next phase of development. In 2025, our focus will be on enhancing the quality of LLMs, broadening GenAI functionalities across Databricks products, and fortifying our platform architecture to facilitate seamless AI interactions at scale.

The Applied AI team at Databricks is at the forefront of pioneering GenAI-powered products. In recent years, we have successfully launched the Databricks Assistant, AI/BI Genie, and Agent Bricks, collaborating with product teams to significantly enhance LLM quality for these offerings. These innovations are utilized by hundreds of thousands of Databricks users daily. We are dedicated to solving complex challenges such as code suggestion, error detection and correction, text-to-SQL generation, automatic pipeline generation, and knowledge QA.As we continue to evolve our GenAI products, we are looking for multiple GenAI Engineers at various experience levels to lead the next phase of our development. Our goals for 2025 include improving LLM quality, broadening GenAI capabilities across Databricks products, and reinforcing our platform architecture to facilitate seamless AI interactions on a large scale.

About This Role Join Databricks as a Software Engineer focused on GenAI inference, where you will play a pivotal role in designing, developing, and enhancing the inference engine that drives our Foundation Model API. Collaborating at the intersection of research and production, you will ensure our large language model (LLM) serving systems are optimized for speed, scalability, and efficiency. Your contributions will span the entire GenAI inference stack, from kernels and runtimes to orchestration and memory management. What You Will Do Participate in the design and implementation of the inference engine, collaborating on a model-serving stack tailored for large-scale LLM inference. Work closely with researchers to integrate new model architectures or features such as sparsity, activation compression, and mixture-of-experts into the engine. Optimize latency, throughput, memory efficiency, and hardware utilization across GPUs and other accelerators. Build and maintain tools for instrumentation, profiling, and tracing to identify bottlenecks and inform optimization efforts. Develop scalable routing, batching, scheduling, memory management, and dynamic loading mechanisms for inference workloads. Ensure reliability, reproducibility, and fault tolerance in inference pipelines, including A/B launches, rollback, and model versioning. Integrate with federated and distributed inference infrastructure, orchestrating across nodes, balancing load, and managing communication overhead. Engage in cross-functional collaboration with platform engineers, cloud infrastructure, and security/compliance teams. Document and share insights, contributing to internal best practices and open-source initiatives as appropriate.

Location: San Francisco Bay Area, California REQ ID FEQ127R103 Mission Join our team as a Solutions Architect, where you will play a pivotal role in empowering large enterprises to harness the full potential of their data. Collaborating closely with the Enterprise Account Executive (AE), you will define and implement the technical strategy for our most significant accounts, driving deeper and wider adoption of machine learning (ML) and artificial intelligence (AI) solutions. Your expertise in value-based selling, technical account management, and leadership will be key in maximizing client success. You will work alongside a dedicated team that builds proofs of concept and showcases Databricks' innovative products. A strong technical understanding of ML and AI applications across various use cases relevant to the High Tech and Manufacturing sectors is essential. Outcomes Act as the primary technical voice for Databricks across multiple clients. Guide customers through a transformational journey, facilitating their evaluation and adoption of Databricks solutions. Implement tailored technical strategies within accounts based on a deep understanding of their needs. Foster a network of technical champions within client organizations. Align technical strategies with Databricks solutions effectively. Provide structured mentorship to enhance the skills of team members. Earn respect from peers through your experience, insights, and contributions. Competencies: Expertise in establishing and leading virtual teams to achieve success within client accounts. Experience managing large global accounts with an annual recurring revenue (ARR) exceeding $1 million. Ability to build relationships with executives and key influencers. Skill in presenting persuasive viewpoints to decision-makers, steering them toward successful outcomes. Strong technical background in big data, data science, and cloud technologies. Proven ability to drive data-driven business transformation. Programming experience in Python, R, Scala, or Java. Preferred: Databricks Certification.

About Taskrabbit:Taskrabbit is an innovative marketplace platform that seamlessly connects individuals with Taskers to manage everyday home tasks, including furniture assembly, handyman services, moving assistance, and much more.At Taskrabbit, we aim to transform lives one task at a time. We celebrate innovation, inclusion, and hard work, fostering a collaborative, pragmatic, and fast-paced culture. We seek talented, entrepreneurially minded, data-driven individuals who possess a passion for empowering others to pursue their passions. In partnership with IKEA, we are creating more opportunities for individuals to earn a consistent, meaningful income on their terms by establishing enduring relationships with clients in communities globally.Taskrabbit operates as a hybrid company, with team members located across the US and EU, and has been recognized as a Built In — Best Places to Work for 2022, 2023, and 2024, receiving accolades across various national and regional categories. Join us at Taskrabbit, where your contributions will be significant, your ideas appreciated, and your potential maximized!This position operates on a hybrid schedule, requiring two days of in-office collaboration per week. It can be based in our San Francisco office or our new New York City office (opening March 2026).About the RoleMachine Learning is a foundational element at Taskrabbit, and we are in search of an experienced Senior Machine Learning Engineer to join our team and help mold the future of ML/AI at Taskrabbit. This distinct, full-stack role is designed for someone who is enthusiastic about the entire machine learning lifecycle—from initial research and model development to constructing the robust infrastructure necessary for deploying and scaling your innovations.As a Senior Machine Learning Engineer, you will engage with exciting challenges that directly influence how users discover and interact with home services on the Taskrabbit platform. You will play a vital role in enhancing our capabilities in areas such as search ranking, content discovery, and recommendation systems. Collaborating closely with data scientists and fellow engineers, you will design and implement cutting-edge algorithms, ensuring the scalability, reliability, and optimization of our models in production alongside software engineers.

About VSCO At VSCO, we empower photographers with an innovative platform that provides essential tools, a vibrant community, and the visibility needed for creative and professional growth. We cultivate an authentic creative environment that welcomes photographers of all skill levels, offering a space that inspires opportunity, collaboration, and connection. Our mission is to support photographers in their journeys, enabling them to thrive and connect with fellow creatives and businesses through our comprehensive suite of tools, available on both mobile and desktop. We seek individuals who are passionate and proactive in advancing our mission. Our team members have the opportunity to make a significant impact, and we believe that collaborative efforts yield stronger results. Our core values are essential to our team culture and guide our hiring process. Learn more about what you can expect when joining VSCO on our Careers Page. About The Role As a Senior Machine Learning Engineer, you will harness the power of AI and machine learning to create innovative, reliable user-facing product features. You will leverage your extensive technical background and hands-on experience in deploying machine learning models to deliver impactful solutions based on real-world feedback. Your focus on measurable outcomes and customer satisfaction drives your work, blending innovation with practical implementation. You will be highly skilled in Python and adept across the data and machine learning stack, enabling you to develop and launch models efficiently while ensuring scalability and maintainability. Whether working with traditional algorithms or cutting-edge deep learning and generative AI, you will expertly navigate the complexity of each problem, managing every phase from defining the challenge to deployment and iterative improvement. Your dedication to software engineering excellence will inform your thoughtful approach to system design for machine learning, encompassing data quality, pipeline design, feature workflows, model serving, and ongoing monitoring and enhancement. By integrating machine learning deeply within our cohesive product experiences, you will collaborate effectively with cross-functional teams, aligning on objectives, defining success metrics, and driving meaningful outcomes. You will stay informed about the rapidly evolving AI landscape, maintaining a discerning perspective that allows your team to focus on significant advancements while avoiding distractions. The Day to Day Design and implement ML-powered features for search, discovery, personalization, and more.

P-1380 Join Databricks as a Senior Applied AI Engineer, where you will harness the power of machine learning, scheduling, and optimization algorithms to enhance the efficiency and performance of our engineering systems and infrastructure. Our Applied AI team tackles some of the most challenging and fascinating issues in the industry, ensuring that Databricks infrastructure and products operate at peak performance and cost efficiency. This role is critical, as our customers depend on us to deliver the most optimized workloads. Your Impact: Develop comprehensive systems from the ground up within a dynamic team of seasoned professionals. Influence the direction of our applied machine learning investment areas by collaborating with engineering and product teams across the organization. Lead the design and implementation of advanced AI models and systems that enhance the capabilities and performance of Databricks' products, infrastructure, and services. Architect and deploy robust, scalable machine learning infrastructure, including data storage, processing, model training, serving components, and monitoring systems to facilitate seamless integration of AI/ML models into production environments. Explore innovative modeling techniques in the realm of machine learning for systems. Contribute to the wider AI community by publishing research, presenting at conferences, and actively engaging in open-source projects, thereby strengthening Databricks' reputation as an industry leader.

Join Databricks Mosaic AI as a Senior Machine Learning Engineer and take the lead in developing our cutting-edge generative AI platform. Our team, formed in late 2020, empowers businesses by allowing them to securely fine-tune, train, and deploy custom AI models using their own data. This ensures maximum security and control while being compatible with all major cloud providers, allowing for unparalleled flexibility in AI development.Since our integration into Databricks in July 2023, we have been dedicated to tackling some of the world's most challenging problems, from revolutionizing transportation to accelerating medical advancements. We leverage deep data insights to enhance our customers' business capabilities and thrive on overcoming technical challenges to deliver superior data and AI solutions.Role Overview:As a Senior Machine Learning Engineer, you will play a pivotal role in the design and implementation of our generative AI platform, covering the entire ML development lifecycle, including data generation, training, evaluation, serving, and agent-building. Your expertise will be essential in translating user requirements into intuitive product interfaces while constructing robust backend distributed systems that drive these features.

Join Databricks as a Senior Backend Software Engineer and contribute to solving some of the world's most challenging problems. We empower data teams through our state-of-the-art data and AI infrastructure platform, enabling businesses to leverage deep data insights for transformative growth. At Databricks, founded by engineers and driven by a customer-first philosophy, we tackle technical challenges with enthusiasm — from designing next-gen UI/UX for data interfacing to scaling our services across millions of virtual machines.In this role, you will collaborate closely with your team to develop scalable infrastructure and innovative products for the Databricks platform.

About Mariana MineralsMariana Minerals is a pioneering software-driven, vertically integrated minerals company dedicated to supplying the essential minerals that fuel modern energy, artificial intelligence, and defense technologies. We are transforming the minerals supply chain by leveraging extensive industry knowledge alongside cutting-edge software, automation, and data-centric decision-making.The RoleAs we build the critical minerals supply chain from the ground up, we seek a highly skilled Senior Machine Learning Engineer to help drive our autonomous operations.Unlike traditional software companies, we are a mining company that develops our own software solutions. Mariana designs, builds, commissions, and operates our own mines and refineries, where we create proprietary chemical processes. Currently, we are producing battery-grade lithium salts from real oil and gas wastewater at our facilities. Our first commercial-scale lithium production facility, Lithium One, is set to commence initial production in the first half of 2027.In your role as a Senior Machine Learning Engineer, you will spearhead the design and implementation of machine learning systems that directly optimize the operations of our mineral refining facilities and guide significant investment and operational decisions. Your contributions will be visible in tangible metrics such as recovery rates, energy consumption, reagent usage, and equipment uptime.The TechThis position involves some of the most compelling applied AI work available today.Our internal platform, PlantOS, utilizes the same reinforcement learning frameworks that power self-driving cars and humanoid robots, adapted for the autonomous and short-interval control of mineral refining circuits. Our models dynamically adjust operating parameters in real time, optimizing for lithium recovery, reagent consumption, energy efficiency, and equipment uptime simultaneously.The working environment is complex and ever-changing; variations in wastewater compositions, ore grades, and aging equipment require the system to adapt continuously. The ultimate objective is to achieve fully autonomous refining operations—when you implement your solutions here, you will witness the physics of the process evolve.What You’ll DoCollaborate closely with Mariana’s process chemistry and engineering teams to develop innovative, data-driven models of core chemical unit operations.Train and deploy reinforcement learning models to optimize operational processes.

Job OverviewJoin Eragon as a Machine Learning Engineer and lead the charge in transforming innovative AI models into scalable, production-grade systems. This position is pivotal in bridging research and real-world applications by designing and optimizing systems that enhance vital workflows throughout the enterprise.In collaboration with our research, product, and engineering teams, you will convert cutting-edge capabilities into dependable, high-performance systems ready for production.Key ResponsibilitiesModel Development & Deployment: Craft, refine, and deploy machine learning models within production settings.Systems Engineering: Architect scalable pipelines for training, inference, evaluation, and comprehensive monitoring.Performance Optimization: Enhance the latency, throughput, cost-efficiency, and reliability of ML systems.Data & Infrastructure: Manipulate large datasets and ensure seamless integration of models with internal systems and APIs.Cross-Functional Collaboration: Collaborate with product and engineering teams to provide end-to-end AI functionalities.Evaluation & Monitoring: Develop robust evaluation frameworks and feedback loops to ensure system effectiveness.

P-150 At Databricks, we are dedicated to empowering data teams to tackle some of the world's most challenging problems, ranging from security threat detection to the development of cancer treatments. We achieve this by creating and managing the leading data and AI infrastructure platform, allowing our clients to concentrate on the high-value challenges central to their missions. Founded in 2013 by the original creators of Apache Spark™, Databricks has evolved from a modest office in Berkeley, California, to a global enterprise with over 1000 employees. Our platform is trusted by thousands of organizations, from startups to Fortune 100 companies, establishing us as one of the fastest-growing SaaS firms worldwide. Our engineering teams design and develop highly technical products that meet significant real-world needs. We continually push the limits of data and AI technology while maintaining the resilience, security, and scalability essential for our customers' success on our platform. We operate one of the largest-scale software platforms, comprising millions of virtual machines that generate terabytes of logs and process exabytes of data daily. Given our scale, we frequently encounter cloud hardware, network, and operating system faults, and our software must effectively shield our customers from any disruptions. As a Software Engineer focused on backend development, you will collaborate closely with your team and product management to prioritize, design, implement, test, and operate microservices for the Databricks platform and products. This role includes writing software in Scala/Java, building data pipelines using Apache Spark™ and Apache Kafka, integrating with third-party applications, and interacting with cloud APIs such as AWS, Azure, CloudFormation, and Terraform. Below are examples of teams you can join: Data Science and Machine Learning Infrastructure: Develop services and infrastructure that bridge the gap between machine learning and distributed systems. Our technology supports flagship collaborative workspaces, notebooks, IDE integrations, and project management tools. We also facilitate large-scale machine learning through environment management, distributed training, and lifecycle management via MLflow. Compute Fabric: Construct the resource management infrastructure that supports all big data and machine learning workloads on the Databricks platform in a robust, flexible, secure, and cloud-agnostic manner. This software oversees millions of virtual machines.

At Sciforium, we are pioneering the future of AI infrastructure by creating cutting-edge multimodal AI models and a proprietary, high-efficiency serving platform. With substantial financial backing and direct support from AMD engineers, our team is rapidly expanding as we develop the comprehensive stack that drives advanced AI models and real-time applications.About the RoleIn the capacity of a Machine Learning Engineer, you will engage with the entire foundation-model stack, encompassing pretraining and scaling, post-training and Reinforcement Learning, sandbox environments for evaluation and agentic learning, and deployment + inference optimization. You’ll have the opportunity to rapidly iterate on research ideas, contribute to production-grade infrastructure, and help deliver models capable of addressing real-world challenges at scale.Your ResponsibilitiesThis position offers diverse tracks - candidates can specialize or contribute across multiple areas. Key responsibilities include:Pretraining & ScalingTrain expansive byte-native foundation models utilizing vast, heterogeneous data sources.Formulate stable training methodologies and scaling laws tailored for innovative architectures.Enhance throughput, memory efficiency, and resource utilization across extensive GPU clusters.Establish and maintain distributed training infrastructures alongside fault-tolerant pipelines.Post-training & Reinforcement LearningBuild out post-training frameworks (SFT, preference optimization, RLHF/RLAIF, RL).Curate and produce specialized datasets aimed at enhancing specific model capabilities.Develop reward models and evaluation systems to facilitate ongoing improvements.Investigate inference-time learning and computational strategies to boost performance.Sandbox Environments & EvaluationCreate scalable sandbox environments for agent assessment and learning.Generate realistic, high-signal automated evaluations for reasoning, tool usage, and safety.Design both offline and online environments that support RL-style training at scale.Implement instrumentation for observability, reproducibility, and rapid iteration.Deployment & Inference OptimizationOptimize deployment strategies to ensure models are efficient and effective in real-world applications.

Join Strava, a leader in the sports technology sector, as a Machine Learning Engineer. In this exciting role, you will apply your expertise in machine learning and data science to develop innovative solutions that enhance the experience of millions of athletes worldwide. Collaborate with cross-functional teams to create algorithms that analyze vast datasets and provide actionable insights to our users.

Join Our Mission to Empower Communities! At GoFundMe, we are the leading global platform dedicated to helping people support each other. Since our inception in 2010, we've facilitated over $40 billion in fundraising, uniting individuals and nonprofits in a safe and engaging environment. We are seeking a passionate and experienced Senior Manager, Machine Learning Engineering to join our Applied Science team. In this pivotal role, you will spearhead a dynamic team tasked with creating impactful machine learning solutions that enhance user experiences and drive business growth. Your responsibilities will include designing, developing, and deploying cutting-edge services and applications utilizing advanced machine learning and AI methodologies, such as generative AI, personalized recommendations, search systems, and classification techniques. This position requires exemplary leadership capabilities to ensure both strategic vision and tactical execution, backed by your extensive knowledge of machine learning, AI, and secure software practices. This role is based in the vibrant San Francisco Bay Area, with an expectation of being in the office three times a week.

Are you a dedicated Developer Advocate with a passion for bridging the gap between technology and the developer community? If you thrive on connecting with others, hosting engaging events, and diving into the nuances of application development and developer experience, we want to hear from you!As a Senior Developer Advocate at Databricks on the Neon DevRel team, you'll play a pivotal role in linking our engineering teams with an expanding community of application and agent developers utilizing Postgres and Neon. Your technical acumen will be essential in maintaining and enhancing our SDKs, educating users, and shaping the future of Agent and Developer Experience (AX & DX) for Neon.Developer experience will be central to your responsibilities. You will help enhance our TypeScript SDK and CLI, manage Neon’s MCP server, and develop other developer-oriented tools, all while keeping ahead of the evolving AI landscape. Your expertise in DevTools and AX will be shared through presentations, meetups, blog articles, videos, and community interactions, empowering developers to build more effectively with Neon and demonstrating our commitment to exceptional Developer Experience (DX) and Agent Experience (AX).Collaboration will be key as you work closely with the neon.com community, product teams, and engineering staff, ensuring that developer feedback directly influences product direction. You'll also partner with fellow advocates, program managers, and the broader Databricks DevRel organization to create a cohesive and impactful strategy for agent experiences.The ideal candidate will embody the core values of our DevRel team: a profound enthusiasm for application development, strong empathy for developer needs, and the ability to convey complex concepts with clarity and effectiveness.In this role, you’ll foster and expand the Neon community through top-notch technical content and hands-on experiences, continuously feeding insights back into the product to further enhance the developer experience.

At Lyft, we are driven by our mission to connect and serve our communities. We strive to foster a workplace where every team member feels valued and has the opportunity to excel. With over half a billion rides and counting, Lyft is tackling complex challenges on a grand scale, utilizing cutting-edge AI and Machine Learning technologies to enhance customer experiences. The Artificial Intelligence, Machine Learning, and Operations Research Platforms team (AIMLOR) is on the lookout for a Senior Machine Learning Engineer who will play a pivotal role in constructing AI Platform components that empower essential AI applications across Lyft. Mastery in Generative AI and platform development is crucial for this position. You will contribute to our platform that facilitates real-time, online, and offline AI and ML model execution, development, and iteration. Collaborating with a team of talented Machine Learning and Software Engineers, you will work on intricate problems and define solutions that make a direct impact on our systems throughout the organization. If you are enthusiastic about building an AI Platform at scale with applications spanning every aspect of our company, we want to hear from you. If you are a creative thinker with a strong background in AI and machine learning systems and are passionate about leveraging data to solve business challenges in a dynamic, innovative, and collaborative environment, we invite you to apply.

Join Our Innovative Team at KiddomAt Kiddom, we are revolutionizing education through our cutting-edge platform that champions student equity and growth. We seamlessly integrate high-quality instructional materials with engaging digital learning experiences. Our robust curriculum management tools empower schools and districts to effectively manage their curriculum, leading to personalized learning experiences that cater to the unique needs and aspirations of local communities. Our high-quality curriculum is enriched with insightful data analytics for teachers and leaders, fostering continuous improvement in instructional strategies, school programming, and professional development.Your Role and ResponsibilitiesAs a vital member of Kiddom’s AI team, you will play a critical role in developing the backbone of our search, recommendation, and insights systems. Your contributions will directly enhance the experience of teachers and students by providing timely insights, personalized content, and intelligent assistance tailored to their needs.

Join Reducto as a Machine Learning EngineerAt Reducto, we empower AI teams to harness real-world enterprise data with unparalleled precision.Much of the enterprise data—ranging from financial documents to healthcare records—remains trapped in unstructured formats such as PDFs and spreadsheets. Our vision models are designed to interpret these documents in a human-like manner, enabling the development of innovative products, training of machine learning models, and automation of processes on a large scale.Our rapid growth is a testament to our success, having achieved a staggering 7x year-over-year revenue increase, collaborating with numerous companies from prominent AI teams like Harvey, Vanta, and Scale to major enterprises including FAANG and leading trading firms.With over $100 million raised from esteemed investors such as A16z, Benchmark, and First Round Capital, we are on the lookout for a talented Machine Learning Engineer to assist in training and deploying models crucial for our core product's success.