Staff Software Engineer - Machine Learning Observability

The Machine Learning Observability team at Datadog is at the forefront of developing innovative tools designed to monitor, interpret, and enhance AI systems deployed in production environments, with a special focus on Large Language Models (LLMs) and generative AI technologies. Our solutions provide comprehensive and scalable observability for AI workloads, including drift detection, model evaluation, and behavior tracing, empowering our clients to deploy AI confidently. As a Staff Software Engineer, you will be instrumental in driving the development of new features and core capabilities within Datadog’s LLM Observability product. You will influence product strategy, lead experimental initiatives, and leverage your extensive knowledge of AI systems and software engineering to tackle complex challenges in the rapidly evolving AI domain. Your contributions will have a significant impact on how our customers monitor, diagnose, and optimize LLM-powered applications in production. Join us in creating the essential tools that ensure AI systems are observable, comprehensible, and dependable in real-world applications. At Datadog, we value our office culture which fosters relationships, collaboration, and creativity. We operate within a hybrid work model to enable our Datadogs to achieve a work-life balance that suits them best.

At WHOOP, we are revolutionizing health and fitness with our state-of-the-art wearable technology, dedicated to enhancing human performance and extending healthspan. Our innovative approach enables members to gain profound insights into their bodies, behaviors, and daily activities, empowering them to make healthier choices and achieve peak performance.We are on the lookout for a Senior Machine Learning Engineer to join our dynamic Foundation AI team. This team is responsible for developing the multimodal foundation models that drive WHOOP’s next generation of intelligent, personalized, and health-optimizing solutions. These models seamlessly integrate data from wearable sensors, linguistic inputs, biomarkers, clinical data, and self-reported metrics to build scalable AI systems that accurately understand human physiology and behavior.In this pivotal role, you will be a key individual contributor leading the research, development, and implementation of large-scale multimodal models. You will work collaboratively with data scientists, ML engineers, and cross-functional partners to advance the frontiers of deep learning and ensure that our models provide significant value to WHOOP members.

Join our innovative Perception team at Zoox, where we harness the power of cutting-edge sensor data to decode the intricacies of dynamic driving environments. In this pivotal role, you will leverage world-class sensor data and a robust infrastructure to test and validate groundbreaking algorithms. Your contributions will be instrumental in developing advanced algorithms for segmentation, tracking, classification, and high-level scene understanding, allowing you to engage with any or all of these exciting components.As a Senior Staff Machine Learning Engineer, you will spearhead the creation of machine learning algorithms that significantly influence both onboard and offboard autonomy and validation processes. You will work collaboratively with specialized teams in Prediction, Planning, Simulation, and Safety Validation, shaping our overarching technical framework. Your innovative approach will bridge team boundaries, enabling the prototyping of new methodologies that guide the long-term technical vision across multiple divisions within the organization. The impact of your work will range from achieving immediate company objectives to leading pioneering exploratory initiatives.

The Machine Learning (ML) Data Services team is on the lookout for a passionate and seasoned Staff Technical Lead Manager (TLM) to spearhead the evolution and operation of our essential machine learning data infrastructure. This pivotal position requires a unique fusion of in-depth technical knowledge in machine learning systems, extensive data processing, and a proven track record in leadership to effectively manage both individual contributors and set technical direction. The ML Data Services team plays a vital role in ensuring the provision of reliable, high-quality, and easily accessible data for all machine learning applications. As custodians of the 'Data Engineering' phase of the ML lifecycle, we handle the most crucial input for any ML system: the data itself. Our infrastructure supports hundreds of data pipelines and processes petabytes of data, directly facilitating the success of our flagship products. Key Responsibilities: Technical Leadership and Strategic Direction Develop and execute the technical roadmap and long-term strategy for the ML Data Services platform, ensuring alignment with the growing needs of ML engineers across the organization. Act as the leading technical authority for the team, designing complex, highly scalable, and fault-tolerant systems for data ingestion, transformation, and serving. Guide the technical vision for sourcing diverse training data, encompassing real-world drive logs, resimulation outputs, auto-labeling pipelines, and human labeling efforts. Oversee the implementation of robust mechanisms for monitoring data quality, tracking data lineage, and ensuring comprehensive data governance and compliance. Provide hands-on technical mentorship, design reviews, and guidance to engineers, promoting best practices, high code quality, and operational excellence. Management and Team Development Lead a team of software engineers, conducting performance reviews, facilitating career development, and providing coaching and mentoring. Balance a hybrid role as a technical leader-manager, dedicating 50% of efforts to management and 50% to technical leadership, architecture, and hands-on coding/design. Drive recruitment efforts to attract, onboard, and retain exceptional engineering talent, fostering a diverse and inclusive team culture. Promote engineering excellence, operational rigor, and a strong sense of ownership within the team. Collaboration and Execution Work closely with ML model and Infrastructure teams to understand data requirements and deliver solutions that expedite ML development cycles. Lead the execution of major projects, ensuring timely delivery and alignment with business objectives. Effectively communicate complex technical concepts to a variety of stakeholders.

LOCATION: This position can be based anywhere in Canada, with a preference for candidates in the eastern or central time zones to collaborate effectively with our teams in the US, Europe, and India.ABOUT THE ROLEWiser Solutions is on the lookout for a Principal Machine Learning Engineer to steer and implement our AI and data science initiatives. This senior technical leadership role requires a unique individual who possesses extensive knowledge in machine learning, data science, and production engineering, coupled with the business insight needed to transform intricate capabilities into tangible customer value.In this role, you will serve as the leading technical authority on AI at Wiser. Your responsibilities will include defining the architectural vision, showcasing our capabilities to clients and partners, and delivering production systems that yield measurable business results. This position demands an individual who can seamlessly navigate between strategic planning and hands-on execution, capable of engaging with executives one day and troubleshooting production pipelines the next.We are fostering an AI-driven engineering culture at Wiser, where AI tools and methodologies are integrated into our workflows—not just our products. We seek a Principal AI Engineer who not only develops AI solutions but also exemplifies AI-enhanced working practices and aids the broader engineering team in their adoption. If you’re passionate about the transformative power of AI in software development and actively embrace this shift in your daily work, we want to hear from you.Key ResponsibilitiesStrategic LeadershipCollaborate with product and business leadership to define and refine Wiser's technical AI and data science strategy.Communicate Wiser's AI capabilities to clients, partners, and advisors, outlining our approach, roadmap, and unique value proposition.Spot high-impact opportunities where AI can effectively address client challenges or provide competitive advantages.Set technical standards, patterns, and best practices that guide engineering decisions across the organization.Technical ExecutionDesign and develop production AI systems, including applications for LLM, RAG pipelines, semantic search, and traditional ML models.Create rigorous evaluation frameworks, experimentation methodologies, and monitoring systems that ensure AI solutions deliver reliable, measurable outcomes.Integrate classical data science techniques (statistical modeling, experimentation design, feature engineering) with modern generative AI methods.Oversee the technical quality of AI systems end-to-end, from data pipelines and model deployment to production observability.Cross-Functional CollaborationWork alongside product management to translate business needs into technical solutions and validate them against customer expectations.Guide and uplift the AI/data science team (3-5 engineers), enhancing the technical proficiency across the board.

Join our innovative team at Flagship Pioneering Inc. as a (Senior) Machine Learning Engineer, where you'll be at the forefront of developing cutting-edge artificial intelligence solutions. In this role, you will leverage your expertise in machine learning algorithms and data analysis to drive impactful results across various projects. Collaborate with a diverse team of scientists and engineers to create scalable models that can be implemented in real-world applications.

The Machine Learning Data Services team is on the lookout for an accomplished and driven Staff Technical Lead Manager (TLM) to spearhead the design and operation of our foundational machine learning data infrastructure. This pivotal role necessitates a unique combination of extensive technical know-how in machine learning systems, large-scale data processing, and a proven track record in leadership to guide both team members and technical initiatives. Our ML Data Services team plays a vital role in ensuring the provision of dependable, high-quality, and easily accessible data for all machine learning applications. As custodians of the 'Data Engineering' phase within the machine learning lifecycle, we oversee the most crucial input to any ML system: the data itself. Our robust infrastructure supports numerous data pipelines and vast datasets, directly contributing to the success of our flagship products. Key Responsibilities Technical Leadership and Strategy Define and drive the technical roadmap and long-term strategy for the ML Data Services platform, ensuring alignment with the evolving needs of machine learning engineers across the organization. Act as the principal technical authority for the team, architecting sophisticated, highly scalable, and fault-tolerant systems for data ingestion, transformation, and serving. Lead the technical vision for acquiring a diverse range of training data sources, incorporating ingestion from real-world drive logs, resimulation outputs, auto-labeling pipelines, and human labeling efforts. Initiate and oversee the implementation of rigorous mechanisms for monitoring data quality, tracking data lineage, and ensuring comprehensive data governance and compliance. Deliver hands-on technical guidance, conduct design reviews, and mentor engineers, ensuring adherence to best practices, high code quality, and operational excellence. Team Management and Growth Supervise a team of software engineers, encompassing performance evaluations, career development, coaching, and mentoring. Serve as a hybrid technical leader-manager, balancing 50% management responsibilities with 50% technical leadership, architecture, and hands-on coding/design. Lead recruitment efforts to attract, onboard, and retain top engineering talent while fostering a diverse and inclusive team culture. Promote engineering excellence, operational rigor, and a strong sense of ownership within the team. Collaboration and Execution Engage closely with ML model teams and Infrastructure teams to grasp data needs and deliver solutions that expedite machine learning development cycles. Steer the execution of significant projects, ensuring timely delivery and alignment with business objectives. Articulate complex technical concepts clearly to various stakeholders.

Join Our Team: At Motional, our mission is to revolutionize transportation through the development of autonomous vehicles. As part of our cutting-edge team, you will collaborate with top-tier machine learning engineers and research scientists to bring self-driving technology to life, making a positive impact on society. We focus on creating advanced tech stacks that analyze complex driving environments and generate safe, comfortable trajectories for our robotaxi services using state-of-the-art deep neural networks.Your Role:As a Senior Machine Learning Engineer specializing in prediction, you will play a pivotal role in training, evaluating, and deploying our machine learning models for scene understanding, behavior prediction, and planning. Your daily responsibilities will include:Designing and conducting high-impact experiments through collaboration with fellow machine learning engineers.Prototyping and enhancing metrics for assessing the performance of our behavioral models across various scenarios and capabilities.Ensuring timely updates of major models, including detailed analysis of offline and on-road evaluation data.Staying abreast of the latest industry trends, proposing innovative architectures and network designs based on current literature.Maintaining a robust training and evaluation codebase, particularly focusing on dataset generation and evaluation.

About Air Space IntelligenceAt Air Space Intelligence (ASI), we are at the forefront of mission-critical technology that transforms decision-making across the aviation, defense, energy, and critical infrastructure sectors. Supported by leading investors such as Andreessen Horowitz, Spark Capital, and Renegade Partners, ASI empowers organizations to achieve operational decision superiority—reducing days of analysis into mere seconds of actionable insights. Join us as we redefine the limits of possibility.Your Role:As a vital member of our engineering team, you will be responsible for designing and deploying robust production-grade systems that seamlessly integrate machine learning models into scalable software pipelines. You will develop and implement features that utilize machine learning to address real-world optimization and prediction challenges, leveraging cutting-edge technologies such as Kubernetes, AWS, and MLOps tools. Your approach will reflect a software engineer's mindset—focusing on reliability, maintainability, and high performance at scale.Our Core Values:Expertise in Python, along with experience in production machine learning tools and frameworks like TensorFlow, PyTorch, and scikit-learn.Hands-on experience with large language models (LLMs) in production settings, encompassing prompt engineering, fine-tuning, retrieval-augmented generation (RAG) systems, and frameworks such as LangChain.Deep understanding of data structures, algorithms, and software engineering principles.Familiarity with classical machine learning, deep learning with an emphasis on transformer architectures, and MLOps methodologies.Experience in building and maintaining scalable, reliable machine learning systems with comprehensive data pipelines, including proficiency with Apache Beam, MLflow, and similar tools.Dedication to high-quality machine learning engineering practices, including data versioning, experiment tracking, model governance, and automated testing pipelines.A preference for simplicity and clarity when tackling complex challenges.An inquisitive mind and eagerness to collaborate with others.Strong communication skills and the ability to work effectively across cross-functional teams.Our Hiring Philosophy:We view the interview process not merely as a screening test but as an opportunity to simulate what it will be like to work together. Our hiring process is tailored around you.

Join the innovative Offline Driving Intelligence (ODIN) team at Zoox, where we harness cutting-edge AI technologies to develop algorithms that offer a profound understanding of our environment. Our focus is on creating robust models that operate offline, guiding the development of our autonomous robot to navigate safely and efficiently through intricate settings.As a pivotal member of the ODIN team, you will be at the forefront of engineering advanced multimodal large language models that significantly enhance environmental perception. You will refine these models for off-vehicle analysis while collaborating closely with our onboard team to maximize their performance in our robotaxi service. Your contributions will ensure timely hazard detection and accurate interpretation of driving constraints with minimal latency. Collaborating with elite engineers and researchers, you will utilize high-quality sensor data and state-of-the-art infrastructure to validate your algorithms in real-world scenarios, making a direct impact on the productivity, safety, and functionality of Zoox's autonomous systems.

Join DigitalOcean as a Senior Engineer I in Observability, where you will play a crucial role in enhancing our platform's visibility and performance. You will work closely with cross-functional teams to build and optimize observability tools, ensuring that our infrastructure runs smoothly and efficiently. This is an opportunity to leverage your technical expertise to drive innovation and improve our customer experience.

Klaviyo is looking for an Engineering Manager to guide the Machine Learning Platform team in Boston, MA. This position leads a group of engineers focused on building and improving machine learning solutions that support Klaviyo’s products. Role overview The Engineering Manager will oversee the development and enhancement of the machine learning platform. This includes working closely with cross-functional partners to ensure the platform remains scalable and high-performing as it evolves. What you will do Lead and mentor a team of engineers working on machine learning initiatives Collaborate with other teams to deliver impactful projects Drive improvements in platform scalability and performance Help shape the direction of Klaviyo’s machine learning capabilities Requirements Experience managing engineering teams Background in machine learning platforms or related technologies Strong collaboration skills with cross-functional groups

WHOOP, a leading innovator in health and fitness wearables, is dedicated to enhancing human performance and extending healthspan. We empower our members by providing profound insights into their physical condition and daily habits. As part of our Health team, you will contribute to developing cutting-edge algorithms and features that enhance our health offerings. This role involves pioneering work in various domains including women’s health, medical-grade metrics, wellness monitoring, and longevity research, merging continuous physiological data with clinical research to create impactful health solutions. As a Senior Machine Learning Engineer, you will design, construct, and implement ML systems that deliver personalized health metrics to millions. Your expertise will bridge data science, backend engineering, and cloud infrastructure, focusing on robust, scalable ML solutions derived from physiological and behavioral data. Strong coding, system design skills, and the ability to produce production-ready ML services are essential for success in this position.

Join our innovative team at hike-medical as a Senior Machine Learning Engineer. In this pivotal role, you will harness state-of-the-art machine learning techniques to develop cutting-edge healthcare solutions. Your expertise will drive the design and implementation of robust algorithms, enhancing the efficiency and accuracy of our medical technologies.

At Lendbuzz, we are committed to transforming financial opportunities through personalized and equitable solutions. Our innovative technology empowers underserved borrowers to gain better access to credit. We foster a culture of diversity and inclusion, valuing independent thought and critical analysis.We are currently looking for a passionate Machine Learning Engineer Co-op to contribute to our team, particularly in the development of our sales and collection AI chatbot initiatives within our Language Understanding and Semantic Analysis group. The ideal candidate will be curious about the auto loan industry and eager to create practical applications using Large Language Models (LLMs). This role offers the opportunity to gain hands-on experience in large-scale data annotation, data cleaning, multilingual model evaluation, and product design.

Join the innovative Offline Driving Intelligence (ODIN) team at Zoox, where we harness cutting-edge AI technologies to develop algorithms that enhance our understanding of the world around us. Our focus is on utilizing large models primarily in offline environments to create impactful solutions for our self-driving robot, facilitating safe and efficient navigation through complex settings.As a vital member of the ODIN team, you will be tasked with the development of advanced multimodal large language models that significantly improve environmental comprehension. Your role will involve fine-tuning these models for off-vehicle analysis, collaborating closely with the onboard engineering team to implement solutions that enable our robotaxi platform to swiftly identify hazards and interpret driving restrictions with minimal latency. Working alongside a team of esteemed engineers and researchers, you will leverage high-quality sensor data and state-of-the-art infrastructure to validate your algorithms in real-world scenarios, making a direct contribution to the productivity, safety, and capabilities of Zoox's autonomous systems.

Join Our Team at Motional: As a Senior Machine Learning Engineer specializing in ML Planning, you will collaborate with top-tier machine learning engineers and research scientists to revolutionize the future of self-driving vehicles. Your work will contribute to creating a positive social impact by enhancing the technology that interprets complex driving environments and devises safe, efficient driving strategies for our robotaxi fleet.Your Responsibilities:Lead the design and execution of innovative experiments, leveraging insights from cross-functional teams of machine learning engineers.Prototype and implement advanced metrics to assess the performance of our behavioral models across diverse scenarios and capabilities of our robotaxi.Ensure timely deployment of significant model updates, analyzing both offline and on-road evaluation results.Stay abreast of industry trends and propose novel architectures and network designs informed by the latest research.Maintain a robust and high-quality training and evaluation codebase, focusing on dataset generation and assessment.

About UsCyvl is an innovative tech startup based in Boston, reshaping the way governments map and manage their transportation infrastructure. Our cutting-edge hardware and software solutions utilize advanced 3D mapping sensors to collect LiDAR, imagery, and GPS data, seamlessly integrated into our clients' vehicles. This data is processed through our AI-driven cloud pipelines to produce actionable geospatial insights that optimize city operations, saving time, money, and resources.At Cyvl, our mission is to empower governments in constructing and maintaining public infrastructure they can be proud of. We accelerate decision-making through our sensors and Infrastructure Intelligence Platform. This role is pivotal for our intelligence extraction process, responsible for transforming millions of street-level images and dense LiDAR point clouds into structured, queryable knowledge about the physical realm.We are a dynamic and rapidly growing team that believes in addressing real-world challenges with authenticity, simplicity, and courage. Each team member is empowered to take initiative, achieve results, and make a lasting impact on the communities we serve.

At Compass, we are driven by our mission to help everyone find their perfect place in the world. Established in 2012, we are transforming the real estate landscape with our innovative end-to-end platform, empowering residential real estate agents to provide exceptional service to both sellers and buyers.As a Staff Software Engineer within the Transaction Journey organization, you will leverage your expertise in microservices architecture to develop impactful products for our customers. You will take the lead in designing and developing services that enhance the consumer experience while supporting the expansion of the world’s most scalable brokerage. You will guide a team in creating a collaborative transaction management platform that simplifies the home buying and selling process, enabling agents to efficiently manage transactions from initial contact to closing, including accessing local forms, sending documents for electronic signatures, and managing offers for buyers and sellers.We are looking for an engineer who is passionate about crafting well-defined APIs that are user-friendly. Your insights in product and business decisions are invaluable, and you have a strong desire to learn and share knowledge with your colleagues. Your communication skills are exceptional, and you prioritize understanding others before conveying your ideas.Your design systems are fault-tolerant, scalable, highly available, well-tested, and adhere to best practices such as the single responsibility principle.Your code is modular and reusable, and you take pride in delivering robust, well-tested, peer-reviewed code that follows industry best practices. You hold strong opinions regarding code structure, style, and development processes.At Compass, You Will:Provide strategic direction and ownership of Compass software architecture.Build, develop, and scale the platform that empowers real estate professionals, buyers, and sellers.Become a domain expert in real estate technology, serving as an empathetic partner to our customers.Inspire, recruit, and mentor fellow engineers.Lead the architecture of our distributed microservices environment.Engage in a scalable engineering culture that utilizes modern principles of decoupled systems and automated CI/CD/testing/monitoring to enhance efficiencies.Execute standard agile development methodologies.Join a dynamic team with high visibility and exciting projects on the horizon.

Join the Innovative Team at SunoAt Suno, we are not just a music company; we are a creative powerhouse dedicated to igniting imagination through the magic of sound. Harnessing the capabilities of the world’s most advanced AI music model, we provide an exceptional creative platform, including our groundbreaking generative audio workstation, Suno Studio. We believe in empowering everyone—from shower singers to seasoned artists—to create, share, and discover music, making musical expression accessible to all.Role OverviewWe are seeking passionate early members of our research team to collaborate closely with our founding members. You will play a pivotal role in shaping the future of our machine learning initiatives by making critical technical decisions on the development and deployment of our cutting-edge ML models, boasting an H100/scientist ratio exceeding 100x.Explore the Suno job posting here!