Evaluations Engineer, Applied AI

Join Scale AI's ML platform team (RLXF) as a Machine Learning Research Engineer, where you will play a pivotal role in developing our advanced distributed framework for training and inference of large language models. This platform is vital for enabling machine learning engineers, researchers, data scientists, and operators to conduct rapid and automated training, as well as evaluation of LLMs and data quality.At Scale, we occupy a unique position in the AI landscape, serving as an essential provider of training and evaluation data along with comprehensive solutions for the entire ML lifecycle. You will collaborate closely with Scale's ML teams and researchers to enhance the foundational platform that underpins our ML research and development initiatives. Your contributions will be crucial in optimizing the platform to support the next generation of LLM training, inference, and data curation.If you are passionate about driving the future of AI through groundbreaking innovations, we want to hear from you!

About This Role Join Scale AI's Applied ML team as a Machine Learning Research Engineer, focusing on the development of advanced data infrastructure for leading agentic large language models (LLMs) such as ChatGPT, Gemini, and Llama. You will be responsible for architecting scalable multi-agent systems aimed at validating agentic reasoning and behaviors, enhancing human expertise, and conducting research to address real-world agent reliability failures, even in the face of strong benchmarks. Your contributions will directly impact the deployment of production fixes. This role is ideal for exceptional engineers who possess a deep research rigor and a strong commitment to creating practical, high-impact systems. You will iterate rapidly using data, leverage AI tools for accelerated development, and collaborate closely with engineering, product, and research teams. If you have a knack for transforming cutting-edge agent research into dependable deployed systems, we would love to hear from you.

Join David AIAt David AI, we are pioneering the audio data research landscape. Our research and development approach to data ensures that we deliver datasets with the same precision and rigor that leading AI labs apply to their models. Our mission is to seamlessly integrate AI into everyday life, leveraging audio as a key channel. As we witness advancements in audio AI and the emergence of new use cases, we recognize that high-quality training data is the critical component. This is where David AI steps in.Founded in 2024 by a group of former engineers and operators from Scale AI, we have rapidly established partnerships with major FAANG companies and AI labs. Recently, we secured a $50M Series B funding round from prominent investors including Meritech, NVIDIA, Jack Altman (Alt Capital), Amplify Partners, and First Round Capital.Our team is sharp, humble, and ambitious. We are on the lookout for talented individuals in research, engineering, product management, and operations to join us in our mission to redefine the audio AI landscape.About Our Machine Learning TeamOur Machine Learning team operates at the forefront of innovative research and practical application, transforming raw audio into high-quality data for top AI labs and enterprises. We manage the entire machine learning lifecycle—from exploring novel speech processing algorithms to deploying models that handle terabytes of audio data daily.Your RoleAs an Applied ML Engineer at David AI, you will develop state-of-the-art speech and audio models, establish production inference systems, and create robust pipelines that demonstrate the true potential of high-quality data.Key ResponsibilitiesResearch and Design: Create solutions using advanced signal processing algorithms and cutting-edge ML models tailored for speech and audio applications.Development: Build production-grade inference algorithms, pipelines, and APIs in collaboration with cross-functional teams to extract valuable insights for our clients.Collaboration: Work alongside our Operations team to gather valuable training and evaluation datasets to enhance our model quality.Architecture: Design systems that ensure durable and resilient inference and evaluations.

About UsAt Applied Compute, we are pioneering the development of Specific Intelligence for enterprises, creating agents that continuously learn from a company’s processes, data, expertise, and objectives. Our mission is to bridge the gap between isolated AI capabilities and their effective application within real business environments. Traditional AI systems often fall short as they lack the ability to adapt based on feedback. Our innovative continual learning layer captures context, memory, and decision-making processes across the enterprise, enabling specialized agents to engage in meaningful work.What Excites Us: We operate at the exciting intersection of product development and cutting-edge research. Our product team designs the platform that empowers a new generation of digital coworkers, while our research team drives advancements in post-training and reinforcement learning to enhance user experiences. As an applied research engineer, you will work directly with clients to implement models in production, combining robust product development with deep research insights to facilitate AI integration in enterprises.Meet Our Team: Our diverse team consists of engineers, researchers, and operators, many of whom are former founders. We have previously built reinforcement learning infrastructure at OpenAI, established data foundations at Scale AI, and contributed to significant systems at companies like Together, Two Sigma, and Watershed. We collaborate with Fortune 50 clients, including DoorDash, Mercor, and Cognition, and are proud to be backed by reputable investors such as Benchmark, Sequoia, and Lux.Who Thrives Here: We seek individuals who are passionate about applying innovative research and complex systems to solve real-world challenges. You should feel comfortable navigating new environments rapidly—be it a fresh codebase, a client’s data architecture, or an unfamiliar problem domain. A genuine enjoyment for customer interaction, empathy, and a deep understanding of their operational workflows are essential. Candidates with entrepreneurial backgrounds, extensive side projects, or a proven track record of end-to-end ownership typically excel in our environment.

OverviewPluralis Research is at the forefront of Protocol Learning, innovating a decentralized approach to train and deploy AI models that democratizes access beyond just well-funded corporations. By aggregating computational resources from diverse participants, we incentivize collaboration while safeguarding against centralized control of model weights, paving the way for a truly open and cooperative environment for advanced AI.We are seeking a talented Machine Learning Training Platform Engineer to design, develop, and scale the core infrastructure that powers our decentralized ML training platform. In this role, you will have ownership over essential systems including infrastructure orchestration, distributed computing, and service integration, facilitating ongoing experimentation and large-scale model training.ResponsibilitiesMulti-Cloud Infrastructure: Create resource management systems that provision and orchestrate computing resources across AWS, GCP, and Azure using infrastructure-as-code tools like Pulumi or Terraform. Manage dynamic scaling, state synchronization, and concurrent operations across hundreds of diverse nodes.Distributed Training Systems: Design fault-tolerant infrastructure for distributed machine learning, including GPU clusters, NVIDIA runtime, S3 checkpointing, large dataset management and streaming, health monitoring, and resilient retry strategies.Real-World Networking: Develop systems that simulate and manage real-world network conditions—such as bandwidth shaping, latency injection, and packet loss—while accommodating dynamic node churn and ensuring efficient data flow across workers with varying connectivity, as our training occurs on consumer nodes and non-co-located infrastructure.

OverviewPluralis Research is at the forefront of innovation in Protocol Learning, specializing in the collaborative training of foundational models. Our approach ensures that no single participant ever has or can obtain a complete version of the model. This initiative aims to create community-driven, collectively owned frontier models that operate on self-sustaining economic principles.We are seeking experienced Senior or Staff Machine Learning Engineers with over 5 years of expertise in distributed systems and large-scale machine learning training. In this role, you will design and implement a groundbreaking substrate for training distributed ML models that function effectively over consumer-grade internet connections.

Join Our Innovative TeamAt OpenAI, we are pioneering the field of artificial intelligence, empowering innovation and shaping the future through transformative research. Our mission is to democratize AI, ensuring its benefits are accessible to all. We are on the lookout for forward-thinking Research Engineers to join our Applied Group, where you will convert groundbreaking research into practical applications that can revolutionize industries, enhance human creativity, and tackle complex challenges.Your Impactful RoleAs a Research Engineer within OpenAI's Applied Group, you will collaborate with some of the brightest minds in AI. Your work will involve deploying cutting-edge models in production settings, transforming theoretical breakthroughs into impactful solutions. If you are passionate about making AI technology accessible and effective, this is your opportunity to leave a significant impact.In this role, you will:Innovate and Deploy: Create and implement advanced machine learning models addressing real-world issues. Translate OpenAI's research from theory to practice, developing AI-driven applications that make a meaningful difference.Collaborate with Experts: Engage closely with researchers, software engineers, and product managers to comprehend intricate business challenges and deliver AI-based solutions. Become part of a vibrant team where creativity and ideas flourish.Optimize and Scale: Develop scalable data pipelines, fine-tune models for peak performance and precision, and ensure readiness for production. Contribute to projects that leverage state-of-the-art technology and innovative methodologies.Learn and Lead: Stay at the forefront of advancements in machine learning and AI. Participate in code reviews, share insights, and exemplify best practices to maintain high standards in engineering.Make a Difference: Oversee and maintain deployed models, ensuring they consistently deliver value. Your contributions will directly shape how AI benefits individuals, businesses, and society as a whole.You may excel in this position if you possess:A Master's or PhD in Computer Science, Machine Learning, Data Science, or a related discipline.Proven experience in deep learning and transformer models.Expertise with frameworks such as PyTorch or TensorFlow.A robust understanding of data structures, algorithms, and software engineering principles.Experience with cloud platforms and deploying machine learning models in production.

Join Our Team at MacroscopeAt Macroscope, we are dedicated to being the definitive source of truth for any software development company. Our mission is to empower leaders with clarity and provide engineers with the time they need to innovate.We enable leaders to gain insights into the evolution of their products and codebases—tracking changes, understanding team contributions, and identifying progress—all grounded in the ultimate source of truth: the code itself.Founded by experienced entrepreneurs who have successfully built and sold multiple companies, and held executive positions in public tech firms, we are backed by top-tier venture capital firms such as Lightspeed Venture Partners, Thrive Capital, Google Ventures, and Adverb.The RoleWe are seeking a Senior Applied Machine Learning Engineer who will be responsible for designing, developing, and optimizing the ML and AI systems that drive our core offerings. You will have full ownership of the systems, overseeing everything from data collection and evaluation to model experimentation and large-scale production deployment.This cross-functional position entails leading the ML/AI lifecycle for one of our most vital features: AI Code Review. Collaborating closely with our co-founders, you will make pivotal decisions that shape our product's development—ranging from building high-quality datasets to interpreting experimental results and enhancing model performance architecture. Additionally, you will play a significant role in crafting and implementing software that seamlessly integrates our models with our backend applications and user experience, offering a unique opportunity to influence our product's evolution significantly.Technology Stack: Typescript/React (frontend), Golang (backend), Temporal, Google Cloud (GCP), Postgres, Terraform, and custom-built AST "code walkers" in several programming languages including Golang, Typescript, Swift, Python, and Rust.

Join us at Foxglove, where we are revolutionizing the robotics industry by building robust data infrastructure for real-world applications.As robotics transitions from research environments to practical implementations in factories, warehouses, vehicles, and field operations, data becomes essential for engineers to troubleshoot failures, understand unexpected behaviors, and enhance robotic systems.At Foxglove, we provide the observability, visualization, and data infrastructure that enable robotics and autonomous systems teams to efficiently ingest, store, query, replay, and analyze extensive volumes of multimodal sensor data from live systems and production fleets.About the RoleWe are seeking a talented Applied Machine Learning Engineer with strong infrastructure insights to design, deploy, and scale the machine learning systems that power our data platform. In this impactful role, you will be responsible for optimizing production ML infrastructure—from enhancing inference pipeline throughput to establishing training and evaluation workflows. You will focus on high-priority challenges, such as developing retrieval applications for petabyte-scale multimodal robotics data, utilizing cutting-edge models to create high-performance search and data mining products, and fostering an internal ML flywheel for rapid iteration. This is a hands-on, application-driven position rather than a research-focused role.Key ResponsibilitiesDeploy and manage inference infrastructure for production ML workloads, focusing on model serving, scalability, and cost efficiency.Build and oversee vector database integrations and embedding applications to facilitate semantic search across various multimodal robotics data types (image, video, point cloud, and time series).Design and implement evaluation and training infrastructure to enhance model performance rapidly.Lead cloud architecture decisions and tools to optimize inference latency, throughput, cost, and reliability at scale.Collaborate closely with product engineers to deliver application-driven ML features that empower developers at the forefront of robotics and physical AI, steering clear of prototype experiments.Identify appropriate off-the-shelf solutions for production and determine when to build versus buy.

Join Our Innovative Team at ArcadeAt Arcade, we are revolutionizing the way physical products are created with our cutting-edge AI platform. We empower individuals to turn their creative ideas into tangible products seamlessly, utilizing natural language and generative AI. Our mission is to democratize product design, making it as effortless as sharing a post online.Backed by a remarkable $42M in funding from industry-leading investors including Reid Hoffman and Ashton Kutcher, our company is a rising star in the tech landscape. Guided by our founder Mariam Naficy and a team steeped in AI and design expertise, we are at the forefront of a new frontier that merges AI, personal expression, and on-demand manufacturing.Your Role as an Applied AI EngineerWe are on the lookout for an Applied AI Engineer to enhance our generative AI capabilities. This position combines hands-on model development with the integration of advanced AI techniques into our production systems. You will collaborate with diverse teams to conduct research, experiment with models, and implement AI-driven products.

About UsAt Speak, our mission is to revolutionize language learning.Learning a new language can transform lives by unlocking opportunities in diverse cultures, careers, and communities. With over two billion individuals around the globe striving to learn a language, we recognize that traditional one-on-one tutoring remains difficult to access at scale and has seen little innovation over recent decades. Speak is pioneering an AI-driven, human-level tutor accessible right from your pocket, providing a conversation-first experience where learners can practice speaking, receive immediate feedback, and progress through meticulously crafted lessons. Our goal is to facilitate a comprehensive journey from beginner to proficient speaker across various languages.Launched in South Korea in 2019, Speak has quickly become the leading language learning app in the region, now reaching learners across numerous markets and offering instruction in 15+ languages. Supported by over $150 million in venture capital from prestigious investors such as OpenAI, Accel, Founders Fund, and Khosla Ventures, our team is distributed across San Francisco, Seoul, Tokyo, Taipei, and Ljubljana.Role OverviewWe are seeking a skilled Machine Learning Engineer specializing in speech to join our innovative team. In this role, you will take charge of the entire modeling pipeline for speech recognition, encompassing training, experimentation, deployment, and ongoing monitoring. Collaborating closely with Product teams, you will design cutting-edge learning experiences and assess the effectiveness of production models on our users. As part of a nimble and dynamic team, you'll contribute as both a developer and a thought partner on projects related to ASR, assessments, pronunciation improvements, content personalization, and more. This is an exhilarating opportunity to be part of an ML team focused on crafting personalized learning experiences that will transform language education for millions worldwide.

We are seeking a talented Applied Machine Learning Engineer with a comprehensive understanding of the generative media landscape. You should possess an up-to-date knowledge of emerging methodologies and be capable of identifying gaps in the current market. Your role will involve innovating and developing machine learning models that enhance user experiences, requiring both novel training techniques and the fine-tuning of existing models with fresh datasets.

Join the Innovative Team at CodeRabbit!At CodeRabbit, we are at the forefront of research and development, dedicated to creating cutting-edge human-machine collaboration systems. Our mission is to revolutionize code review processes through a next-generation partnership between skilled engineers and sophisticated AI algorithms. By harnessing the power of generative AI, we aim to enhance software development efficiency and quality, achieving results that far exceed individual capabilities.Role OverviewAs an Applied Generative AI Engineer, you will be instrumental in designing, building, and deploying state-of-the-art generative AI systems that drive our code review and developer productivity tools. Your work will center around integrating cutting-edge methodologies such as Retrieval-Augmented Generation (RAG), Reinforcement Learning from Human Feedback (RLHF), and multi-step agentic reasoning into impactful product workflows.Collaboration will be key! You will work closely with engineers, product managers, and technical leads to build intelligent systems that deliver tangible value, transforming how developers write, review, and ship code.Key ResponsibilitiesDesign and enhance LLM-based systems for contextually rich and high-quality code reviews.Create and optimize agentic workflows that reason across various steps and contexts.Develop and manage knowledge base and retrieval pipelines, including chunking, embeddings, and semantic search.Deploy generative AI models into production and monitor their performance.Collaborate with cross-functional teams to ensure AI outputs meet user needs and align with product objectives.Analyze user feedback and usage data to iteratively refine system performance.Implement RLHF, ranking, and reward modeling techniques to enhance response quality over time.Stay updated on the latest advancements in generative AI and apply them to emerging use cases.QualificationsEducation: Bachelor’s degree in Computer Science, Engineering, Artificial Intelligence, or a related field, or equivalent practical experience.Experience: Minimum of 3 years of experience working with machine learning or LLM-based systems in production environments, with at least 2 years dedicated to generative AI.Technical Skills: Proficiency in programming languages and frameworks relevant to AI development.

At Netic, we are revolutionizing the essential services sector with our advanced AI-driven revenue engine, which supports the backbone of the American economy.Backed by $43M in funding from illustrious investors such as Founders Fund, Greylock, Hanabi, and Dylan Field, who spearheaded our Series B, we have empowered our clients to secure hundreds of thousands of jobs across various service industries throughout North America. Our platform has enabled companies to operate with an AI-first approach.Join our innovative team of relentless builders hailing from renowned organizations like Scale, Databricks, HRT, Meta, MIT, Stanford, and Harvard. Together, we are applying frontier AI to solve complex challenges in the physical economy, where data is intricate and the results are both immediate and impactful.As an Applied AI Research Engineer, you will immerse yourself in pioneering research, gain a thorough understanding of the business functions we automate, and lead targeted machine learning projects that yield remarkable outcomes.

At Scale AI, we are at the forefront of artificial intelligence, driving innovation through our advanced data, infrastructure, and tooling that empower the most sophisticated models worldwide. Our teams thrive at the intersection of pioneering research, extensive engineering, and practical deployment, collaborating with leading labs, enterprises, and government entities to explore the vast potential of Generative AI. As AI technology evolves from static models to dynamic, intelligent systems, Scale AI is dedicated to establishing the essential research foundations, evaluation methodologies, and reinforcement learning infrastructure that will shape this transformative era. Join our high-impact research organization, where you will contribute to advancing large language models, post-training evaluation, and agent-based reinforcement learning environments, influencing the future of AI development and implementation. As the Research Scientist Manager, you will spearhead a distinguished team of research scientists and engineers, define the strategic research roadmap, and oversee projects from initial prototyping to final deployment. You will excel in a fast-paced environment, harmonizing deep technical leadership with effective people management, visionary goal setting, and successful delivery.

Join Our Team at Distyl AIDistyl AI is at the forefront of developing advanced, production-grade AI systems that enhance operational workflows for Fortune 500 companies. Through our strategic alliance with OpenAI, proprietary software accelerators, and extensive enterprise AI expertise, we deliver effective AI solutions with rapid results, often in a matter of weeks.Our innovative products have transformed operations for clients across various sectors, including insurance, consumer packaged goods, and non-profit organizations. By joining our dynamic team, you will play a crucial role in helping clients discover, build, and leverage value from their Generative AI investments, frequently for the first time. We prioritize a customer-first approach, focusing on their challenges and holding ourselves accountable for generating financial impact while enhancing user experiences.Distyl is led by accomplished leaders from prestigious companies like Palantir and Apple, with backing from top-tier investors including Lightspeed, Khosla, Coatue, and Dell Technologies Capital, among others.Your RoleAt Distyl, we are redefining the application of AI in enterprise environments. We seek imaginative researchers who aspire to go beyond mere incremental advancements and, instead, aim to innovatively reshape how software is utilized.Our research team comprises individuals from diverse academic backgrounds with robust research accomplishments. We operate in an AI-native manner and thrive in an environment that encourages unconventional thinking.Key ResponsibilitiesSimilar to generating hypotheses in AI for scientific exploration, the System Discovery team focuses on identifying new classes of AI systems. Researchers will investigate various architectures, modalities, and combinations to discover how AI can fundamentally transform work processes.Make connections across different domains to unveil new system archetypes. Engage in experimental problem definitions and develop prototypes that facilitate novel forms of human–AI collaboration.Who You AreBroad Systems Development Experience: You possess a diverse range of experience, having built systems in various paradigms, including retrieval pipelines, reasoning agents, evaluation harnesses, multimodal integrations, or workflow graph systems, and can distill these experiences into innovative solutions.

About HebbiaHebbia is an innovative AI platform designed specifically for investors and bankers, empowering them to generate alpha and unlock new opportunities.Founded in 2020 by George Sivulka and backed by industry leaders like Peter Thiel and Andreessen Horowitz, Hebbia supports investment decisions for major firms including BlackRock, KKR, Carlyle, Centerview, and accounts for 40% of the world's largest asset managers. Our flagship product, Matrix, is recognized for its unparalleled accuracy, speed, and transparency in AI-driven analysis, managing assets exceeding $30 trillion globally.We provide critical insights that give finance professionals a competitive advantage by revealing signals that are invisible to the human eye and identifying hidden opportunities while expediting decision-making with remarkable speed and certainty. We aim to revolutionize the way capital is allocated, risk is mitigated, and value is generated across markets.Hebbia is not just a tool; it is the competitive edge that enhances performance, alpha, and market leadership.The TeamOur Agents team is dedicated to building sophisticated reasoning, copiloting, and retrieval capabilities that unlock significant insights for real-world applications. We develop everything from foundational document understanding features to co-piloting experiences for matrix and extensive, multi-source research. Our proprietary agentic frameworks are designed for scalability, utilizing distributed systems.We focus on creating systems that are not only successful but also reliable, explainable, and adaptable for the vast data our clients encounter. Our mission is to unveil the unknowable unknown for customers worldwide.Our goal is to create a product that becomes indispensable to our users, offering an experience as delightful as their favorite consumer products. We prioritize swift innovation and the development of first-of-their-kind systems.

Join Perplexity as an Applied Machine Learning Engineer and be at the forefront of innovation in artificial intelligence. You will design, develop, and refine advanced AI models that enhance user experiences globally. Your expertise in machine learning will allow you to create scalable solutions for user personalization, query comprehension, and content discovery, catering to the curiosity of millions.Key ResponsibilitiesUtilize cutting-edge ML and LLM techniques to address challenges in:Personalization (LLM memory, context summarization, retrieval, and ranking);Query Understanding (intent modeling, rewriting, agentic decomposition);Content Discovery (feed ranking and surfacing).Conduct thorough evaluations of LLM/ML models through offline and online methods, designing comprehensive experiments and metrics that yield insights into quality and impact.Manage the full model lifecycle from research to deployment, including data analysis, modeling, evaluation, A/B testing, and iterative enhancements.Collaborate with cross-functional teams, including engineers, PMs, data scientists, and designers, to ensure AI implementations yield significant product enhancements.Stay updated on ML/AI advancements by assessing and integrating new research and algorithms into the product lifecycle.Preferred QualificationsOver 5 years of experience in developing and deploying robust ML/AI models for large-scale, user-centric or data-driven applications.Extensive knowledge in deep learning frameworks (PyTorch, TensorFlow, JAX), LLMs, information retrieval, content summarization, recommendation systems, NLP, and ranking.Proficient software engineering skills (Python, production-level codebases, collaborative development).Comprehensive experience across the entire ML lifecycle: data analysis, feature engineering, model development, evaluation, and ongoing monitoring/improvement.Effective collaborator and communicator; thrives in fast-paced, cross-functional environments.Inquisitive, motivated by user/product impact, and passionate about advancing applied ML and AI.Bachelor's, Master's, or PhD in Computer Science, Engineering, or a related field (or equivalent experience).

Artificial Intelligence (AI) is becoming increasingly crucial across all sectors of society. At Scale AI, our mission is to expedite the advancement of AI applications. With nine years of experience, we have established ourselves as the leading AI data foundry, facilitating groundbreaking developments in AI, including generative AI, defense applications, and autonomous vehicles. Following our recent investment from Meta, we are committed to enhancing our capabilities by developing cutting-edge post-training algorithms that are essential for optimizing complex agents in enterprises globally.The Enterprise ML Research Lab is at the forefront of this AI revolution. We are dedicated to crafting a suite of proprietary research tools and resources that cater to all of our enterprise clients. As a Machine Learning Research Engineer focusing on Agents, you will apply our Agent Reinforcement Learning (RL) training and building algorithms to real-world enterprise datasets across our clients and benchmarks. Your role will involve developing top-tier Agents that achieve state-of-the-art results through a blend of post-training and agent-building algorithms.If you are passionate about influencing the trajectory of the modern Generative AI movement, we would love to hear from you!