Research Engineer in Model Evaluations

Zyphra is an innovative leader in artificial intelligence, located in the heart of San Francisco, California.Role Overview:As a Research Engineer specializing in Language Model Pre-Training, you will play a pivotal role in defining our language model strategy through comprehensive pretraining development. Your close collaboration with our pretraining team will ensure that your insights contribute to the advancement of our next-generation models.Key Responsibilities:Conduct large-scale training runs and implement model parallelization techniques.Optimize the performance of our pretraining stack.Oversee dataset collection, processing, and evaluation.Research architecture and methodologies, including optimizer ablations.Qualifications:Demonstrated engineering prowess in developing reliable and robust systems.A quick learner with a passion for implementing innovative ideas.Exceptional communication and collaboration skills, capable of working effectively on both research and engineering implementations at scale.Preferred Skills:Profound expertise in addressing machine learning challenges and training models.Experience training on large-scale (multi-node) GPU clusters.In-depth understanding of model training pipelines, including model/data parallelism and distributed optimizers.Strong methodology for conducting rigorous ablations and hypothesis testing.Familiarity with large-scale, high-performance data processing pipelines.High proficiency in PyTorch and Python programming.Ability to navigate and understand extensive pre-existing codebases swiftly.Published research in machine learning in reputable venues is an advantage.Postgraduate degree in a relevant scientific field (Computer Science, Electrical Engineering, Mathematics, Physics).Why Join Zyphra?We value a research methodology that emphasizes thoughtful, methodical progress towards ambitious objectives. Both deep research and engineering excellence are given equal importance.Join us in an environment that fosters innovation, collaboration, and professional growth.

At Thinking Machines Lab, we are dedicated to empowering humanity through the advancement of collaborative general intelligence. Our vision is to create a future where everyone can harness the power of AI to meet their individual needs and aspirations.Our team is composed of passionate scientists, engineers, and innovators who have developed some of the most influential AI technologies, such as ChatGPT and Character.ai, as well as cutting-edge open-weight models like Mistral and acclaimed open-source projects including PyTorch, OpenAI Gym, Fairseq, and Segment Anything.About the RoleThe role of Pre-Training Researcher is pivotal to our strategic roadmap, focused on enhancing our understanding of how large models learn from data. You will investigate novel pre-training methodologies, architectures, and learning objectives aimed at making model training more efficient, robust, and aligned with human values.This position combines fundamental research with practical engineering, as we seamlessly integrate both disciplines within our team. You will be expected to produce high-performance code and engage with technical literature. This is an ideal opportunity for individuals who thrive on theoretical exploration as well as hands-on experimentation, and who aspire to influence the foundational methods by which AI learns.This is an evergreen role, meaning we keep this position open to welcome expressions of interest in this research field. We receive numerous applications, and while there may not always be an immediate fit, we encourage you to apply. We consistently review applications and will reach out as new opportunities arise. If you gain additional experience, you are welcome to reapply, but please limit your applications to once every six months. We may also post specific openings for project or team needs, where direct applications are welcome in addition to this evergreen role.What You’ll DoResearch and innovate new methodologies for pre-training.Engage in areas such as scaling, architecture, algorithms, or optimization of large-scale training runs based on your research interests and expertise.Design data curricula and sampling strategies that enhance learning dynamics and model generalization.Collaborate with infrastructure and data teams to conduct large-scale experiments in an efficient and reproducible manner.Publish and present research that propels the entire community forward, sharing code, datasets, and insights to accelerate progress across both industry and academia.

Overview: At Spellbrush, we are revolutionizing the gaming experience by crafting a 3D first-person adventure game where an AI companion plays a pivotal role. Imagine MiSide enhanced with language learning models, seamlessly integrated into gameplay rather than merely serving as a role-play chatbot.About UsAt Spellbrush, we are dedicated to creating exceptional anime games, and we proudly stand as a global leader in generative AI. Our flagship project is niji・journey.Our mission is straightforward: to use AI to animate characters and redefine narrative-driven gaming.What We’re CreatingWe have engineered an innovative in-house LLM storytelling system that merges AI, narrative, and gameplay, transcending the limitations of conventional chat-only encounters.This results in an AI companion that collaborates with players in solving puzzles, retains memories across different worlds, and alters the progression of each chapter.About The RoleJoin our elite team to redefine video game experiences. Collaborate with leading minds in the industry, including the creator of Warudo and Cytoid, as well as a Google DeepMind veteran behind Project Astra, and top-tier AI researchers.As an integral early member of this team, you will enjoy significant artistic and research autonomy in shaping what could be the next era of LLM-driven storytelling.

Join Our Innovative TeamTavus is a forward-thinking research laboratory at the forefront of human-computer interaction. We are dedicated to creating AI Humans, a revolutionary interface that bridges the gap between individuals and machines, eliminating the friction commonly encountered in today's technology. Our advanced human simulation models empower machines to see, hear, respond, and even exhibit lifelike appearances, facilitating genuine, face-to-face interactions. By merging human emotional intelligence with machine efficiency, our AI Humans serve as reliable, empathetic agents available around the clock, in any language, tailored to user needs.Picture a therapist within reach for everyone, a personal trainer that fits seamlessly into your schedule, or a comprehensive team of medical assistants providing focused attention to every patient. With Tavus, individuals and organizations can develop AI Humans to foster connection, understanding, and responsiveness on a grand scale.As a Series A startup, we are backed by prestigious investors such as Sequoia Capital, Y Combinator, and Scale Venture Partners. Join us in crafting a future where humans and machines communicate effortlessly.Your RoleWe seek an AI Researcher to join our dynamic AI team and explore the frontiers of large language modeling within conversational AI. If you excel in fast-paced startup settings, enjoy experimenting with innovative ideas, and are eager to see your contributions realized in production, you will thrive here.Your Mission Conduct in-depth research on large language modeling and its adaptation for Conversational Avatars (e.g., Neural Avatars, Talking-Heads).Create methodologies to effectively model both verbal and non-verbal communication, dynamically adjusting avatar behaviors in real-time.Experiment with fine-tuning, adaptation, and conditioning techniques to enhance the expressiveness, control, and task specificity of LLMs.Collaborate with the Applied ML team to transition research from prototype to full-scale deployment.Stay informed on the latest advancements and contribute to defining the next breakthroughs.

Join Perplexity as a Research Engineering Manager, where you will spearhead a team of exceptional AI researchers and engineers dedicated to crafting the advanced models that power our innovative products. Our talented team has pioneered some of the most sophisticated models in agentic research, query understanding, and other critical domains that demand precision and depth. As we broaden our user base and expand our product offerings, our proprietary models are increasingly essential for delivering a premium experience to the world's most discerning users.You will explore our extensive datasets of conversational and agentic queries, applying state-of-the-art training methodologies to enhance AI model performance. Through proactive technical and organizational leadership, you will empower your team to create cutting-edge models for the applications that are most significant to our business and our users.

Join Our Innovative TeamAt OpenAI, our Training team is at the forefront of developing advanced language models that drive our research and products, getting us closer to achieving Artificial General Intelligence (AGI). This mission demands a blend of cutting-edge research to enhance our architecture, datasets, and optimization methods, alongside strategic long-term initiatives that boost the efficiency and capabilities of future models. We ensure that our models, including recent breakthroughs like GPT-4-Turbo and GPT-4o, adhere to the highest standards of excellence.Your RoleAs an integral member of our architecture team, you will spearhead architectural advancements for OpenAI’s leading models, enhancing their intelligence and efficiency while introducing novel capabilities. Your expertise in large language model (LLM) architectures and model inference will be crucial as you adopt a hands-on, empirical approach to problem-solving. Whether brainstorming creative breakthroughs, refining foundational systems, designing evaluations, or diagnosing performance issues, your diverse skill set will be invaluable.This position is located in San Francisco, where we embrace a hybrid work environment of three days in the office each week, and we provide relocation support for new hires.Your Key Responsibilities:Innovate, prototype, and upscale new architectures to elevate model intelligence.Conduct and evaluate experiments both independently and collaboratively.Analyze, debug, and enhance both model performance and computational efficiency.Contribute to the development of training and inference infrastructure.Who You Are:You possess experience with significant contributions to major LLM training projects.You excel at independently evaluating and enhancing deep learning architectures.You are driven to responsibly implement LLMs in real-world applications.You are knowledgeable about state-of-the-art transformer modifications aimed at improving efficiency.About OpenAIOpenAI is a pioneering AI research and deployment organization committed to ensuring that artificial general intelligence benefits humanity. We focus on developing safe and effective AI technologies that empower individuals and organizations across the globe.

Who Are We?Postman is the premier API platform, empowering over 45 million developers and 500,000 organizations, including 98% of the Fortune 500. We are dedicated to fostering an API-first world by simplifying the entire API lifecycle and enhancing collaboration, enabling users to create superior APIs swiftly.Headquartered in San Francisco, we have offices in major cities like Boston, New York, Austin, Tokyo, London, and Bangalore, where our journey began. As a privately held entity, we have garnered support from notable investors including Battery Ventures, BOND, Coatue, CRV, Insight Partners, and Nexus Venture Partners. Discover more at postman.com or connect with us on X via @getpostman.P.S: We highly recommend exploring The "API-First World" graphic novel to gain insights into our vision at Postman.The OpportunityAs an Applied Scientist focusing on Small Language Models and AI Training, you will spearhead research and development initiatives aimed at crafting efficient, high-performance language models for real-world applications. Collaborating closely with research, engineering, and product teams, you will enhance model training techniques, optimize architectures, and scale AI solutions. Your contributions will play a crucial role in creating AI systems that are safe, interpretable, and impactful across various applications.

Zyphra is an innovative artificial intelligence company located in the heart of San Francisco, California.The Opportunity:Join our dynamic team as a Research Engineer - Audio & Speech Models, where you will play a pivotal role in advancing Zyphra’s Audio Team. You will be instrumental in developing cutting-edge open-source text-to-speech and audio models. Your contributions will span the full spectrum of the model training process, from data collection and processing to the design of innovative architectures and training approaches.Your Responsibilities:Conduct large-scale audio training operationsOptimize the performance of our training infrastructureCollect, process, and evaluate audio datasetsImplement architectural and methodological improvements through rigorous testingWhat We Seek:A strong research mindset with the ability to navigate projects from ideation to implementation and documentation.Proficiency in rapid prototyping and implementation, allowing for swift experimentation.Effective collaboration skills in a fast-paced research environment.A quick learner who is eager to embrace and implement new concepts.Excellent communication abilities, enabling you to contribute to both research and engineering tasks at scale.Preferred Qualifications:Expertise in training audio models, such as text-to-speech, ASR, speech-to-speech, or emotion recognition.Experience with training audio autoencoders.Solid understanding of signal processing, particularly in audio.Familiarity with diffusion models, consistency models, or GANs.Experience with large-scale (multi-node) GPU training environments.Strong understanding of experimental methodologies for conducting rigorous tests and ablations.Interest in large-scale, parallel data processing pipelines.Competence in PyTorch and Python programming.Experience contributing to large, established codebases with rapid adaptation.

Join Anthropic as a Senior Research Scientist on our Reward Models team, where you will spearhead groundbreaking research aimed at enhancing our understanding of human preferences at scale. Your innovative contributions will directly influence how our AI models, including Claude, align with human values and optimize for user needs. You will delve into the forefront of reward modeling for large language models, designing novel architectures and training methodologies for Reinforcement Learning from Human Feedback (RLHF). Your research will explore advanced evaluation techniques, including rubric-based grading, and tackle challenges such as reward hacking. Collaboration is key, as you'll work alongside teams in Finetuning, Alignment Science, and our broader research organization to ensure your findings result in tangible advancements in AI capabilities and safety. This role offers you an opportunity to address critical AI alignment challenges, leveraging cutting-edge models and substantial computational resources to further the science of safe and capable AI systems.

ABOUT BASETENAt Baseten, we are at the forefront of enabling transformative AI solutions for some of the world's leading companies, including Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma, and Writer. Our innovative platform combines cutting-edge AI research, adaptable infrastructure, and developer-friendly tools to facilitate the production of advanced models. Recently, we celebrated our rapid growth with a successful $300M Series E funding round from notable investors like BOND, IVP, Spark Capital, Greylock, and Conviction. We invite you to join our dynamic team and contribute to the evolution of AI product deployment.THE ROLEAs a Senior Software Engineer specializing in Model Training at Baseten, you will play a pivotal role in constructing the infrastructure essential for the large-scale training and fine-tuning of foundational AI models. Your responsibilities will include designing and implementing distributed training systems, optimizing GPU utilization, and establishing scalable pipelines that empower Baseten and our clientele to adapt models with efficiency and reliability. This role demands a high level of technical expertise and hands-on involvement: you will be responsible for critical components of our training stack, collaborate with product and infrastructure teams to identify customer needs, and drive advancements in scalable training infrastructure.EXAMPLE WORK:Training open-source models that surpass GPT-5 capabilities for a leading digital insurerExploring specialized, continuously learning models as the future of AIOverview of our training documentationResearch initiatives we've undertakenRESPONSIBILITIESDesign, construct, and sustain distributed training infrastructures for large foundation modelsDevelop scalable pipelines for fine-tuning and training across diverse GPU/accelerator clustersEnhance training performance through optimization of algorithms and infrastructureCollaborate closely with cross-functional teams to align technical solutions with business objectivesStay abreast of advancements in the field of machine learning and AI to continually improve our training processes

OpenAI is hiring a Software Engineer for Post-Training Research in San Francisco. This position centers on improving the performance and capabilities of advanced machine learning models after their initial training phase. Role overview Work closely with a skilled team to explore new ways of strengthening AI systems. The focus is on researching and developing methods that push the boundaries of what these models can achieve once training is complete. Collaboration Expect to contribute to ongoing research efforts and share insights with colleagues who are passionate about advancing AI. Teamwork and knowledge exchange are key parts of this role. Location This position is based in San Francisco.

Zyphra is a cutting-edge artificial intelligence firm headquartered in the vibrant city of San Francisco, California.Position Overview:As a Research Scientist specializing in Model Architectures, you will play a pivotal role in Zyphra’s AI Architecture Research Team. Your responsibilities will include the design and thorough evaluation of innovative model architectures and training methodologies aimed at enhancing essential modeling capabilities (e.g., loss per flop or loss per parameter) and tackling core limitations inherent in current models. You will collaborate closely with our pre-training team to ensure that your findings are seamlessly integrated into our next-generation models.Qualifications:A strong research acumen and intuition.Proven ability to navigate research projects from initial conception to execution and final write-up.Exceptional implementation and prototyping skills, with the capability to swiftly transform ideas into experimental outcomes.A collaborative spirit and the ability to thrive in a fast-paced research environment.A deep curiosity and enthusiasm for understanding intelligence.Requirements:Experience with long-term memory, RAG/retrieval systems, dynamic/adaptive computation, and alternative credit assignment strategies.Knowledge of reinforcement learning, control theory, and signal processing techniques.A passion for exploring and critically evaluating unconventional ideas, with the ability to maintain a unique perspective.Familiarity with modern training pipelines and the hardware necessities for designing efficient architectures compatible with GPU hardware.Strong understanding of experimental methodologies for conducting rigorous ablations and hypothesis testing.High proficiency in PyTorch and Python programming.Ability to quickly assimilate into large pre-existing codebases and contribute effectively.Prior publication of machine learning research in reputable venues.Postgraduate degree in a scientific discipline (e.g., Computer Science, Electrical Engineering, Mathematics, Physics).Why Join Zyphra?We emphasize a structured research methodology that systematically addresses ambitious challenges in AI.

Join our innovative team at Zyphra as a Research Engineer specializing in Brain-Computer Interface (BCI) Models. In this pivotal role, you will contribute to groundbreaking research and development initiatives in the field of neuroscience and artificial intelligence. Your expertise will help shape the future of communication between humans and machines, enhancing the quality of life for countless individuals.As a Research Engineer, you will be responsible for designing, implementing, and testing advanced BCI models, collaborating closely with a diverse team of scientists and engineers. Your work will play a crucial role in advancing our understanding of neural dynamics and their applications in technology.

OverviewPluralis Research is at the forefront of Protocol Learning, innovating a decentralized approach to train and deploy AI models that democratizes access beyond just well-funded corporations. By aggregating computational resources from diverse participants, we incentivize collaboration while safeguarding against centralized control of model weights, paving the way for a truly open and cooperative environment for advanced AI.We are seeking a talented Machine Learning Training Platform Engineer to design, develop, and scale the core infrastructure that powers our decentralized ML training platform. In this role, you will have ownership over essential systems including infrastructure orchestration, distributed computing, and service integration, facilitating ongoing experimentation and large-scale model training.ResponsibilitiesMulti-Cloud Infrastructure: Create resource management systems that provision and orchestrate computing resources across AWS, GCP, and Azure using infrastructure-as-code tools like Pulumi or Terraform. Manage dynamic scaling, state synchronization, and concurrent operations across hundreds of diverse nodes.Distributed Training Systems: Design fault-tolerant infrastructure for distributed machine learning, including GPU clusters, NVIDIA runtime, S3 checkpointing, large dataset management and streaming, health monitoring, and resilient retry strategies.Real-World Networking: Develop systems that simulate and manage real-world network conditions—such as bandwidth shaping, latency injection, and packet loss—while accommodating dynamic node churn and ensuring efficient data flow across workers with varying connectivity, as our training occurs on consumer nodes and non-co-located infrastructure.

At Magic, we are dedicated to creating safe artificial general intelligence (AGI) that propels humanity forward in tackling the most pressing global challenges. We believe that the most effective route to achieving safe AGI involves automating the research and code generation processes to enhance models and resolve alignment issues more reliably than humans can achieve independently. Our methodology incorporates cutting-edge pre-training at scale, domain-specific reinforcement learning (RL), ultra-long context capabilities, and optimized inference-time computations.Role OverviewIn your role as a Software Engineer on the Pre-training Systems team, you will be responsible for designing and managing the distributed infrastructure necessary for training Magic’s long-context models at scale.This position emphasizes large-scale model training utilizing extensive GPU clusters. You will operate at the intersection of deep learning and distributed systems, ensuring that training processes are efficient, reliable, and reproducible under extreme conditions.Magic’s long-context models present complex systems challenges, such as sustained memory usage, communication overhead across thousands of devices, long-duration jobs requiring fault tolerance, and efficient sequence packing within hardware limitations. You will take ownership of the systems that ensure large-scale pre-training is both stable and rapid.Your ContributionsScale distributed training across large GPU clusters, implementing data, tensor, and pipeline parallelism.Optimize communication strategies and gradient synchronization.Enhance checkpointing, fault tolerance, and job recovery mechanisms.Profile and resolve performance bottlenecks across computing, networking, and storage.Advance experiment reproducibility and orchestration workflows.Boost hardware utilization and overall training throughput.Collaborate with Kernel and Research teams to align model architecture with system capabilities.Qualifications We SeekSolid foundation in software engineering and distributed systems.Experience with training large models in multi-node GPU environments.In-depth understanding of parallelism techniques and performance trade-offs.Experience in debugging cross-layer issues within production ML systems.Demonstrated ownership mentality and capability to manage critical infrastructure.Proven track record in enhancing the performance or reliability of large-scale systems.

Join Baseten as a Post-Training Research Engineer and contribute to groundbreaking advancements in machine learning and AI. In this role, you will leverage your engineering skills to analyze and enhance models post-training, ensuring optimal performance and efficiency.

Join Cartesia as a Model Architecture ResearcherAt Cartesia, our vision is to revolutionize AI by creating interactive intelligence that is seamlessly integrated into your daily life. Unlike current models, our goal is to develop systems capable of processing extensive streams of audio, video, and text—1 billion text tokens, 10 billion audio tokens, and 1 trillion video tokens—directly on devices.As pioneers in innovative model architectures, our founding team, which originated from the Stanford AI Lab, has developed State Space Models (SSMs)—a groundbreaking foundation for training efficient, large-scale models. Our diverse team merges deep expertise in model innovation with a design-focused engineering approach, allowing us to create and deploy state-of-the-art models and applications.Backed by leading investors such as Index Ventures, Lightspeed Venture Partners, and many others, including industry veterans and advisors, we are poised to shape the future of AI.Your ContributionIn this role, you will drive forward-thinking research in neural network architecture, focusing on alternative models like state space models, efficient transformers, and hybrid architectures.Create innovative architectures that enhance model performance, inference speed, and adaptability in various environments, from cloud infrastructures to on-device implementations.Develop advanced capabilities for models, including statefulness, long-range memory, and novel conditioning mechanisms to boost expressiveness and generalization.Analyze architectural decisions and their effects on model characteristics such as scalability, robustness, latency, and energy consumption.Create frameworks and tools to assess architectural advancements, benchmarking their performance in both research and production contexts.Collaborate with interdisciplinary teams to translate architectural insights into scalable systems that deliver real-world impact.Your QualificationsExtensive experience in architecture design with a focus on advanced models such as state space models, transformers, and RNN/CNN variants.In-depth understanding of the interplay between architectural designs and system constraints, particularly in cloud and on-device deployments.Strong proficiency in the design and evaluation of neural network architectures.

About Scale AI At Scale AI, we are committed to propelling the advancement of AI technologies. For over eight years, we have been a pioneer in the AI data sector, supporting groundbreaking innovations in areas such as generative AI, defense solutions, and autonomous driving. Following our recent Series F funding round, we are enhancing access to premium data to accelerate the journey towards Artificial General Intelligence (AGI). Building on our legacy of model evaluation for both enterprise and governmental clients, we are expanding our capabilities to establish new benchmarks for evaluations in both public and private domains. About This Role This position is at the leading edge of AI research and practical implementation, concentrating on reasoning within large language models (LLMs). The successful candidate will investigate critical data types vital for evolving LLM-based agents, including browser and software engineering agents. You will significantly influence Scale’s data strategy by pinpointing optimal data sources and methodologies to enhance LLM reasoning. To excel in this role, you will require a profound understanding of LLMs, planning algorithms, and fresh approaches to agentic reasoning, alongside inventive solutions to challenges in data generation, model interaction, and evaluation. Your contributions will lead to transformative research on language model reasoning, facilitate collaboration with external researchers, and engage closely with engineering teams to translate cutting-edge advancements into scalable, real-world applications.

Tavus – Multimodal AI Model OptimizationResearch EngineerAt Tavus, we are pioneering the human aspect of AI technology. Our objective is to make human-AI interactions as seamless and natural as in-person conversations, allowing for a human touch in areas that were once considered unscalable.We accomplish this through groundbreaking research in multimodal AI, focusing on human-to-human communication modeling (encompassing language, audio, and video) and the development of audio-visual avatar behaviors. Our innovative models drive applications ranging from text-to-video AI avatars to real-time conversational video experiences across sectors such as healthcare, recruitment, sales, and education.By empowering AI to perceive, listen, and engage with an authentic human-like presence, we are laying the groundwork for the next generation of AI workers, assistants, and companions.As a Series B company, we are supported by renowned investors, including Sequoia, Y Combinator, and Scale VC. Join us as we shape the future of human-AI interaction.The RoleWe are seeking an accomplished Research Scientist/Engineer with expertise in model optimization to be a vital part of our core AI team.The ideal candidate thrives in dynamic startup environments, is adept at setting priorities independently, and is open to making calculated decisions. We are moving swiftly and need individuals who can help navigate our path forward.Your MissionTransform state-of-the-art research models into fast, efficient, and production-ready systems through techniques such as sparsification, distillation, and quantization.Oversee the optimization lifecycle for critical models: establish metrics, conduct experiments, and evaluate trade-offs among latency, cost, and quality.Collaborate closely with researchers and engineers to convert innovative concepts into deployable solutions.RequirementsExtensive experience in deep learning with PyTorch.Practical experience in model optimization and compression, including knowledge distillation, pruning/sparsification, quantization, and mixed precision.Familiarity with efficient architectures such as low-rank adapters.Strong grasp of inference performance and GPU/accelerator fundamentals.Proficient in Python coding and adherence to best practices in research engineering.Experience with large models and datasets in cloud environments.Capability to read ML literature, reproduce results, and modify ideas accordingly.