company

Senior or Staff Machine Learning Systems Engineer – LLMs

TRM LabsSan Francisco, CA
On-site Full-time $200K/yr - $240K/yr

Clicking Apply Now takes you to AutoApply where you can tailor your resume and apply.


Unlock Your Potential

Generate Job-Optimized Resume

One Click And Our AI Optimizes Your Resume to Match The Job Description.

Is Your Resume Optimized For This Role?

Find Out If You're Highlighting The Right Skills And Fix What's Missing

Experience Level

Senior

Qualifications

Key QualificationsStrong experience in building and maintaining CI/CD workflows for ML systems. Proficiency in automating model versioning and compliance checks. Expertise in developing scalable AI infrastructure including databases and observability tools. Solid understanding of AI model integration into applications. Experience with cutting-edge AI tools and frameworks. Ability to drive AI governance and reliability initiatives. Strong analytical and problem-solving skills.

About the job

Join Us in Building a Safer World.

At TRM Labs, we specialize in blockchain analytics and AI solutions aimed at assisting law enforcement, national security agencies, financial institutions, and cryptocurrency businesses in identifying, investigating, and preventing crypto-related fraud and financial crime. Our innovative platforms leverage blockchain intelligence and AI technology to trace funds, detect illicit activity, and construct comprehensive threat profiles. Trusted by leading organizations worldwide, TRM Labs is committed to enabling a safer and more secure environment for all.

Our AI Engineering Team is dedicated to pioneering next-generation AI applications, particularly in the realm of Large Language Models (LLMs) and agentic systems. Our goal is to develop resilient pipelines and high-performance infrastructure that facilitate the swift, safe, and scalable deployment of AI systems.

We manage extensive petabyte-scale pipelines, ensuring model serving with millisecond latency while providing the necessary observability and governance to make AI production-ready. Our team actively evaluates and integrates leading-edge tools in the LLM and agent space, including open-source stacks, vector databases, evaluation frameworks, and orchestration tools to accelerate TRM’s innovation pace.

As a Senior or Staff ML Systems Engineer – LLM, you will play a pivotal role in constructing and scaling our technical infrastructure for AI/ML systems. Your responsibilities will include:

  • Creating reusable CI/CD workflows for model training, evaluation, and deployment, integrating tools such as Langfuse, GitHub Actions, and experiment tracking.

  • Automating model versioning, approval processes, and compliance checks across various environments.

  • Developing a modular and scalable AI infrastructure stack that encompasses vector databases, feature stores, model registries, and observability tools.

  • Collaborating with engineering and data science teams to embed AI models and agents into real-time applications and workflows.

  • Continuously assessing and incorporating state-of-the-art AI tools (e.g., LangChain, LlamaIndex, vLLM, MLflow, BentoML).

  • Promoting AI reliability and governance while enabling experimentation, ensuring compliance, security, and continuous uptime.

  • Enhancing AI/ML Model Performance and ensuring data accuracy and consistency, leading to improved model training and inference.

  • Implementing infrastructure to facilitate both offline and online evaluation of LLMs and agents.

About TRM Labs

TRM Labs is at the forefront of blockchain analytics and AI technology, dedicated to providing innovative solutions that empower law enforcement and financial institutions to combat financial crime and fraud. By harnessing the power of data and cutting-edge technology, we aim to create a safer world for all.

Similar jobs

Tailoring 0 resumes

We'll move completed jobs to Ready to Apply automatically.