About the job
At Perplexity, we empower millions of users every day with accurate, high-quality answers through our innovative LLM-first search engine and specialized data sources. Join our Answer Quality team, where we focus on enhancing user experience by ensuring our prompts, tools, search capabilities, and specialized datasets, combined with both cutting-edge and in-house models, provide the best results. As a Data Scientist/Engineer, you will analyze online signals derived from user interactions to connect shifts in answer quality with actual user behavior.
Key Responsibilities
Identify and validate online signals from user interactions that act as dependable indicators of true answer quality.
Develop and implement innovative online metrics for tracking in A/B testing and product health dashboards, ensuring alignment with ground-truth evaluations.
Evaluate experimental results to validate these metrics, ensuring they accurately reflect user satisfaction and guide product decisions.
Construct and sustain data pipelines that compute these metrics at scale, providing actionable quality signals to Search, Product, and model training teams.
Share insights and foster collaboration with Product and Search teams to enhance clarity and understanding.
Contribute to a small, high-impact team where your work is instrumental in shaping how Perplexity measures and enhances Answer Quality.
Qualifications
Master's degree in a technical field or equivalent professional experience.
4+ years of experience in roles such as Data Scientist, Analytics Engineer, or similar positions.
Proven experience with search, recommendation, or LLM-based products, particularly in designing online metrics and analyzing A/B tests.
Strong coding skills in Python and SQL, with the ability to produce production-grade code.
In-depth knowledge of statistical analysis methodologies.
Experience using Business Intelligence (BI) tools for data visualization and reporting.
Comfortable with coding workflows and using AI-assisted development tools for rapid iteration.
Preferred Qualifications
Familiarity with Apache Spark and Databricks.
Experience in developing or validating LLM-as-a-judge systems.
Previous experience supporting large-scale customer-facing products.

