About the job
Responsibilities include:
- Designing and implementing role-play evaluation scenarios that mimic authentic customer service interactions in various sectors, including:
- Travel booking and support
- Financial services
- Telecommunications and technical assistance
- Assisting in the creation of diverse and representative datasets crucial for evaluating conversational audio agents.
- Assessing model performance through a standardized framework of qualitative and quantitative metrics.
- Ensuring evaluations align with real customer expectations regarding clarity, efficiency, and natural conversational flow.
Evaluation Metrics:
Model performance will be evaluated using an array of conversational, technical, and audio-specific criteria, including but not limited to:
- Task completion accuracy and efficiency
- Conversational naturalness (tone, flow, coherence)
- Audio comprehension and response quality
- Instruction adherence and contextual understanding
- Basic computer programming literacy, which includes:
- Understanding of JSON structures
- Familiarity with functions and methods
- Ability to reason with structured data and simple logic
- Technical communication clarity while addressing support-style problem-solving
Technical & Equipment Requirements:
- Excellent verbal communication skills in a simulated customer support environment
- Access to a high-quality microphone for clear and reliable audio input during evaluations
- Comfort with structured prompts, evaluation rubrics, and technical guidelines

