Specific Responsibilities

Evaluation Suites

• Build and run eval suites per product, decomposed by failure mode, running on schedule and on every release — regression testing so nothing ships if it broke what worked.
• Keep evals cost-effective as the product line grows.

Reporting, Data & Collaboration

• Own the reporting loop — findings from evals and platform data in front of the team and stakeholders, including surfacing unintended or problematic model behaviour before learners do.
• Steward the core datasets the team depends on, including classified customer-support data.
• Partner with the AI Product Manager on instrumentation — they instrument the product, you build the evals over what is captured. This is a measurement role, not infrastructure — no model hosting or serving.

Skill Requirements - Essential

• Python & data: solid Python and a data inclination, comfortable shaping and analysing messy LLM-generated data.
• Decomposition: the ability to look at an AI product and decompose it into success and failure metrics.
• Eval landscape: familiarity with Langfuse, RAGAS, DSPy, or similar — depth in one, awareness of the rest. These tools are learnable; we hire the fundamentals underneath them.
• Desirable (not required): experience keeping evals cheap at scale; dashboarding and reporting; classical statistics.

Essential Traits for Success

• You want to own a function, not execute tickets.
• You communicate well and like collaborating, you will support every builder on the team.
• You can point to any project, even a small one, where you measured an AI system honestly.

Person Specification/Attributes

• Courage: Willingness to speak up, challenge the status quo, and embrace new challenges.
• Humility: Openness to learning, seeking help when needed, and a focus on serving others.
• Adventure: A passion for setting ambitious goals, tackling difficult tasks, and finding joy in the journey.
• Initiative: Proactive problem-solving, a sense of ownership, and a willingness to go above and beyond.
• Resilience: The ability to bounce back from setbacks, persevere through challenges, and emerge stronger.

Employment Type

This role is a full-time position.

Preferred Time Zones

The preferred time zones are GMT+2.

Due to the considerable amount of virtual working and interaction with colleagues and customers in different physical locations internationally, it is essential that the successful applicant has the drive and ethic to succeed working in small teams physically but in larger efforts virtually. Self-drive to communicate constantly using web collaboration and video conferencing is essential. As an employee, you will be encouraged to continually develop your capability & attain certifications to reflect your growth as an individual.

Originally posted on Himalayas