AI Evaluation Engineer

Job description

The AI Evaluation Engineer role in Saudi Arabia focuses on designing and building evaluation, testing, and monitoring infrastructure to ensure AI agents perform reliably, safely, and efficiently. Develops automated testing systems for complex reasoning, multi‑turn conversations, and agent interactions with APIs, databases, and code execution. Works across AI engineering, software testing, and infrastructure to create scalable evaluation pipelines that go beyond traditional QA. Builds robust regression tests, statistical evaluation methods, and adversarial scenarios to identify weaknesses before deployment. Manages evaluation architecture from scratch in a globally distributed remote environment, directly contributing to the reliability, security, and improvement of advanced AI systems

Requirements

• proficient in designing, building, and maintaining automated CI/CD test pipelines
• proficient in developing golden datasets for regression and edge cases
• proficient in implementing fast, cost‑effective evaluation checks
• proficient in creating comprehensive multi‑turn conversation test suites
• proficient in designing adversarial test scenarios
• proficient in building statistical evaluation frameworks
• proficient in integrating evaluation flows with GitHub Actions or similar CI systems
• proficient in applying monitoring and tracing solutions such as OpenTelemetry, LangSmith, Arize Phoenix, or W&B Weave
• proficient in optimizing evaluation efficiency and reducing LLM call costs
• proficient in validating structured outputs like JSON schemas and OpenAPI
• proficient in accounting
• proficient in Git

Skills