Artificial Intelligence Evaluation Engineer

Job description

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for an AI Evaluation Engineer based in Saudi Arabia. As the founding AI Evaluation Engineer, you will design and build the evaluation, testing, and observation infrastructure that ensures production-level AI agents perform reliably, securely, and efficiently. You will develop automated test systems for complex reasoning, multi-turn conversations, and agent interactions involving APIs, databases, and code execution. Working across AI engineering, software testing, and infrastructure, you'll create scalable evaluation pipelines that go beyond traditional QA and manual spot testing. You will create robust regression testing, statistical evaluation methods, and adversarial scenarios to identify vulnerabilities before deployment. This is an opportunity to take ownership of the assessment architecture from the ground up while working with modern AI technologies in a fully remote, globally distributed environment. Your work will contribute directly to the reliability, security, and continuous improvement of advanced AI systems. ### Responsibilities Design, build, and maintain automated CI/CD testing pipelines to validate AI agent behavior prior to deployment and throughout the development lifecycle. Develop and maintain gold datasets containing claims regres, edge cases, and synthetic scenarios to measure agent accuracy, safety, performance, and response time. Implement quick and cost-effective deterministic assessment checks before offering LLM-as-a-qa curricula This includes JSON schema validation, regular expressions, API status checking, and similarity inclusion. Create comprehensive test suites for multi-turn agent conversations, evaluating context retention, state consistency, reasoning behavior, and safeguards against endless tool invocation loops. Design and implement adversarial testing scenarios to evaluate resiliency against spot injection, malformed input, unexpected tool responses, and other potential security vulnerabilities. Build statistical evaluation frameworks for measuring non-deterministic AI behavior using success rates across multiple runs, confidence intervals, and other quantitative reliability measures. Integrate evaluation workflow into GitHub Actions or similar CI systems using SDK level implementation and automated testing practices. Implement monitoring and tracing solutions using tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or equivalent platforms to monitor agent execution and diagnose failures. Improve evaluation efficiency and cost through caching, mocking, and other techniques that reduce unnecessary LLM API calls. Validate structured output, including JSON chemas, OpenAPI specifications, and function call payloads, to ensure agents interact reliably with external systems. Evaluate and optimize agent performance under load, taking into account variable response times and non-deterministic endpoint behavior. Independently design and own assessment architecture, creating best practices, testing standards, and scalable processes for continuous quality assurance. ## Requirements E Strong proficiency in Python or TypeScript, especially for API testing, headless execution, asynchronous programming, and automated testing systems. Hands-on experience building LLM evaluation pipelines, AI testing frameworks, or statistical testing systems capable of measuring performance across repeated runs. Experience integrating automated tests into CI/CD environments, especially GitHub Actions or similar platforms, with implementation at SDK level. Strong experience in validating structured data and output, including JSON schema, OpenAPI specifications, and function call payloads. Familiarity with monitoring and tracking tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or custom logging solutions. Ability to independently design, implement and maintain assessment architecture from scratch, with a strong sense of ownership and technical judgement. Excellent written and spoken English, with the ability to communicate clear technical results in a distributed team. Availability to overlap with US Central Time business hours by at least three hours each day. Experience with code execution sandboxes or AI agent frameworks such as LangChain, LlamaIndex, AutoGen or CrewAI. Strong understanding of embedding models and vector databases and their applications in AI evaluation and similarity-based testing. Experience in load testing of unspecified endpoints and evaluating system behavior under different workloads. A proactive and analytical approach to problem solving, with the ability to work independently, adapt to evolving technologies, and continually improve methodologies Test. ## Benefits Remote work: Work from anywhere with a reliable internet connection, with the flexibility to choose your preferred time zone and work hours. Flexible schedule: Organize your workday according to your productivity while maintaining the desired overlap with US Central Time. Professional Development: You can access paid training and other opportunities to expand your technical expertise and support your career growth. Modern Technology: Work with AI tools, contemporary engineering frameworks and practices in an environment that values innovation and continuous improvement. Consistent Project Opportunities: Benefit from a constant flow of attractive and challenging product uct development work. Performance Recognition: Strong performance and dedication can be rewarded through annual pay increases. Negotiable Compensation: Your starting hourly rate will be determined based on your skills, experience, and qualifications. Independent Contractor Arrangement: Compensation is based on hours worked, with payments made monthly via bank transfer. Flexible work environment: Enjoy the independence of working in an environment that supports your focus, independence and professional development. How the employer Works: We use an AI-powered matching process to ensure your application is reviewed quickly, objectively and fairly against the essential requirements of the job. Our system selects the best candidates and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews and assessments) are managed by their internal team. We appreciate your interest and wish you the best! This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including the GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time. LE-CL1

Requirements

• ## Requirements * Strong proficiency in Python or TypeScript, especially for API testing, headless execution, asynchronous programming, and automated testing systems. Hands-on experience building LLM evaluation pipelines, AI testing frameworks, or statistical testing systems capable of measuring performance across repeated runs. Experience integrating automated tests into CI/CD environments, especially GitHub Actions or similar platforms, with implementation at SDK level. Ability to independently design, implement and maintain an evaluation structure from scratch, with a strong sense of ownership and technical judgement. Excellent written and spoken English, with the ability to clearly communicate technical findings in a distributed team. Availability to overlap US Central Time business hours by at least three hours each day. Experience with code execution sandboxes or AI agent frameworks such as LangChain, LlamaIndex, AutoGen or CrewAI. Experience in load testing of unspecified endpoints and evaluating system behavior under different workloads. A proactive and analytical approach to problem solving, with the ability to work independently, adapt to evolving technologies, and continually improve testing methodologies. ## Benefits * **Fully remote working:** Work from anywhere using a reliable internet connection, with the flexibility to choose your preferred time zone and work schedules. FLEXIBLE SCHEDULE:** Organize your workday around your productivity while maintaining the desired overlap with US Central Time. Negotiable Compensation: ** Initial hourly rate etc. will be determined We will hire you based on your skills, experience and qualifications

Skills

Benefits

• ## Benefits * **Fully remote work:** Work from anywhere with a reliable internet connection, with the flexibility to choose your preferred time zone and working hours.