QA Automation Engineer - AI

Overview

Anaplan is hiring a QA Automation Engineer to focus on quality assurance for generative AI and agentic systems. The position is based in London, United Kingdom, within the European region. The employer develops an AI-infused scenario planning and analysis platform used by global companies.

What you do

The role involves developing testing strategies, evaluation frameworks, and quality metrics for large language model powered applications. Responsibilities include building end-to-end automated test coverage across evaluations, application programming interfaces, user interfaces, and performance tests using Python. The selected engineer will run and maintain automated test suites for prompt testing, execute regression tests to detect changes in model behavior, and maintain evaluation frameworks measuring performance across accuracy, relevance, safety, consistency, and latency. Additional duties include curating test datasets and golden examples representing diverse user scenarios, conducting adversarial testing to identify model hallucinations or biases, supporting production alerting systems, assisting with user acceptance testing, and documenting testing procedures.

Requirements

Candidates need solid quality assurance or test engineering experience along with exposure to or a keen interest in artificial intelligence and machine learning systems. Necessary qualifications include hands-on Python coding experience, an understanding of generative artificial intelligence technologies such as large language models, prompt engineering, and artificial intelligence application patterns. Applicants must have experience with test automation frameworks and scripting including Python, Pytest, application programming interface testing, and user interface testing. Practical knowledge of functional, integration, regression, performance, and security testing methodologies is required, alongside experience with application programming interface testing tools like Postman and REST Assured, backend testing, and familiarity with continuous integration and continuous deployment pipelines and automated testing integration.