QA Automation Engineer (Senior)
Job Description
- Build evaluation suites for LLM and ML features: curated test sets, scoring against reference answers, LLM-as-judge where it is justified, and human review where it is not.
- Define what good enough means with the delivery team before launch, in numbers a client can accept, and hold releases to it.
- Test retrieval pipelines end to end, covering chunking and indexing, retrieval quality, grounding of the final answer, and behaviour when the right document is simply not there.
- Write regression tests that tolerate non-deterministic output, using semantic similarity, structured-output validation, tolerance bands, and repeated sampling rather than exact string matching.
- Cover failure modes that matter to clients: hallucination, prompt injection through user or document content, leakage of data across tenants or sessions, refusal and over-refusal, and degradation on Thai-language input.
- Track cost and latency alongside accuracy, so that a quality improvement that triples token spend is visible before it ships.
- Detect drift after release by comparing production samples against the baseline, and raise it early rather than after the client does.
- Design and maintain automated test suites for APIs, services, data pipelines, and web applications built for client projects.
- Run tests in CI, keep them fast and stable, and treat a flaky suite as a defect in its own right.
- Set the test strategy at the start of an engagement: what gets automated, what stays manual, what the exit criteria are, and what evidence the client will receive.
- 4-5 years in test automation, QA engineering, or software engineering with substantial testing ownership.
- Strong programming ability in Python, or in TypeScript or Java with the willingness to work primarily in Python.
- Hands-on experience testing an LLM or ML-based product in production or close to it.
- Practical command of at least one automation framework such as Pytest, Playwright, Cypress, or RestAssured, and experience running suites in CI.
- Working knowledge of API and data testing, including SQL and validating data through a pipeline.
- The judgement to decide what deserves automation and what does not, and to say so when a suite is producing noise rather than signal.
- Clear written and spoken communication in Thai and English.
- Hybrid Working options with Flexible Working Hours.
- Your Choice of Gear: Mac or Windows You pick your preferred work equipment.
- Comprehensive Group Health Insurance.
- Monthly Grab Transportation credits.
- Job-related training courses and certificate fee coverage.
- 10+ days of annual leave.
- Free snacks and drinks available at the office.
More Info
Job Type:
Industry:
Function:
Employment Type:
Key Skills
About Company
Skills:
Java, Appium, Sql, Nosql, Databases, Git, Javascript, Mobile Applications, Robot Framework, Selenium, Restful Apis, Python, CI/CD pipelines, Cypress
Skills:
Typescript, Javascript, Firebase Crashlytics, Performance Testing, Maestro, Chaos testing, MCP-based device automation, Playwright, API test automation, Claude, Firebase Remote Config, Bruno, Fuzz testing, AI-assisted testing, CI/CD pipelines, Observability tooling, Agent-driven device automation, Firebase Test Lab
Skills:
AWS, Sql, Java, Kubernetes, Fix Protocol, Docker, Maven, Database Testing, Selenium, Cucumber-JVM, Playwright