Automation Tester

Description

• Design and implement testing strategies for AI/LLM-based systems
• Develop and apply evaluation frameworks for AI outputs
• Assess and validate:
• Accuracy and relevance of responses
• Consistency and reliability
• Hallucinations and edge cases

Work with LLM-as-judge patterns and automated evaluation techniques

Define quality standards for non-deterministic AI systems

Collaborate closely with engineering, product, and data teams

QE – Automation (API & Event-Based)

Key Responsibilities

• Design and implement testing strategies for AI/LLM-based systems
• Develop and apply evaluation frameworks for AI outputs
• Assess and validate:
• Accuracy and relevance of responses
• Consistency and reliability
• Hallucinations and edge cases

Work with LLM-as-judge patterns and automated evaluation techniques

Define quality standards for non-deterministic AI systems

Collaborate closely with engineering, product, and data teams

Required Skills & Experience

• Design and implement testing strategies for AI/LLM-based systems
• Develop and apply evaluation frameworks for AI outputs
• Assess and validate:
• Accuracy and relevance of responses
• Consistency and reliability
• Hallucinations and edge cases

Work with LLM-as-judge patterns and automated evaluation techniques

Define quality standards for non-deterministic AI systems

Collaborate closely with engineering, product, and data teams

Additional Details

Industry Background: Open (no specific industry Experience required)

3x days a week in Burgess hill office.

Skills

• java
• kotlin
• kafka
• Testing
• Automation
• Quality assurance

Job Title:

Automation Tester

Location:

Burgess Hill, UK

Job Type:

Contract

#J-18808-Ljbffr