Eidetic
← Selected achievements Employment history · 2019–2026

Engineering systems at Reward Gateway.

I led quality engineering for a complex employee engagement platform serving thousands of clients and millions of users. Hands-on work covered AI evaluation, delivery infrastructure and incident analysis.

Role
Head of Quality Engineering
Team
Grew from 4 to 15 engineers
Organisation
Supported more than 200 people in product and engineering

The evidence behind an AI release

Reward Gateway’s first customer-facing AI assistant answered employees’ questions from their own company’s content, on web, iOS and Android. I built the test suite to assess release readiness, wrote the gap analysis and risk register for the production go/no-go, and supplied the QA test report for the panel.

AI validation and continuous evaluationVersioned evaluation datasets contain reference answers, expected content and typical, edge and adversarial cases. The layered suite contains approximately 200 tests, including approximately 50 security tests. Security tests covered authentication and tenant isolation, prompt injection and guardrails, and personal data protection. Testing also covered data ingest, load and performance, and AWS token costs. The AI assistant is one box across web, iOS and Android; selected checks use model stubbing. Repeated runs are evaluated with bounded assertions, semantic assertions and classifier validation. Retrieval relevance and scope, answer grounding, citation accuracy and consistency, and reliability are distinct validation concerns. Test evidence feeds automated regression gates, then a go/no-go report with coverage and gaps for the panel's release decision. Failed checks feed failure analysis. Production feedback from Datadog monitoring also feeds failure analysis, which informs new cases in the versioned datasets. Connections show testing and evidence relationships, not internal architecture or complete coverage.Versioned datasetsReference answers · expected contentTypical · edge · adversarial cases≈200 tests totalUnit · integration · API · functionalData ingest · load & performanceAWS token costs≈50 security testsAuthentication · tenant isolationPrompt injection · guardrailsPersonal data protectionAI assistantWeb · iOS · AndroidModel stubbingSelected checksRepeated runsMulti-run samplingEvaluation methodsBounded assertionsSemantic assertionsClassifier validationRetrievalRelevance · scopeAnswer qualityGrounding · citation accuracy · consistencyReliabilityLatency · streaming · failuresRegression gatesAutomated checksGo/no-go reportCoverage & gapsPanel release decisionFailure analysisNew regression casesProduction feedbackDatadog monitoringAI validation and continuous evaluationVersioned evaluation datasets contain reference answers, expected content and typical, edge and adversarial cases. The layered suite contains approximately 200 tests, including approximately 50 security tests. Security tests covered authentication and tenant isolation, prompt injection and guardrails, and personal data protection. Testing also covered data ingest, load and performance, and AWS token costs. The AI assistant is one box across web, iOS and Android; selected checks use model stubbing. Repeated runs are evaluated with bounded assertions, semantic assertions and classifier validation. Retrieval relevance and scope, answer grounding, citation accuracy and consistency, and reliability are distinct validation concerns. Test evidence feeds automated regression gates, then a go/no-go report with coverage and gaps for the panel's release decision. Failed checks feed failure analysis. Production feedback from Datadog monitoring also feeds failure analysis, which informs new cases in the versioned datasets. Connections show testing and evidence relationships, not internal architecture or complete coverage.Versioned datasetsReference answers · expected contentTypical · edge · adversarial cases≈200 tests totalUnit · integration · API · functionalData ingest · load & performanceAWS token costs≈50 security testsAuthentication · tenant isolationPrompt injection · guardrailsPersonal data protectionAI assistantWeb · iOS · AndroidModel stubbingSelected checksRepeated runsMulti-run samplingEvaluation methodsBounded assertionsSemantic assertionsClassifier validationRetrievalRelevance · scopeAnswer qualityGrounding · citation accuracy · consistencyReliabilityLatency · streaming · failuresRegression gatesAutomated checksGo/no-go reportCoverage & gapsPanel release decisionFailure analysisNew regression casesProduction feedbackDatadog monitoring
Versioned cases, repeated evaluation and regression gates support release decisions. Failures and production feedback inform the next test cycle.

The test suite covered:

  • Security. Authentication, tenant isolation, personal data and prompt injection, including oversized and mixed-language attempts to get past guardrails.
  • Answer quality. Required and prohibited content, retrieval relevance, grounding, citation accuracy and consistency.
  • Running costs and resilience. Latency, streaming, failure handling, load, data ingest and AWS token cost.
  • After launch. Versioned datasets and regression gates checked every change. Test failures and issues identified through Datadog monitoring informed new regression tests, and follow-up-question rate was tracked as a product signal.

Understanding the cost of production incidents

A Python pipeline pulled hundreds of production incidents from ninety days of incident, documentation and monitoring data. The result was an estimated £227,000-a-year business case for preventing them.

Test infrastructure for reliable releases

I built the test framework and CI integration supporting around 1,000 end-to-end tests, most of which were written by my team. Tests ran at pull-request, release and nightly stages. The infrastructure remains in use following two changes of ownership, including the Edenred acquisition.

  • Fast feedback. Suites ran in parallel by product area, using repeatable environments with predefined test data. Reports combined test results with application-error checks.
  • Introducing shared test frameworks. I introduced shared test frameworks and trained teams on them. Quality owned the frameworks; backend engineers owned API tests covering end-to-end journeys across services.
  • Localisation. Missing translations caught automatically in CI.
  • Capacity. Representative workloads, controlled scaling and order-fulfilment checks.
  • Traceability. Over seven thousand test cases linked to releases, with scope agreed before build.
  • Knowing when to remove a check. A pull-request control was retired when it stopped providing reliable evidence.

Scaling quality as the team grew

Grew the function from 4 to 15 engineers across London and Sofia, and made delivery quality a shared engineering responsibility.

  • Operating model. Strategic vision, RACI and target operating model, backed by shared tools, standards and coaching.
  • Shared metrics. Datadog quality metrics gave every team the same view of delivery quality.
  • Progression. A technical career framework for quality engineers.
  • New routes in. Nine colleagues from outside engineering completed an 18-week quality engineering intern programme covering manual testing, test management and automation, and were recommended for engineering roles.

Accessibility evidence for enterprise procurement

Automated checks covered more than 150 user journeys across web, iOS and Android. I jointly scoped the WCAG 2.2 AA audit and VPAT report, including the user journeys assessed. It is the accessibility evidence enterprise buyers ask for. At the request of Edenred Group’s external auditors, I provided Reward Gateway’s digital accessibility CAPEX and OPEX figures for its 2025 Universal Registration Document.

AI testing and accessibility evaluations informed supplier recommendations, documenting limitations.