Eidetic
← Selected achievements Employment history · 2019–2026

Engineering systems at Reward Gateway.

During my time at Reward Gateway, I built the delivery, analysis and evaluation systems below, and grew the quality engineering team that operated them.

Period
2019–2026
Scope
More than 200 people in product and engineering
Role
Head of Quality Engineering, later Principal Quality Engineer

Context

My work on two AI products involved different stages of development: I built the evaluation and security suite for an existing assistant, and developed a risk and test plan for a second product.

AI engineering and release readiness

Built the evidence to assess an AI assistant for release.

The product answered employee questions from each company’s own content across web, iOS and Android. I traced the existing system from content publishing to the answer shown to the employee, built the evaluation and security suite, and showed the go/no-go panel what we could and couldn’t prove.

How the assistant worked and the system-level evidence I builtA component diagram of the existing multi-tenant assistant. Web, iOS and Android connect through API and identity to assistant orchestration. The assistant coordinates content ingestion, tenant-scoped retrieval, model orchestration, conversation state and evaluation telemetry. OpenSearch is the retrieval index. AWS Bedrock contains safeguards, ranking and generation. The 180 test cases, 44 security tests, 13 isolation tests and stubbed model boundary sit in a separate system-level assurance boundary.CLIENT SURFACESEmployeesurfacesWeb · iOS · AndroidCONTENT SOURCEContentpublishingMULTI-TENANT ASSISTANT PLATFORMAPI & identityauthentication · tenant contextAssistant orchestrationquery coordinationContentingestionasync updatesRetrievaltenant-scopedModelorchestrationConversationstateRetrieval indexOpenSearchEvaluationtelemetryAWS BEDROCKSafeguardsRankingGenerationSYSTEM-LEVEL ASSURANCEEvaluation suites180test cases44security13isolationStubbed model boundaryHow the assistant worked and the system-level evidence I builtA component diagram of the existing multi-tenant assistant. Web, iOS and Android connect through API and identity to assistant orchestration. The assistant coordinates content ingestion, tenant-scoped retrieval, model orchestration, conversation state and evaluation telemetry. OpenSearch is the retrieval index. AWS Bedrock contains safeguards, ranking and generation. The 180 test cases, 44 security tests, 13 isolation tests and stubbed model boundary sit in a separate system-level assurance boundary.CLIENT SURFACESEmployee surfacesWeb · iOS · AndroidCONTENT SOURCEContent publishingMULTI-TENANT ASSISTANT PLATFORMAPI & identityauthentication · tenant contextAssistant orchestrationquery coordinationContent ingestionasync updatesRetrievaltenant-scopedModelorchestrationConversationstateRetrieval indexOpenSearchEvaluationtelemetryAWS BEDROCKSafeguardsRankingGenerationSYSTEM-LEVEL ASSURANCEEvaluation suites180test cases44security13isolationStubbed model boundary
I traced the two paths shown here so I could test the right boundaries and be clear about what the results did and did not cover.

Evidence and limits

  • System. The existing assistant used OpenSearch for tenant-scoped retrieval and AWS Bedrock for safeguards, ranking and generation.
  • Tests. I built an evaluation suite of 180 tests across unit, integration, API and functional layers.
    • 44 security tests covered authentication, guardrails, personal data, prompt injection and tenant isolation.
    • 13 of those security tests focused on tenant isolation.
  • Variable answers. Because answers varied between runs, I checked:
    • Required and prohibited content.
    • Grounding and retrieval scope.
    • Meaning and consistency through semantic assertions, classifier-based validation and multi-run sampling.
  • Product evidence. Benchmark answers were reviewed by subject-matter experts. I tracked follow-up-question rate as a product signal.
  • Release evidence. I documented what the tests covered and where further evidence was needed in the go/no-go report. The decision belonged to the panel.

Earlier AI work

  • Pull-request review. I built a GenAI agent in 2024 that:
    • Reviewed open pull requests and analysed test code and method naming.
    • Used codebase conventions to inform its review comments.
  • Vendor evaluation. I tested AI testing and accessibility tools and reported limitations that affected whether I could recommend them.

Accessibility across web and mobile

I led accessibility evaluation against WCAG 2.2 AA and conformance reporting using VPAT across web, iOS and Android.

  • Automated checks covered more than 150 user journeys.
  • The audit evidence supported enterprise customers’ purchasing requirements.

Delivery infrastructure

  • Orchestration. I designed systems coordinating roughly one thousand end-to-end tests across pull-request, release and nightly runs. They remained operational through two changes of ownership, including the Edenred acquisition.
  • Test evidence. I built a library of more than seven thousand cases and linked test evidence to released changes. I moved test planning earlier, so teams agreed what to test before implementation.
  • Review controls. I used the evidence to review an ineffective pull-request control and removed it.

Using incident analysis to guide engineering work

  • Analysis. I built a Python pipeline across incident, documentation and monitoring systems to analyse 451 production incidents over ninety days.
  • Business case. The analysis supported a £227,000 annual business case for preventing failures. This was an estimate of potential value, not realised savings.

Growing the engineering team

  • Team. I grew the team from four to fifteen engineers across London and Sofia, supporting more than two hundred people in product and engineering.
  • Progression. I designed a technical career framework and ran an intern programme.