Augustine Uzokwe, a QA lead, evaluated software testing processes by developing RTIA, a multi-agent tool that converts raw requirements into backlog-ready stories with acceptance criteria. The project demonstrates that while AI features rely on traditional code, they require additional quality assurance layers because LLM outputs are non-deterministic. Traditional unit and integration tests cannot verify the content quality or consistency of an AI response, necessitating an 'eval suite' of reference inputs and scoring metrics to act as a gate in CI pipelines.

Testing an AI feature involves addressing specific gaps. First, 'eval gates' are essential; they enforce thresholds for quality metrics like 'ac_coverage' to ensure generated output remains accurate. Second, caching requires caution because models evolve on external servers; therefore, regression jobs must disable caches to prevent stale measurements. Third, model provider selection is a recurring decision, as demonstrated by switching from Claude Opus to Gemini Flash to reduce costs by approximately an order of magnitude.

Fourth, adversarial defenses require a two-tiered approach: input scanners must block credentials before they reach the model, while separate methods must extract structured requirements to neutralize prompt injections. Fifth, observability extends beyond standard metrics to include full prompt-output tracing and cost tracking per call, though automated aggregate monitoring of quality drifts remains a prerequisite for customer-facing features.

Finally, defining 'done' for AI requires enforcing six specific conditions: schema compliance, requirement coverage, consistency checks, pre-screen defenses for security, budget constraints, and cache invalidation. These lessons confirm that while AI can execute tasks and verify them against established standards, human judgment remains necessary to define those standards and determine the acceptable quality thresholds for unpredictable systems.

Augustine Uzokwe, a QA lead, evaluated software testing processes by developing RTIA, a multi-agent tool that converts raw requirements into backlog-ready stories with acceptance criteria. The project demonstrates that while AI features rely on traditional code, they require additional quality assurance layers because LLM outputs are non-deterministic. Traditional unit and integration tests cannot verify the content quality or consistency of an AI response, necessitating an 'eval suite' of reference inputs and scoring metrics to act as a gate in CI pipelines.

Testing an AI feature involves addressing specific gaps. First, 'eval gates' are essential; they enforce thresholds for quality metrics like 'ac_coverage' to ensure generated output remains accurate. Second, caching requires caution because models evolve on external servers; therefore, regression jobs must disable caches to prevent stale measurements. Third, model provider selection is a recurring decision, as demonstrated by switching from Claude Opus to Gemini Flash to reduce costs by approximately an order of magnitude.

Fourth, adversarial defenses require a two-tiered approach: input scanners must block credentials before they reach the model, while separate methods must extract structured requirements to neutralize prompt injections. Fifth, observability extends beyond standard metrics to include full prompt-output tracing and cost tracking per call, though automated aggregate monitoring of quality drifts remains a prerequisite for customer-facing features.

Finally, defining 'done' for AI requires enforcing six specific conditions: schema compliance, requirement coverage, consistency checks, pre-screen defenses for security, budget constraints, and cache invalidation. These lessons confirm that while AI can execute tasks and verify them against established standards, human judgment remains necessary to define those standards and determine the acceptable quality thresholds for unpredictable systems.