From prototype to production: Engineering agents with Agent DLC

Over the past year, AI agents have moved from exploration into engineering. Enterprise teams are shipping large language model (LLM) systems that call tools, orchestrate APIs, and answer questions over internal knowledge. A successful demo can show what an agent is capable of. Before release, the team needs to answer a further question: “ Is this agent ready for production? ” The person responsible for that decision needs evidence that the agent meets the agreed requirements. To make that decision consistently, teams need a repeatable, affordable way to assess the agent’s current behavior and whether the latest change has improved it. In traditional software development, teams answer these questions with unit tests, continuous integration and continuous delivery (CI/CD), and passing builds. Those mechanisms are necessary but not sufficient for agents, for three reasons, each requiring additional checks. This post explains these three properties and presents a measured case where an aggregate metric hides a coverage problem. It then introduces the Agent Development Lifecycle (Agent DLC): one lifecycle, two loops, and a single explicit standard for judging every change. A note on terminology: Throughout this post, gold standard means one specific artifact: the bundle of verifiable criteria, their thresholds, and the approved examples they are checked against. Business owners approve this standard as the basis for acceptance decisions. 1. Three properties of agents that require additional engineering practices Conventional software testing and CI/CD still apply to agents, but three properties require additional checks. First, agents are non-deterministic: identical inputs can produce different outputs and execution paths. Second, changes to prompts, tool descriptions, retrieval settings, or memory configuration can alter behavior in ways that a text review alone can’t reveal. Third, model endpoints, knowledge bases, and external APIs can change independently of the repository and affect agent behavior. Table 1 summarizes the engineering practices needed to address each property. Table 1. What each property invalidates, and what it requires instead. Property The conventional answer Why it falls short What Agent DLC requires Non-determinism One assertion per case in CI Measures a single sample of a distribution. Passes and fails for reasons unrelated to your change Run each example k times. Judge on pass rate and failure distribution Prompts are source code Review the diff, approve the change Behavior can’t be read off the text of a prompt Every prompt, tool, retrieval, and memory change is versioned with an evaluation attached, and the lineage is kept Dependencies move on their own Pin versions, validate on upgrade Doesn’t fire because nothing in your repository changed Re-run the standard on a schedule and alert on deviations from the baseline 2. A measured example: High retrieval precision can hide missing information The following example shows how the choice of metrics affects diagnosis. We built a reference implementation to exercise the method end to end: an internal policy assistant answering HR policy questions over a document knowledge base. We chose a narrow, well-scoped domain to make the evaluation and diagnosis straightforward to follow. The following results were measured on this reference implementation. It isn’t a customer outcome and it isn’t a benchmark result. The accompanying AWS Samples repository includes infrastructure deployment, the HR agent, custom evaluators, and scripts for baseline evaluation and prompt-change re-evaluation ( 09-run-eval.sh and 10-optimize-prompt.sh ). It also includes optional checks for judge repeatability, model comparison, cost, and latency. The scripts reproduce the workflow. Scores can vary across runs and repository versions. Three golden questions, five measures each: Groundedness : Decompose the answer into claims and check each against the retrieved sources. Chunk-level source precision : Is each retrieved chunk, as a whole, necessary for this question? Fact-level source precision : Decompose each chunk into individual facts and check whether each fact is necessary. Retrieval coverage : For each sub-question, does any retrieved chunk answer it? Response precision : Of the claims in the answer, how many are necessary? Table 2. First evaluation round. Measured on the reference implementation. Not a customer outcome. Question Groundedness Chunk-level precision Fact-level precision Retrieval coverage Verdict Performance review 0.83 1.00 0.59 1.00 Pass Benefits enrollment 0.90 1.00 0.53 1.00 Pass Sick-leave certificate 0.39 1.00 0.09 0.00 Fail Chunk-level precision is 1.00 for all three questions. A release gate based only on this metric would accept all three answers. Fact-level precision tells a different story: 0.59, 0.53, 0.09. For the third question, roughly nine out of ten facts in the retrieved material are irrelevant to what was asked. Chunk-level precision only asks whether a passage is on topic. It doesn’t measure how much of the passage is relevant. Then we changed exactly one thing: the system prompt. We added three constraints: ground strictly in retrieved content, ignore irrelevant retrieved content, and keep answers focused. The agent, the knowledge base, the evaluators, and the questions were untouched, so the before-and-after differs by one variable. Table 3. After a prompt-only change. One change, two opposite outcomes. Performance review, before Performance review, after Sick leave, before Sick leave, after Groundedness 0.83 0.87 0.39 (fail) 0.31 (still fail) Fact-level precision 0.59 0.85 0.09 0.04 Response precision 0.58 0.73 0.33 — The first question improved substantially. The third got worse. The evaluation suggested where to investigate before we made the change. Look at the third row of Table 2 again: chunk-level precision 1.00, fact-level precision 0.09, retrieval coverage 0.00. Together, these scores suggested a knowledge-coverage problem: the retrieved material was nominally on topic but didn’t answer the question. Telling the model to “ground strictly in retrieved content” couldn’t supply the missing information. We checked that diagnosis against the source documents used in the experiment. The corpus lacked a dedicated sick-leave policy document, although the leave-management policy contained brief references to sick leave. The next step was to address policy coverage and then check retrieval, rather than keep changing the prompt. In this case, the source material lacks the information needed to answer the question, so prompt changes alone can’t resolve the failure. In this example, evaluation helped identify what needed investigation before further changes. Another result shows why groundedness needs to be checked alongside other criteria. The benefits-enrollment answer passed with groundedness 0.90, and it contained this sentence: “I also noticed from your profile that your preferred contact method is Social Media.” No such profile exists. The policy document lists contact preference as an option inside a process step. The public, crowd-sourced, noisy FAQ corpus contains a pair answering that question with “social media”. The model combined a policy option with an unrelated FAQ answer and presented it as a fact about the user. The high groundedness score didn’t catch this false personal claim. Passing a quality threshold doesn’t establish that responsibility constraints have been met. These scores can vary between runs, and a borderline Pass can become a Fail. Interpret score differences alongside repeated runs and the underlying evidence. Section 5 discusses how to check evaluator reliability. This small reference-implementation example illustrates how evaluation can guide diagnosis. It doesn’t establish a general improvement rate or validate a production release. 3. Agent DLC: One lifecycle, two loops Agent DLC connects six activities around one explicit gold standard