Reliable AI Agents in Production: A CTO Checklist

An agent that drafts a reply is a writing tool. An agent that sends the reply, issues the refund or merges the pull request is a production system, and it needs the same discipline as any other code that can change money, data or customer trust. A demo shows the agent succeeding once. Production asks what happens on the thousandth run, with messy input, a flaky API and a model version that changed last week.

Below is the checklist a CTO should walk through before an agent gets write access to anything real. It covers how you know the agent works, what it may touch, when a human decides, how you see what happened, what it costs, and how it recovers when a run crashes halfway through.

Write the evals before you widen access

An eval is a repeatable test of agent behavior: a task, an expected outcome and a way to grade the attempt. Without one, every prompt change and every model upgrade is a guess.

Anthropic's engineering team published a useful guide on this in January, Demystifying evals for AI agents (opens in a new tab). Two points from it are worth adopting directly:

• Start small and real. The guide says 20 to 50 simple tasks drawn from real failures is a good start. You do not need hundreds of cases before the first run. Take the manual checks your team already does before a release, plus the failures that show up in your support queue, and turn them into tasks.
• Measure consistency for agents that act. The guide separates pass@k (at least one of k attempts succeeds) from pass^k (all k attempts succeed). If an agent books meetings or edits records, one lucky success is irrelevant. You want pass^k, because a customer sees every run.

Grade with code wherever you can. Did the right tool get called with the right arguments? Did the database end up in the expected state? Use a model as the grader only for things code cannot check, such as tone or whether a summary kept the key facts, and check those model grades against human judgment on a sample.

Run the suite on every prompt change, tool change and model version change. A new model release can make your agent better at the benchmark and worse at your workflow.

Decide what the agent may touch

Guardrails are automatic checks that run around the model. They are cheap, fast and dumb, which is what you want. Useful ones:

• Input checks that reject requests outside the agent's job before the expensive model call runs.
• Output checks that block responses containing personal data, internal URLs or promises your policy does not allow.
• Tool-level limits enforced in your own code: a refund tool that refuses amounts above a threshold, a database tool that only runs read queries, an email tool that only sends to verified customer addresses.

The last group matters most. A prompt that says "never refund more than the order value" is a suggestion. A tool that returns an error when the amount exceeds the order value is a control. Treat every tool as a public API that an untrusted caller can hit with any arguments, because text from emails, web pages and documents can steer the model.

Give the agent the narrowest credentials that let it do the job. A service account with read access to one table is easier to reason about than a shared admin key.

Put a human on the irreversible steps

Some actions should pause and wait for a person. The usual line is anything that is hard to undo, costs money, or leaves your company in writing: payments, deletions, outbound messages to customers, production deploys, contract changes.

The major agent SDKs now support this pattern directly. In the OpenAI Agents SDK, a tool can be marked with needs_approval, which pauses the run and returns the pending tool calls as interruptions. Your application shows them to a reviewer, records an approve or reject decision, and resumes the same run from saved state. The human-in-the-loop docs (opens in a new tab) also carry a warning worth repeating: saved run state is not authenticated by the SDK, so keep it in server storage, check that the reviewer is allowed to approve, and reject replayed submissions.

Microsoft's agent harness, which reached general release in July (opens in a new tab), ships tool approval on by default, with standing "don't ask again" rules and heuristic auto-approval for calls it considers safe. Be careful with that last part. Auto-approval rules are policy, and policy should be written down, reviewed and tested like code.

Design the approval screen for the reviewer. Show the exact action, the arguments and the reason the agent gave, and make reject as easy as approve. A reviewer who clicks through 200 approvals a day has stopped reviewing.

Trace every run

When an agent does something strange, you need to see each model call, each tool call, the arguments, the results and the tokens spent, in order. Logs of the final answer are not enough.

Use OpenTelemetry so traces land in the tools your team already runs. The GenAI semantic conventions define span types for agent work, including invoke_agent, create_agent and execute_tool, with attributes such as gen_ai.agent.name and gen_ai.provider.name. They now live in their own semantic-conventions-genai repository (opens in a new tab), and the agent spans are still marked as Development status. Expect attribute names to change. Pin the version your instrumentation emits and budget a small amount of time for upgrades.

Microsoft's harness turns on OpenTelemetry by default, and the OpenAI Agents SDK has built-in tracing of model calls, tool calls and handoffs. Whichever you use, check three things before launch:

1. Every trace carries a run or conversation ID you can find from a support ticket.
2. Prompts and tool results that contain customer data are redacted or access-controlled in the trace store.
3. Someone is on the hook to read traces weekly, not only after an incident.

Traces are also where your next eval cases come from. Every bad run you find should become a test.

Cap the cost per task

Agent cost is harder to predict than chatbot cost because the agent decides how many steps to take. A task that usually needs four model calls can loop for forty when a tool keeps failing.

Practical controls:

• Set a maximum number of turns per run in the framework, with a clear failure message when it is hit.
• Keep a token or spend budget per task and per customer, checked in your code before each model call.
• Use the cheapest model that passes your evals for each step. Routing, classification and summarization often run well on a small model, with the large model reserved for planning or hard reasoning.
• Turn on prompt caching for long, stable system prompts and tool definitions. Most major providers discount cached input.
• Put cost per successful task on your dashboard. Cost per call hides the runs that loop and then fail.

Put an alert on the tail, not the average. The expensive runs are usually the broken ones.

Make long runs survive failure

Agents that wait for approvals, call slow APIs or run for minutes will eventually hit a deploy, a timeout or a crashed worker mid-task. If the agent has already charged a card and then restarts from the beginning, it may charge it again.

Two approaches work:

• Durable workflow engines. Temporal said in May that its integration with the OpenAI Agents SDK is generally available (opens in a new tab), and the same post covers support for Google's Agent Development Kit. Each step is recorded, so a crashed run resumes where it stopped instead of repeating side effects.
• Your own checkpoints. Save run state after each step, and make every tool idempotent with a request key so a retry cannot double-apply an action.

Either way, test it. Kill the worker halfway through a run in staging and confirm the outcome is correct.

Sandbox anything that runs code

If the agent reads files, runs shell commands or edits code, run it somewhere it cannot damage. In April, OpenAI added native sandbox execution and a model-native harness to the Agents SDK (opens in a new tab), launching first in Python. Sandboxes from providers such as E2B, Modal, Daytona or plain containers give the agent a disposable machine with no access to production secrets. Keep network egress closed by default and open it per domain.

The checklist

Print this and walk through it with the team that owns the agent.

If any row is empty, keep the agent in suggest-only mode: it drafts, a human sends. That is a fine place to run for a few weeks while the evals and traces build up evidence.

Start narrow, then widen

The safest path to an agent that acts is a sequence of small permission grants, each one backed by evals and traces from the step before. Read-only first, then drafts, then low-value actions with approval, then auto-approval for the cases your data says are safe.

Planning an agent that needs to take real actions in your systems? Book a free 30-min call (opens in a new tab) and we will help you work through this checklist for your stack.

Related service: AI Agents & Automation