AI features that behave on the bad day

Works in the demo, breaks in production?

A model feature that passed every eval and then returned the wrong thing under real inputs is rarely a model problem. A model call is one more external dependency you don't control: it times out, it answers twice, it hands back malformed JSON, and it can cost ten times on a bad input what it cost in the demo. Between the prompt and the row in your database there are a few places where that turns into a bug, and each one has its own check.

1. The output isn't validated against a schema, so a malformed or half-empty answer is treated as success.
2. A slow call blocks the request instead of going to a queue.
3. A timeout or rate-limit isn't retried — or is retried without an idempotency key, so a tool action runs twice.
4. Cost and tokens aren't capped or logged, so one adversarial input is a bill.
5. There is no eval on your own data, so "it works" is a feeling, not a number.
6. Nothing records what the model was asked and what it answered, so the one call that went wrong can't be debugged.

Work

• AI feature production-readiness review

  5–7 days · $2,500–4,000

  One model-backed feature. Read-only: your traces, logs, prompts and the code around the call. You get a document — where it fails on real inputs, what to fix first, what can wait — and a small eval set built from your own data that you keep running afterwards.

• Retry test for an agent that moves money

  1–2 weeks · $4,000–7,000

  Your agent or MCP server, run unmodified against fake providers that follow only your providers' documented rules. The money call commits, then times out, loses its answer or fails a later step, and your model decides what to do next. You get the count of duplicate money movements per tool and scenario, the exact text the model saw each time, what the agent told the merchant, and for each tool the key, the read and the wording that close the gap, with one re-run after the fix. No provider account needed.

• Prototype to production

  1–3 weeks · $4,000–9,000

  An AI prototype that works in the demo, made to survive real data and real load: the failure modes above, closed one by one, and the one decision — harden it or drop it — made on evidence rather than on the demo.

• Model integration in production

  2–4 weeks · $6,000–12,000

  A model API wired into your backend the way a payment provider should be: structured output you validate, timeouts and retries, idempotent tool calls, slow work on a queue, cost and latency ceilings, logs of what was asked and answered, tests on real samples, and a runbook.

• Incident retainer

  $2,000–4,000 a month

  A set number of hours each month for model calls, queues and cost under load — shared with payment work if you have both.

Diagnostics are fixed-price, 50% upfront. The first stage needs read-only access only. AI-assisted development is how I work day to day, so small integrations usually ship in days, with tests.

How a review starts

1. You describe the symptom and send one call that went wrong — the input, what you expected, what came back. I reply with which of the places above it usually is, and what I would need to read.
2. You export what you can: traces or logs of the calls, the prompt, a sample of real inputs. The first pass runs on those files, on your side if you prefer.
3. Within a week: the document, an eval set built from your data, and a walk-through of both.

Public work

One request, two refunds. Six payment toolkits published by the providers, run unmodified against a local fake that follows only each provider's documented rules, and six models driving them through a money call that reached the provider and came back as a timeout or an error. Money moved twice in 70 of 600 trials, 47 with a plain system prompt and 23 with a careful one; one model accounts for 50 of them and two for none. No toolkit sends an idempotency key by itself, and after a lost answer the tool usually reports the call as failed rather than as unknown, which is what sends a model back to repeat it. Seven toolkits were measured and six are published; one is held back until its maintainers have had time to respond. github.com/terminalstate/double-take

Six models on one contract-extraction task. The same prompt through six model APIs, scored against human labels on 83 public non-disclosure agreements. All six came out within noise of each other (F1 0.88–0.89), while the cost ran from $0.42 to $14.88 per thousand documents. Most of the remaining errors were made by most of the models at once, and read against the documents they come mainly from how the labels write names and define fields — which is why an eval starts with the definitions. Eleven first answers failed local validation — a bare year where a date belonged, a misspelled key, reasoning that ran out of room — so the check stays even when the API promises structure. Standard library, reproducible with one command. github.com/terminalstate/extraction-bench

The same task on a bad day. What happens around the answer: every output mechanism three model APIs offer, a repair for each kind of failure, and all 83 documents sent at once. Format failures followed each model's habits — one wrapped every answer in a code fence and kept doing it when asked not to — and a local repair fixed 98 of 99 of them for free; when the content was wrong, telling the model what was wrong fixed 20 of 20, a plain retry 7. With its thinking switched off, one model stopped cutting answers off, at less than half the cost and the same accuracy. The burst met no limit at one provider, a concurrency limit at another, and at the third a per-minute token limit that half the requests did not get past in five attempts. extraction-bench/reliability.md

The same lessons as code. model-call asks a model API for a JSON object that matches a schema and does what the two studies found works: every answer is checked, its shape is repaired locally without inventing values, and only then comes one more request, chosen by what went wrong — the answer back with its problems, a larger output limit, or the same request again. A deadline on the whole call, a cost ceiling reserved before each request, pacing for batches, a log of every call without keys, and a ledger so that a repeated tool call cannot charge a card twice. Standard library, MIT. github.com/terminalstate/model-call

The same discipline, on webhooks. callback-audit is a read-only tool that finds where a payment provider's callback is lost between the provider and your database — the webhook equivalent of the checks above: validate before you acknowledge, retry safely, never treat a repeat as new. MIT-licensed, with four real bugs reproduced against widely-used integrations and reported upstream. See the payments page.

Contact

Email: hello@terminalstate.dev. One call that went wrong is enough to start.