LLMOps: Running AI Features in Production

Shipping a demo GPT wrapper is easy. Running customer-facing AI requires prompt versioning, eval harnesses, fallback models, and cost caps that finance can live with.

Prompt and model versioning

• Store prompts in git with semantic tags (summarize-v3)
• Pin model IDs per environment; staging can preview newer models
• Log prompt hash + model + latency + token count per request

Eval harness

const results = await evalRunner.run({
  dataset: 'support-replies-gold',
  prompt: 'support-reply-v2',
  model: 'gpt-4.1-mini',
  graders: ['factuality', 'tone', 'length'],
});

Block deploys when factuality scores drop below threshold on golden sets.

Fallback and degradation

1. Primary model timeout → smaller fallback model
2. Fallback failure → cached template response + human queue
3. Never infinite retry loops burning tokens

Cost controls

Per-tenant daily token budgets, per-feature concurrency limits, and alerts at 80% of monthly AI spend.

LLMOps is SRE for nondeterministic features—treat outputs as probabilistic services with SLAs and budgets.