LLMOps & Model Selection
The operational discipline of running AI features in production, closing out this Part.
6.10.1Definition
LLMOps is the operational practice of deploying, monitoring, versioning, and cost-managing language-model-powered features in production — the AI-specific analogue of standard DevOps practice (7.5). Model selection is the deliberate choice of which model (and which version) to use for a given task, balancing capability, latency, and cost.
6.10.2Why It Exists
AI features have operational characteristics standard software doesn't — non-deterministic output, per-token cost that scales with usage, and model versions that improve or occasionally shift behavior over time. LLMOps exists to bring the same operational discipline — monitoring, versioning, rollback — that mature software engineering already applies elsewhere (Part VII) to this less predictable, cost-sensitive layer.
6.10.3Model Selection Trade-offs
| Factor | Smaller/faster model | Larger/more capable model |
|---|---|---|
| Latency | Lower — better for real-time, high-volume features | Higher — better suited to complex, less time-sensitive tasks |
| Cost per request | Lower | Higher |
| Task complexity ceiling | Simple classification, extraction, short generation | Complex reasoning, long-form generation, agentic tasks (6.5) |
6.10.4Common Mistakes
- Defaulting to the largest, most expensive model for every task regardless of complexity, inflating cost for requests a smaller model would handle equally well.
- No monitoring of model output quality over time (6.8), missing a gradual quality regression until users notice and complain.
- Pinning to a model version with no upgrade plan, eventually running on a deprecated model with no clear migration path.
- No cost monitoring per feature, discovering an AI feature's operating cost only when a bill arrives far higher than expected.
6.10.5Best Practices
- Match model size to task complexity — route simple, high-volume tasks to smaller/cheaper models and reserve larger models for genuinely complex ones.
- Monitor both output quality (6.8) and per-feature cost continuously in production, not just at launch.
- Track model version changes deliberately and re-run evaluation sets before adopting a new version in production.