AI engineering lifecycle
The concepts in AI engineering are best understood within the context of the development lifecycle. While AI capabilities can become highly sophisticated, they typically start simple and evolve through a disciplined, iterative process:1
Prototype a capability
Development starts by defining a task and prototyping a with a prompt to solve it.
2
Evaluate with ground truth
The prototype is then tested against a of reference examples (so called “”) to measure its quality and effectiveness using . This process is known as an .
3
Observe in production
Once a capability meets quality benchmarks, it’s deployed. In production, graders can be applied to live traffic () to monitor performance and cost in real-time.
4
Iterate with new insights
Insights from production monitoring reveal edge cases and opportunities for improvement. These new examples are used to refine the capability, expand the ground truth collection, and begin the cycle anew.