Model evaluation plan template

Metrics, datasets, targets and owners for evaluating an LLM or ML model, plus the gate a release must pass.

Use this templateFree, opens in your browser. No sign-up.

Model evaluation plan template
Release gate
Metric
What it tells us
Dataset
Target
Owner
Task accuracy
Correct answers vs gold labels
Golden set (500 questions)
≥ 90%
ML eng
Faithfulness
Answer is supported by the context
Golden set + LLM judge
≥ 95%
ML eng
Answer relevance
Answers the actual question
Prod sample (200 / week)
≥ 4 / 5
PM
Harmful output rate
Toxic, biased or unsafe replies
Red-team set (300 prompts)
< 0.5%
Trust & safety
Latency p95
Time to the full answer
Load test
< 3 s
Platform
Cost per 1k requests
Tokens × price
Production logs
< $4
Platform
Human preference
Win rate vs current version
Side-by-side review (100 pairs)
≥ 55%
PM
All targets met on the golden set
No new P0 safety failures
A/B test on 10% of traffic for 1 week
Re-run on every prompt or model change
Model evaluation plan
The model evaluation plan template as it opens in Kelvo: 4 sticky notes and 1 table.

What's on the board

Release gate
All targets met on the golden set · No new P0 safety failures · A/B test on 10% of traffic for 1 week · Re-run on every prompt or model change

Table columns: Metric, What it tells us, Dataset, Target, Owner.

How to use this template

  1. Click Use this template. It opens in the Kelvo app as a new board, saved in your browser.
  2. Double-click any box to change its text. Replace the examples with your own content.
  3. Sketch, then convert. Draw boxes and arrows with the pen, select the strokes and click Convert sketch: they become real boxes and links.
  4. Ask Kelvo right on the board (Pro). With Pro, type @kelvo and what you want in any box or text, for example “@kelvo add error handling”, and press Enter. The change is made there and the request disappears; ⌘Z undoes it.
  5. Keep the prompt next to the diagram. Add a code block from the Insert (+) menu for prompts, schemas or config, right beside the boxes they belong to.

Good for

  • ai
  • llm
  • evaluation
  • quality
  • ml
  • Explaining an LLM feature's architecture to engineers and stakeholders
  • Deciding between prompting, retrieval and fine-tuning
  • Planning evaluations and risks before launch

Related templates

  • RAG pipeline template previewRAG pipeline

    Retrieval-augmented generation end to end: ingest and embed documents offline, then retrieve, rerank and answer with citations.

  • LLM agent loop template previewLLM agent loop

    How a tool-using agent plans, calls tools, observes results and updates memory until the goal is met, with guardrails.

  • Prompt design canvas template previewPrompt design canvas

    Design a production prompt piece by piece — role, context, task, constraints, examples, output format — plus test cases.

  • AI guardrails & risk register template previewAI guardrails & risk register

    Input and output guardrails around an LLM call, with a risk register for hallucination, injection, leakage and bias.

  • Code review checklist template previewCode review checklist

    What to look for in a pull request, from correctness to security, so reviews are fast and consistent.

  • Test pyramid (test strategy) template previewTest pyramid (test strategy)

    How many unit, integration and end-to-end tests to write, what each covers, and how fast they should run.

All ai and machine learning diagram templates · All templates

Use the model evaluation plan template