Data labeling pipeline template
Who does what when labeling training data: guidelines, model pre-labels, double annotation, QA on agreement and retraining.
Use this templateFree, opens in your browser. No sign-up.
What's on the board
- ML engineers
- Raw data: 50k support tickets · Write labeling guide + 25 gold examples · Accepted labels → versioned dataset
- Model (pre-labeling)
- Pre-label + pick the most uncertain items · Retrain model on the new labels
- Annotators
- Label in Label Studio, 2 annotators per item
- QA reviewers
- Agreement ≥ 0.8? · Adjudicate + update the guide
- Quality & cost
- Each annotator ≥ 95% on the gold set · Cohen's kappa tracked per label · ~1,200 items / day at $0.08 each · Re-check 5% of accepted labels weekly
Table columns: Label, Definition, Example.
Also on the board
- Label taxonomy
How to use this template
- Click Use this template. It opens in the Kelvo app as a new board, saved in your browser.
- Double-click any box to change its text. Replace the examples with your own content.
- Sketch, then convert. Draw boxes and arrows with the pen, select the strokes and click Convert sketch: they become real boxes and links.
- Ask Kelvo right on the board (Pro). With Pro, type @kelvo and what you want in any box or text, for example “@kelvo add error handling”, and press Enter. The change is made there and the request disappears; ⌘Z undoes it.
- Keep the prompt next to the diagram. Add a code block from the Insert (+) menu for prompts, schemas or config, right beside the boxes they belong to.
Good for
- ai
- machine learning
- data labeling
- annotation
- active learning
- Explaining an LLM feature's architecture to engineers and stakeholders
- Deciding between prompting, retrieval and fine-tuning
- Planning evaluations and risks before launch
Related templates
MLOps lifecycle
The machine learning loop from problem framing to monitored production model, with retraining when data drifts.
ML feature store
How a feature store serves the same features to training and real-time inference: batch and streaming pipelines, offline and online stores.
ML experiment tracker
Log every model experiment with its hypothesis, change, metrics and decision, so the team knows what worked and what to try next.
RAG pipeline
Retrieval-augmented generation end to end: ingest and embed documents offline, then retrieve, rerank and answer with citations.
LLM agent loop
How a tool-using agent plans, calls tools, observes results and updates memory until the goal is met, with guardrails.
Prompt design canvas
Design a production prompt piece by piece — role, context, task, constraints, examples, output format — plus test cases.
All ai and machine learning diagram templates · All templates