Data labeling pipeline template

Who does what when labeling training data: guidelines, model pre-labels, double annotation, QA on agreement and retraining.

Use this templateFree, opens in your browser. No sign-up.

Data labeling pipeline template
ML engineers
Model (pre-labeling)
Annotators
QA reviewers
Quality & cost
Raw data: 50k support tickets
Write labeling guide + 25 gold examples
Pre-label + pick the most uncertain items
Label in Label Studio, 2 annotators per item
Agreement ≥ 0.8?
Adjudicate + update the guide
Accepted labels → versioned dataset
Retrain model on the new labels
Label taxonomy
Label
Definition
Example
Billing
Charges, refunds, invoices
'I was charged twice in March'
Bug report
Something is broken or wrong
'Export button does nothing'
Feature request
Asks for something new
'Can you add dark mode?'
Account access
Login, SSO, 2FA problems
'Locked out after reset'
Other
None of the above; flag for review
'Do you have a sticker?'
Each annotator ≥ 95% on the gold set
Cohen's kappa tracked per label
~1,200 items / day at $0.08 each
Re-check 5% of accepted labels weekly
Data labeling pipeline
yes
no
clarify rules
next batch
The data labeling pipeline template as it opens in Kelvo: 8 boxes, 1 decision among them, 4 sticky notes, 1 table and 9 links, in 5 sections.

What's on the board

ML engineers
Raw data: 50k support tickets · Write labeling guide + 25 gold examples · Accepted labels → versioned dataset
Model (pre-labeling)
Pre-label + pick the most uncertain items · Retrain model on the new labels
Annotators
Label in Label Studio, 2 annotators per item
QA reviewers
Agreement ≥ 0.8? · Adjudicate + update the guide
Quality & cost
Each annotator ≥ 95% on the gold set · Cohen's kappa tracked per label · ~1,200 items / day at $0.08 each · Re-check 5% of accepted labels weekly

Table columns: Label, Definition, Example.

Also on the board

  • Label taxonomy

How to use this template

  1. Click Use this template. It opens in the Kelvo app as a new board, saved in your browser.
  2. Double-click any box to change its text. Replace the examples with your own content.
  3. Sketch, then convert. Draw boxes and arrows with the pen, select the strokes and click Convert sketch: they become real boxes and links.
  4. Ask Kelvo right on the board (Pro). With Pro, type @kelvo and what you want in any box or text, for example “@kelvo add error handling”, and press Enter. The change is made there and the request disappears; ⌘Z undoes it.
  5. Keep the prompt next to the diagram. Add a code block from the Insert (+) menu for prompts, schemas or config, right beside the boxes they belong to.

Good for

  • ai
  • machine learning
  • data labeling
  • annotation
  • active learning
  • Explaining an LLM feature's architecture to engineers and stakeholders
  • Deciding between prompting, retrieval and fine-tuning
  • Planning evaluations and risks before launch

Related templates

  • MLOps lifecycle template previewMLOps lifecycle

    The machine learning loop from problem framing to monitored production model, with retraining when data drifts.

  • ML feature store template previewML feature store

    How a feature store serves the same features to training and real-time inference: batch and streaming pipelines, offline and online stores.

  • ML experiment tracker template previewML experiment tracker

    Log every model experiment with its hypothesis, change, metrics and decision, so the team knows what worked and what to try next.

  • RAG pipeline template previewRAG pipeline

    Retrieval-augmented generation end to end: ingest and embed documents offline, then retrieve, rerank and answer with citations.

  • LLM agent loop template previewLLM agent loop

    How a tool-using agent plans, calls tools, observes results and updates memory until the goal is met, with guardrails.

  • Prompt design canvas template previewPrompt design canvas

    Design a production prompt piece by piece — role, context, task, constraints, examples, output format — plus test cases.

All ai and machine learning diagram templates · All templates

Use the data labeling pipeline template