ML experiment tracker template

Log every model experiment with its hypothesis, change, metrics and decision, so the team knows what worked and what to try next.

Use this templateFree, opens in your browser. No sign-up.

ML experiment tracker template
Next up
Learnings
Conventions
Model: card fraud classifier · metric: validation AUC (higher is better) · latency budget 10 ms
Run
Hypothesis
Change
Data
Val AUC
p95 latency
Decision
exp-041
Baseline
XGBoost, 120 features
v3 (Oct)
0.842
4 ms
Baseline
exp-042
Recent activity predicts fraud
+14 velocity features (1 h, 24 h, 7 d)
v3 (Oct)
0.861
5 ms
✓ Keep
exp-043
Deeper trees capture interactions
max_depth 6 → 10
v3 (Oct)
0.858
9 ms
✗ Overfits
exp-044
LightGBM is faster at equal quality
Swap to LightGBM
v3 (Oct)
0.863
3 ms
✓ Keep
exp-045
Rare fraud is under-weighted
Class weights 1 : 20
v3 (Oct)
0.866
3 ms
✓ Promote to staging
exp-046
Device fingerprint adds noise
−9 device features
v4 (Nov)
0.849
3 ms
✗ Revert
Merchant category embeddings
Retrain on v4 with Nov chargebacks
Calibrate scores (isotonic)
Velocity features gave the biggest jump
Depth > 8 overfits on v3
Device data is noisy until v5 cleanup
Every run logged in MLflow with git SHA
Same test split for all runs
One change per experiment
ML experiment tracker
The ML experiment tracker template as it opens in Kelvo: 9 sticky notes and 1 table, in 3 sections.

What's on the board

Next up
Merchant category embeddings · Retrain on v4 with Nov chargebacks · Calibrate scores (isotonic)
Learnings
Velocity features gave the biggest jump · Depth > 8 overfits on v3 · Device data is noisy until v5 cleanup
Conventions
Every run logged in MLflow with git SHA · Same test split for all runs · One change per experiment

Table columns: Run, Hypothesis, Change, Data, Val AUC, p95 latency, Decision.

Also on the board

  • Model: card fraud classifier · metric: validation AUC (higher is better) · latency…

How to use this template

  1. Click Use this template. It opens in the Kelvo app as a new board, saved in your browser.
  2. Double-click any box to change its text. Replace the examples with your own content.
  3. Sketch, then convert. Draw boxes and arrows with the pen, select the strokes and click Convert sketch: they become real boxes and links.
  4. Ask Kelvo right on the board (Pro). With Pro, type @kelvo and what you want in any box or text, for example “@kelvo add error handling”, and press Enter. The change is made there and the request disappears; ⌘Z undoes it.
  5. Keep the prompt next to the diagram. Add a code block from the Insert (+) menu for prompts, schemas or config, right beside the boxes they belong to.

Good for

  • ai
  • machine learning
  • experiments
  • data science
  • mlflow
  • Explaining an LLM feature's architecture to engineers and stakeholders
  • Deciding between prompting, retrieval and fine-tuning
  • Planning evaluations and risks before launch

Related templates

  • MLOps lifecycle template previewMLOps lifecycle

    The machine learning loop from problem framing to monitored production model, with retraining when data drifts.

  • Data labeling pipeline template previewData labeling pipeline

    Who does what when labeling training data: guidelines, model pre-labels, double annotation, QA on agreement and retraining.

  • ML feature store template previewML feature store

    How a feature store serves the same features to training and real-time inference: batch and streaming pipelines, offline and online stores.

  • RAG pipeline template previewRAG pipeline

    Retrieval-augmented generation end to end: ingest and embed documents offline, then retrieve, rerank and answer with citations.

  • Opportunity solution tree template previewOpportunity solution tree

    Connect one outcome to customer opportunities, candidate solutions and the experiments that test them.

  • Product discovery board template previewProduct discovery board

    Run continuous discovery on one board: the outcome, customer opportunities, ideas, riskiest assumptions and an experiment log.

All ai and machine learning diagram templates · All templates

Use the ML experiment tracker template