LLM application architecture template

Production building blocks around a model: gateway, guardrails, orchestrator, cache, vector DB, tools and observability.

Use this templateFree, opens in your browser. No sign-up.

LLM application architecture template
Production checklist
Users (web, Slack, API)
API gateway: auth, rate limits
Guardrails: input + output checks
Orchestrator: prompts, routing, tools
LLM gateway: retries, fallback, budgets
Model providers (Claude, open-weights)
Semantic cache (Redis)
Vector DB: docs + embeddings
Tools + internal APIs
Observability: traces, tokens, cost, evals
Latency budget for each step (p95)
Prompts + model versions in git
Fallback model when the primary is down
Never log PII in plain text
Cost alerts per team and feature
LLM application architecture
retrieve
tool calls
logs
The LLM application architecture template as it opens in Kelvo: 10 boxes, 5 sticky notes and 9 links.

What's on the board

Production checklist
Latency budget for each step (p95) · Prompts + model versions in git · Fallback model when the primary is down · Never log PII in plain text · Cost alerts per team and feature

Also on the board

  • Users (web, Slack, API)
  • Model providers (Claude, open-weights)
  • Orchestrator: prompts, routing, tools
  • LLM gateway: retries, fallback, budgets
  • API gateway: auth, rate limits
  • Guardrails: input + output checks
  • Observability: traces, tokens, cost, evals
  • Vector DB: docs + embeddings
  • Semantic cache (Redis)
  • Tools + internal APIs

How to use this template

  1. Click Use this template. It opens in the Kelvo app as a new board, saved in your browser.
  2. Double-click any box to change its text. Replace the examples with your own content.
  3. Sketch, then convert. Draw boxes and arrows with the pen, select the strokes and click Convert sketch: they become real boxes and links.
  4. Ask Kelvo right on the board (Pro). With Pro, type @kelvo and what you want in any box or text, for example “@kelvo add error handling”, and press Enter. The change is made there and the request disappears; ⌘Z undoes it.
  5. Keep the prompt next to the diagram. Add a code block from the Insert (+) menu for prompts, schemas or config, right beside the boxes they belong to.

Good for

  • ai
  • llm
  • architecture
  • architect
  • platform
  • Explaining an LLM feature's architecture to engineers and stakeholders
  • Deciding between prompting, retrieval and fine-tuning
  • Planning evaluations and risks before launch

Related templates

  • LLM agent loop template previewLLM agent loop

    How a tool-using agent plans, calls tools, observes results and updates memory until the goal is met, with guardrails.

  • RAG pipeline template previewRAG pipeline

    Retrieval-augmented generation end to end: ingest and embed documents offline, then retrieve, rerank and answer with citations.

  • Prompt design canvas template previewPrompt design canvas

    Design a production prompt piece by piece — role, context, task, constraints, examples, output format — plus test cases.

  • Model evaluation plan template previewModel evaluation plan

    Metrics, datasets, targets and owners for evaluating an LLM or ML model, plus the gate a release must pass.

  • C4 system context diagram template previewC4 system context diagram

    C4 level 1: the system in scope, the people who use it and the external systems it depends on, with a legend.

  • C4 container diagram template previewC4 container diagram

    C4 level 2: the apps, APIs, databases and queues inside the system boundary, with the technology on each link.

All ai and machine learning diagram templates · All templates

Use the LLM application architecture template