llmrun
Guide

Model Catalog

Anatomy of a catalog entry, VRAM sizing, gated models, multi-GPU, and tool calling.

Anatomy of a catalog entry

models:
    - alias: fast-7b                        # your CLI-friendly label (shown in picker)
      hf_repo: Qwen/Qwen2.5-7B-Instruct    # HuggingFace model to serve with vLLM
      instance_type: g6.xlarge              # EC2 instance type
      disk_gb: 100                          # EBS volume size (model weights are cached here)
      context_length: 8192                  # vLLM --max-model-len
      # quantization: awq                   # optional: awq, gptq, fp8
      # tool_call_parser: hermes            # optional: enables tool/function calling
      # hf_token_env: HF_TOKEN             # env var holding your HF token (gated repos)
      # idle_timeout: 1h                    # per-model override of the global default

All models are served from HuggingFace via vLLM's OpenAI-compatible server.

VRAM sizing

vLLM loads the full model into GPU memory. A rough rule:

  • fp16 (default): ~2 GB per 1B parameters — a 7B model needs ~14 GB, 32B needs ~64 GB
  • AWQ / GPTQ (4-bit): ~0.6 GB per 1B parameters — a 32B model fits in ~20 GB
  • fp8 / int8: ~1 GB per 1B parameters

llmrun up warns you before provisioning if the model likely won't fit the instance.

Common instance / model pairings

InstanceGPUVRAMFits
g6.xlarge1× L424 GB7B fp16, 13B fp16
g5.2xlarge1× A10G24 GB7B fp16, 13B fp16
g6e.2xlarge1× L40S48 GB32B AWQ, 13B fp16
g6e.12xlarge4× L40S192 GB32B fp16, 70B AWQ

Gated models

For models that require accepting HuggingFace terms (e.g. meta-llama/*), set your token:

export HF_TOKEN=hf_...

Then reference the env var in the catalog:

- alias: llama-8b
  hf_repo: meta-llama/Llama-3.1-8B-Instruct
  instance_type: g6.xlarge
  disk_gb: 100
  hf_token_env: HF_TOKEN

Multi-GPU models

For large models that need multiple GPUs, pick a multi-GPU instance. llmrun automatically detects the number of GPUs on boot and passes --tensor-parallel-size N to vLLM:

- alias: qwen-32b
  hf_repo: Qwen/Qwen2.5-32B-Instruct
  instance_type: g6e.12xlarge   # 4× L40S = 192 GB total VRAM
  disk_gb: 200
  context_length: 32768

Tool / function calling

By default vLLM starts with tool calling disabled, so clients that send tool_choice: "auto" (agentic coding tools, MCP-style clients) get a 400 error. Set tool_call_parser to enable it — llmrun passes both --enable-auto-tool-choice and --tool-call-parser <value> to vLLM:

- alias: fast-7b
  hf_repo: Qwen/Qwen2.5-7B-Instruct
  instance_type: g6.xlarge
  disk_gb: 100
  tool_call_parser: hermes

Pick the parser that matches the model family:

Model familytool_call_parser
Qwen 2.5hermes
Qwen 3 (incl. Coder)qwen3_xml
Llama 3.1 / 3.3llama3_json
Mistral / Mixtralmistral
Granitegranite

Check the vLLM tool calling docs for the full, up-to-date list — vLLM adds parsers for new model families over time. After changing this field, re-provision with llmrun down <name> → llmrun up.

CPU fallback

When GPU quota is unavailable, a model can define a CPU fallback. llmrun up checks the GPU vCPU quota before provisioning and offers the fallback interactively if the quota is insufficient. Two engines are supported:

  • ollama — GGUF-style CPU inference; much slower, for functional dev/testing.
  • vllm — vLLM's CPU build (official vllm/vllm-openai-cpu image) serving the HuggingFace repo directly. Slower than GPU, but a full OpenAI-compatible server with tuning knobs.
- alias: fast-7b
  hf_repo: Qwen/Qwen2.5-7B-Instruct
  instance_type: g6.xlarge
  disk_gb: 100
  cpu_fallback:
      instance_type: r8i.8xlarge      # 32 vCPU / 256 GB RAM
      engine: vllm                    # or: ollama
      # quantization: awq             # rarely needed — vLLM auto-detects from the
                                      # model's config.json; a mismatched value crashes
      max_params: 32B                 # guard against oversized models on CPU
      context_length: 32768           # CPU-only override — keep small (KV cache lives in RAM)
      vllm:                           # engine: vllm only
          image: vllm/vllm-openai-cpu:v0.28.0  # optional override (pinned by default; :latest intentionally not used)
          kvcache_space: 16           # VLLM_CPU_KVCACHE_SPACE, GiB (default: 16)
          omp_threads_bind: "0-31"    # VLLM_CPU_OMP_THREADS_BIND (default: all vCPUs)

Sizing on CPU: weights + kvcache_space + ~12 GB runtime overhead must fit in the instance's RAM. A 27B AWQ model needs ~50 GB total — r8i.8xlarge (256 GB) has plenty of headroom. llmrun up warns before provisioning when a model likely won't fit the instance's RAM.

CPU fallback instances

Approximate us-east-1 on-demand prices (the cost preview in llmrun up uses these):

InstancevCPURAM~$/hrFits (AWQ 4-bit, 16 GB KV cache)
c7i.4xlarge1632 GB$0.717B (tight)
m7i.4xlarge1664 GB$0.817B–13B
r7i.4xlarge16128 GB$1.0613B–27B
r8i.8xlarge32256 GB$2.2227B with headroom, 70B
r8i.12xlarge48384 GB$3.3370B+

Rule of thumb: ~0.7 GB of RAM per 1B params (4-bit weights) + 16 GB (KV cache) + ~12 GB (runtime).

Running multiple models at once

Each llmrun up creates an independent deployment on its own local port:

llmrun up   # picks fast-7b → localhost:8000/v1
llmrun up   # picks qwen-32b-awq → localhost:8001/v1

Use llmrun ls to see all running endpoints and llmrun connect --all to re-establish all forwards after a restart.

On this page