Model Catalog
Anatomy of a catalog entry, VRAM sizing, gated models, multi-GPU, and tool calling.
Anatomy of a catalog entry
models:
- alias: fast-7b # your CLI-friendly label (shown in picker)
hf_repo: Qwen/Qwen2.5-7B-Instruct # HuggingFace model to serve with vLLM
instance_type: g6.xlarge # EC2 instance type
disk_gb: 100 # EBS volume size (model weights are cached here)
context_length: 8192 # vLLM --max-model-len
# quantization: awq # optional: awq, gptq, fp8
# tool_call_parser: hermes # optional: enables tool/function calling
# hf_token_env: HF_TOKEN # env var holding your HF token (gated repos)
# idle_timeout: 1h # per-model override of the global defaultAll models are served from HuggingFace via vLLM's OpenAI-compatible server.
VRAM sizing
vLLM loads the full model into GPU memory. A rough rule:
- fp16 (default): ~2 GB per 1B parameters — a 7B model needs ~14 GB, 32B needs ~64 GB
- AWQ / GPTQ (4-bit): ~0.6 GB per 1B parameters — a 32B model fits in ~20 GB
- fp8 / int8: ~1 GB per 1B parameters
llmrun up warns you before provisioning if the model likely won't fit the instance.
Common instance / model pairings
| Instance | GPU | VRAM | Fits |
|---|---|---|---|
g6.xlarge | 1× L4 | 24 GB | 7B fp16, 13B fp16 |
g5.2xlarge | 1× A10G | 24 GB | 7B fp16, 13B fp16 |
g6e.2xlarge | 1× L40S | 48 GB | 32B AWQ, 13B fp16 |
g6e.12xlarge | 4× L40S | 192 GB | 32B fp16, 70B AWQ |
Gated models
For models that require accepting HuggingFace terms (e.g. meta-llama/*), set your token:
export HF_TOKEN=hf_...Then reference the env var in the catalog:
- alias: llama-8b
hf_repo: meta-llama/Llama-3.1-8B-Instruct
instance_type: g6.xlarge
disk_gb: 100
hf_token_env: HF_TOKENMulti-GPU models
For large models that need multiple GPUs, pick a multi-GPU instance. llmrun automatically detects the number of GPUs on boot and passes --tensor-parallel-size N to vLLM:
- alias: qwen-32b
hf_repo: Qwen/Qwen2.5-32B-Instruct
instance_type: g6e.12xlarge # 4× L40S = 192 GB total VRAM
disk_gb: 200
context_length: 32768Tool / function calling
By default vLLM starts with tool calling disabled, so clients that send tool_choice: "auto" (agentic coding tools, MCP-style clients) get a 400 error. Set tool_call_parser to enable it — llmrun passes both --enable-auto-tool-choice and --tool-call-parser <value> to vLLM:
- alias: fast-7b
hf_repo: Qwen/Qwen2.5-7B-Instruct
instance_type: g6.xlarge
disk_gb: 100
tool_call_parser: hermesPick the parser that matches the model family:
| Model family | tool_call_parser |
|---|---|
| Qwen 2.5 | hermes |
| Qwen 3 (incl. Coder) | qwen3_xml |
| Llama 3.1 / 3.3 | llama3_json |
| Mistral / Mixtral | mistral |
| Granite | granite |
Check the vLLM tool calling docs for the full, up-to-date list — vLLM adds parsers for new model families over time. After changing this field, re-provision with llmrun down <name> → llmrun up.
CPU fallback
When GPU quota is unavailable, a model can define a CPU fallback. llmrun up checks the GPU vCPU quota before provisioning and offers the fallback interactively if the quota is insufficient. Two engines are supported:
ollama— GGUF-style CPU inference; much slower, for functional dev/testing.vllm— vLLM's CPU build (officialvllm/vllm-openai-cpuimage) serving the HuggingFace repo directly. Slower than GPU, but a full OpenAI-compatible server with tuning knobs.
- alias: fast-7b
hf_repo: Qwen/Qwen2.5-7B-Instruct
instance_type: g6.xlarge
disk_gb: 100
cpu_fallback:
instance_type: r8i.8xlarge # 32 vCPU / 256 GB RAM
engine: vllm # or: ollama
# quantization: awq # rarely needed — vLLM auto-detects from the
# model's config.json; a mismatched value crashes
max_params: 32B # guard against oversized models on CPU
context_length: 32768 # CPU-only override — keep small (KV cache lives in RAM)
vllm: # engine: vllm only
image: vllm/vllm-openai-cpu:v0.28.0 # optional override (pinned by default; :latest intentionally not used)
kvcache_space: 16 # VLLM_CPU_KVCACHE_SPACE, GiB (default: 16)
omp_threads_bind: "0-31" # VLLM_CPU_OMP_THREADS_BIND (default: all vCPUs)Sizing on CPU: weights + kvcache_space + ~12 GB runtime overhead must fit in the instance's RAM. A 27B AWQ model needs ~50 GB total — r8i.8xlarge (256 GB) has plenty of headroom. llmrun up warns before provisioning when a model likely won't fit the instance's RAM.
CPU fallback instances
Approximate us-east-1 on-demand prices (the cost preview in llmrun up uses these):
| Instance | vCPU | RAM | ~$/hr | Fits (AWQ 4-bit, 16 GB KV cache) |
|---|---|---|---|---|
c7i.4xlarge | 16 | 32 GB | $0.71 | 7B (tight) |
m7i.4xlarge | 16 | 64 GB | $0.81 | 7B–13B |
r7i.4xlarge | 16 | 128 GB | $1.06 | 13B–27B |
r8i.8xlarge | 32 | 256 GB | $2.22 | 27B with headroom, 70B |
r8i.12xlarge | 48 | 384 GB | $3.33 | 70B+ |
Rule of thumb: ~0.7 GB of RAM per 1B params (4-bit weights) + 16 GB (KV cache) + ~12 GB (runtime).
Running multiple models at once
Each llmrun up creates an independent deployment on its own local port:
llmrun up # picks fast-7b → localhost:8000/v1
llmrun up # picks qwen-32b-awq → localhost:8001/v1Use llmrun ls to see all running endpoints and llmrun connect --all to re-establish all forwards after a restart.