Configuration
llmrun.yaml catalog, default values, per-model fields, and the settings resolution order.
llmrun.yaml
Running llmrun init scaffolds an llmrun.yaml in the current directory. This file defines both the model catalog and the defaults for your project.
defaults:
aws_profile: default # AWS CLI profile to use
aws_region: eu-central-1 # AWS region to deploy into
idle_timeout: 30m # auto-stop after this much inactivity
base_port: 8000 # local ports are allocated from here upward
engine: vllm
models:
- alias: fast-7b
hf_repo: Qwen/Qwen2.5-7B-Instruct
instance_type: g6.xlarge
disk_gb: 100
context_length: 8192
# quantization: awq # optional: awq, gptq, fp8
# tool_call_parser: qwen3_xml # optional: vLLM tool-call parser (qwen3_xml, hermes, glm, …)
# hf_token_env: HF_TOKEN # env var holding your HF token (gated repos)
# idle_timeout: 1h # per-model override of the global defaultDefaults
| Field | Default | Description |
|---|---|---|
aws_profile | default | AWS CLI profile (~/.aws/credentials) |
aws_region | — | AWS region. Required — set it or pass --region |
idle_timeout | 30m | Auto-stop after inactivity. Formats: 30m, 1h, 90s |
base_port | 8000 | First local port. Each deployment gets the next free one |
engine | vllm | Serving engine (only vllm supported in v0.1) |
Per-Model Fields
These fields are defined under models: and control per-deployment behavior.
Required
| Field | Description |
|---|---|
alias | CLI-friendly label for this model (shown in deployment picker) |
hf_repo | HuggingFace model ID (e.g., Qwen/Qwen2.5-7B-Instruct) |
instance_type | EC2 instance type (e.g., g5.xlarge, g6e.12xlarge) |
disk_gb | EBS volume size in GB (model weights are cached here) |
Optional
| Field | Default | Description |
|---|---|---|
context_length | — | vLLM --max-model-len (max sequence length in tokens) |
quantization | — | Quantization type: awq, gptq, fp8 (for pre-quantized repos) |
tool_call_parser | — | vLLM --tool-call-parser for tool/function calling. Values: qwen3_xml, hermes, glm, … — must match the model family |
hf_token_env | — | Environment variable holding HF token (for gated/private repos) |
idle_timeout | — | Per-model override of global idle_timeout. Formats: 30m, 1h |
cpu_fallback | — | CPU fallback block (instance_type, engine: ollama|vllm, quantization, max_params, context_length, vllm tuning) — used when GPU quota is unavailable; see Model Catalog |
Resolution order
Settings are resolved in this priority (highest first):
- CLI flags (
--profile,--region) - Environment variables (
AWS_PROFILE,AWS_REGION) llmrun.yamldefaultsin the current directory~/.llmrun/config.jsonglobal config- AWS CLI defaults
Global config
llmrun config # show resolved config
llmrun config --set region=us-east-1 # set a valueGlobal config is stored at ~/.llmrun/config.json and applies across all projects that don't have their own llmrun.yaml.
Deployment state
Each deployment's state (instance ID, local port, region, profile, forward PID) is persisted at:
~/.llmrun/deployments/<name>/deployment.jsonThe Terraform workspace lives alongside it:
~/.llmrun/deployments/<name>/terraform/llmrun down removes both.
Custom home directory
By default llmrun stores all state under ~/.llmrun. Override it with the LLMRUN_HOME environment variable:
export LLMRUN_HOME=/data/llmrunThis affects all paths — global config, deployment state, and Terraform workspaces. Useful for:
- Keeping state on a separate volume
- Running multiple isolated llmrun environments side by side
- CI/CD pipelines where writing to
$HOMEis undesirable
# run with a project-local state directory
LLMRUN_HOME=./.llmrun-state llmrun up