llmrun
Guide

Configuration

llmrun.yaml catalog, default values, per-model fields, and the settings resolution order.

llmrun.yaml

Running llmrun init scaffolds an llmrun.yaml in the current directory. This file defines both the model catalog and the defaults for your project.

defaults:
    aws_profile: default        # AWS CLI profile to use
    aws_region: eu-central-1    # AWS region to deploy into
    idle_timeout: 30m           # auto-stop after this much inactivity
    base_port: 8000             # local ports are allocated from here upward
    engine: vllm

models:
    - alias: fast-7b
      hf_repo: Qwen/Qwen2.5-7B-Instruct
      instance_type: g6.xlarge
      disk_gb: 100
      context_length: 8192
      # quantization: awq                   # optional: awq, gptq, fp8
      # tool_call_parser: qwen3_xml         # optional: vLLM tool-call parser (qwen3_xml, hermes, glm, …)
      # hf_token_env: HF_TOKEN             # env var holding your HF token (gated repos)
      # idle_timeout: 1h                    # per-model override of the global default

Defaults

FieldDefaultDescription
aws_profiledefaultAWS CLI profile (~/.aws/credentials)
aws_region—AWS region. Required — set it or pass --region
idle_timeout30mAuto-stop after inactivity. Formats: 30m, 1h, 90s
base_port8000First local port. Each deployment gets the next free one
enginevllmServing engine (only vllm supported in v0.1)

Per-Model Fields

These fields are defined under models: and control per-deployment behavior.

Required

FieldDescription
aliasCLI-friendly label for this model (shown in deployment picker)
hf_repoHuggingFace model ID (e.g., Qwen/Qwen2.5-7B-Instruct)
instance_typeEC2 instance type (e.g., g5.xlarge, g6e.12xlarge)
disk_gbEBS volume size in GB (model weights are cached here)

Optional

FieldDefaultDescription
context_length—vLLM --max-model-len (max sequence length in tokens)
quantization—Quantization type: awq, gptq, fp8 (for pre-quantized repos)
tool_call_parser—vLLM --tool-call-parser for tool/function calling. Values: qwen3_xml, hermes, glm, … — must match the model family
hf_token_env—Environment variable holding HF token (for gated/private repos)
idle_timeout—Per-model override of global idle_timeout. Formats: 30m, 1h
cpu_fallback—CPU fallback block (instance_type, engine: ollama|vllm, quantization, max_params, context_length, vllm tuning) — used when GPU quota is unavailable; see Model Catalog

Resolution order

Settings are resolved in this priority (highest first):

  1. CLI flags (--profile, --region)
  2. Environment variables (AWS_PROFILE, AWS_REGION)
  3. llmrun.yaml defaults in the current directory
  4. ~/.llmrun/config.json global config
  5. AWS CLI defaults

Global config

llmrun config                           # show resolved config
llmrun config --set region=us-east-1   # set a value

Global config is stored at ~/.llmrun/config.json and applies across all projects that don't have their own llmrun.yaml.

Deployment state

Each deployment's state (instance ID, local port, region, profile, forward PID) is persisted at:

~/.llmrun/deployments/<name>/deployment.json

The Terraform workspace lives alongside it:

~/.llmrun/deployments/<name>/terraform/

llmrun down removes both.

Custom home directory

By default llmrun stores all state under ~/.llmrun. Override it with the LLMRUN_HOME environment variable:

export LLMRUN_HOME=/data/llmrun

This affects all paths — global config, deployment state, and Terraform workspaces. Useful for:

  • Keeping state on a separate volume
  • Running multiple isolated llmrun environments side by side
  • CI/CD pipelines where writing to $HOME is undesirable
# run with a project-local state directory
LLMRUN_HOME=./.llmrun-state llmrun up

On this page