Getting Started
Install llmrun, verify your setup, and spin up your first OpenAI-compatible endpoint.
Prerequisites
Before installing llmrun you need:
- Node.js ≥ 20
- Terraform — used to provision EC2 instances
- AWS CLI v2 with credentials configured (
aws configureor SSO) - AWS Session Manager plugin — for the SSM port-forward that exposes the local endpoint
Run llmrun doctor at any time to check all of these automatically.
Install
npm install -g llmrunQuick start
# 1. Scaffold a catalog in the current folder
llmrun init
# 2. Verify all prerequisites
llmrun doctor
# 3. See what models are available
llmrun models
# 4. Pick a model, approve the cost estimate, provision and connect
llmrun upllmrun up provisions the instance with Terraform, then waits for the model to come up. While it's waiting, open a second terminal and watch the vLLM warm-up (model download and GPU load):
llmrun logsWhen llmrun up completes, the model is available at http://localhost:8000/v1 — a full OpenAI-compatible endpoint.
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"messages": [{ "role": "user", "content": "Hello!" }]
}'Lifecycle
llmrun ls # see all deployments, their local URL and connection state
llmrun logs # tail the vLLM server logs
llmrun stop # stop the instance (keeps EBS + model cache, ~$0.10/day)
llmrun start # restart and re-forward
llmrun down # destroy everything (terraform destroy)Dashboard, cost, and history
llmrun studio # local web dashboard at 127.0.0.1: live metrics, logs,
# ChatGPT-style chat against the served model, and full cost history
llmrun cost --since 30d # compute + stopped-storage cost from the local usage ledger
llmrun history # every deployment ever recorded (including destroyed ones)
llmrun history export backup.zip # single-file backup of the ledgerSee the Studio guide for a tour of the dashboard, with screenshots.
All usage is tracked in a durable, local-only ledger (~/.llmrun/history.jsonl) that survives
llmrun down — so totals and saved studio chat conversations exist regardless of whether a
deployment is still around. llmrun history --clear is the only deletion path.
AWS setup
llmrun needs an AWS account with:
- IAM permissions to create EC2 instances, IAM roles, and security groups
- GPU vCPU quota in your target region — fresh accounts often have a limit of 0 for G/P instance families
Check quota and get a link to request an increase:
llmrun doctorFirst run takes longer
On first boot the instance downloads the model weights from HuggingFace and loads them into GPU memory. A 7B model takes ~5 minutes; a 32B model can take 20–30 minutes. Subsequent llmrun start commands skip the download — weights are cached on the EBS volume.