llmrun
Guide

Getting Started

Install llmrun, verify your setup, and spin up your first OpenAI-compatible endpoint.

Prerequisites

Before installing llmrun you need:

Run llmrun doctor at any time to check all of these automatically.

Install

npm install -g llmrun

Quick start

# 1. Scaffold a catalog in the current folder
llmrun init

# 2. Verify all prerequisites
llmrun doctor

# 3. See what models are available
llmrun models

# 4. Pick a model, approve the cost estimate, provision and connect
llmrun up

llmrun up provisions the instance with Terraform, then waits for the model to come up. While it's waiting, open a second terminal and watch the vLLM warm-up (model download and GPU load):

llmrun logs

When llmrun up completes, the model is available at http://localhost:8000/v1 — a full OpenAI-compatible endpoint.

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{ "role": "user", "content": "Hello!" }]
  }'

Lifecycle

llmrun ls               # see all deployments, their local URL and connection state
llmrun logs             # tail the vLLM server logs
llmrun stop             # stop the instance (keeps EBS + model cache, ~$0.10/day)
llmrun start            # restart and re-forward
llmrun down             # destroy everything (terraform destroy)

Dashboard, cost, and history

llmrun studio           # local web dashboard at 127.0.0.1: live metrics, logs,
                        # ChatGPT-style chat against the served model, and full cost history
llmrun cost --since 30d # compute + stopped-storage cost from the local usage ledger
llmrun history          # every deployment ever recorded (including destroyed ones)
llmrun history export backup.zip  # single-file backup of the ledger

See the Studio guide for a tour of the dashboard, with screenshots.

All usage is tracked in a durable, local-only ledger (~/.llmrun/history.jsonl) that survives llmrun down — so totals and saved studio chat conversations exist regardless of whether a deployment is still around. llmrun history --clear is the only deletion path.

AWS setup

llmrun needs an AWS account with:

  • IAM permissions to create EC2 instances, IAM roles, and security groups
  • GPU vCPU quota in your target region — fresh accounts often have a limit of 0 for G/P instance families

Check quota and get a link to request an increase:

llmrun doctor

First run takes longer

On first boot the instance downloads the model weights from HuggingFace and loads them into GPU memory. A 7B model takes ~5 minutes; a 32B model can take 20–30 minutes. Subsequent llmrun start commands skip the download — weights are cached on the EBS volume.

On this page