How to Set Up LiteLLM Proxy with Docker (2026)

What is LiteLLM?

LiteLLM (github.com/BerriAI/litellm, MIT license) is a Python proxy that accepts OpenAI-format API requests and routes them to the right provider SDK. When your code calls `http://localhost:4000/v1/chat/completions` with model `claude-sonnet`, LiteLLM intercepts it, translates the request to Anthropic's Messages API format, adds your Anthropic API key, and returns the response in the OpenAI format your application expects.

Every provider uses the same config syntax. Switching from GPT-4o to Gemini 1.5 Pro means changing one field in `litellm_config.yaml`. No SDK calls to update, no auth logic to rewrite.

Where LiteLLM fits in a typical AI stack

Without LiteLLM, your application carries direct dependencies on multiple provider SDKs. Each has its own auth format, rate limit behavior, and error structure. With LiteLLM, your application calls one local endpoint. Provider credentials, model routing, retry logic, and error normalization all live in the proxy config. Your application code doesn't need to know which provider it's talking to.

The setup makes most sense in a few situations: when a team shares API keys across developers, when you need automatic failover if a provider starts returning 429s, or when you're mixing cloud APIs with a local Ollama instance and want one endpoint for both.

Step 1: Create the Configuration File

The `litellm_config.yaml` file defines which models LiteLLM exposes and which provider credentials to use for each. It is the only required configuration file.

Create the project directory

mkdir litellm-proxy
cd litellm-proxy

Write the minimal config

Create `litellm_config.yaml` with one model entry to start:

model_list:
  - model_name: gpt-5.2
    litellm_params:
      model: openai/gpt-5.2
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

The `model_name` field is what your application sends in the `model` parameter. The `litellm_params.model` value uses the provider-prefix format: `provider/model-id`. The `os.environ/VARIABLE_NAME` syntax tells LiteLLM to read the value from an environment variable at runtime instead of hardcoding it in the YAML.

The `master_key` is the admin key required on every API request. It also lets you generate budget-capped virtual keys for team members. Use a long random string.

Create the environment file

Store secrets in a `.env` file in the same directory:

# .env
OPENAI_API_KEY=sk-your-actual-openai-key
LITELLM_MASTER_KEY=sk-your-chosen-master-key

echo ".env" >> .gitignore

Step 2: Start LiteLLM with Docker Compose

Create `docker-compose.yaml` in the same directory as `litellm_config.yaml`:

version: "3.8"

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    ports:
      - "4000:4000"
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
    command: ["--config", "/app/config.yaml", "--port", "4000", "--num_workers", "8"]
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4000/health/liveliness"]
      interval: 30s
      timeout: 10s
      retries: 3

The `main-stable` tag pulls the latest stable LiteLLM release. The `--num_workers 8` flag sets the number of Gunicorn workers. For a single developer, 4 workers is enough. For a team gateway handling concurrent requests, 8 to 16 workers handles load more cleanly.

Start the proxy:

docker compose up -d

Check the startup logs:

docker compose logs litellm
# Expected:
# INFO: Application startup complete.
# Uvicorn running on http://0.0.0.0:4000 (Press CTRL+C to quit)

The first run pulls the LiteLLM Docker image, which is approximately 2 GB. On a standard 100 Mbps connection, this takes 2 to 4 minutes. Subsequent starts are instant since the image is cached locally.

Step 3: Test the Proxy and Send Your First Request

Confirm the proxy is healthy before making model calls:

curl http://localhost:4000/health
# Expected:
# {"status":"healthy","litellm_version":"1.40.x","environment_variables":{...}}

Send a test chat completion. Replace the master key with what you set in `.env`:

curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-your-master-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.2",
    "messages": [{"role": "user", "content": "Reply with the word OK only."}],
    "max_tokens": 5
  }'

A working proxy returns a standard OpenAI-format JSON response with a `choices` array. The `model` field in the response shows which provider model handled the request.

Using the Python OpenAI SDK

Because LiteLLM is OpenAI-compatible, any code that uses the OpenAI Python SDK works unchanged. Swap only `base_url` and `api_key`:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000/v1",
    api_key="sk-your-master-key"
)

response = client.chat.completions.create(
    model="gpt-5.2",
    messages=[{"role": "user", "content": "Reply with OK only."}]
)
print(response.choices[0].message.content)
# Output: OK

This pattern works with any library that accepts an OpenAI client: LangChain, LlamaIndex, Instructor, the Vercel AI SDK, and the raw `openai` package in Python, Node.js, or TypeScript. Nothing in the calling code needs to change when you add, remove, or swap providers in the LiteLLM config.

If you prefer using Ollama's native Python library without a proxy layer, see the Ollama Python API guide.

Step 4: Add Multiple LLM Providers

Extend `litellm_config.yaml` with additional providers. Each entry uses the same structure; only the provider prefix and API key variable change.

Add OpenAI, Anthropic, and Groq

model_list:
  - model_name: gpt-5.2
    litellm_params:
      model: openai/gpt-5.2
      api_key: os.environ/OPENAI_API_KEY

  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-6
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: llama-groq
    litellm_params:
      model: groq/llama-3.3-70b-versatile
      api_key: os.environ/GROQ_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Add the new keys to `.env`:

ANTHROPIC_API_KEY=sk-ant-your-key
GROQ_API_KEY=gsk_your-groq-key

Update the `environment` block in `docker-compose.yaml`:

    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - GROQ_API_KEY=${GROQ_API_KEY}
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}

Restart to apply the config change:

docker compose restart litellm

Add a local Ollama model

If Ollama is running on the same machine, use `host.docker.internal` so the container can reach the host's port 11434:

  - model_name: llama-local
    litellm_params:
      model: ollama/llama3.3:8b
      api_base: http://host.docker.internal:11434

On Linux, `host.docker.internal` is not automatically available. Add this to the `litellm` service block in `docker-compose.yaml`:

    extra_hosts:
      - "host.docker.internal:host-gateway"

After restarting, the `llama-local` model is accessible through the proxy at no API cost. Requests to it never leave your machine.

Step 5: Generate and Use Virtual API Keys

Virtual keys let each developer or application use their own API key with independent budget caps and model restrictions. The master key creates and manages virtual keys. Virtual keys are what end users and CI systems use for day-to-day requests.

Generate a virtual key with a budget cap

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-your-master-key" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["gpt-5.2", "claude-sonnet", "llama-local"],
    "max_budget": 20.0,
    "budget_duration": "30d",
    "key_alias": "dev-alice"
  }'

The response includes a `key` field with the new virtual key:

{
  "key": "sk-abc123generated...",
  "models": ["gpt-5.2", "claude-sonnet", "llama-local"],
  "max_budget": 20.0,
  "budget_duration": "30d",
  "key_alias": "dev-alice"
}

Give `sk-abc123generated...` to the developer. They use it exactly like an OpenAI API key, pointed at `http://localhost:4000/v1`. When the key reaches the $20 budget, subsequent requests return a 429 until the 30-day period resets.

Key parameters

List and revoke keys

# List all active keys
curl http://localhost:4000/key/list \
  -H "Authorization: Bearer sk-your-master-key"

# Revoke a specific key
curl -X DELETE http://localhost:4000/key/delete \
  -H "Authorization: Bearer sk-your-master-key" \
  -H "Content-Type: application/json" \
  -d '{"keys": ["sk-abc123generated..."]}'

Step 6: Enable Cost Tracking with PostgreSQL

By default, LiteLLM tracks spend in an in-memory SQLite database that resets when the container restarts. Connect PostgreSQL to persist spend data across restarts and deployments.

Update docker-compose.yaml to add PostgreSQL

version: "3.8"

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    ports:
      - "4000:4000"
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - DATABASE_URL=postgresql://litellm:${DB_PASSWORD}@db:5432/litellm
    command: ["--config", "/app/config.yaml", "--port", "4000", "--num_workers", "8"]
    depends_on:
      db:
        condition: service_healthy
    restart: unless-stopped

  db:
    image: postgres:16
    environment:
      - POSTGRES_DB=litellm
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=${DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 10s
      timeout: 5s
      retries: 5

volumes:
  postgres_data:

Add `DB_PASSWORD` to `.env`:

DB_PASSWORD=a-strong-random-password-here

Start the full stack:

docker compose down
docker compose up -d

LiteLLM runs its own database migrations on first startup with a new PostgreSQL URL. Check the logs to confirm migration completed:

docker compose logs litellm | grep -i "migration"
# Expected: INFO: Running migrations...
# Expected: INFO: Migration complete

Query spend data

# Spend summary for a specific virtual key
curl "http://localhost:4000/key/info?key=sk-abc123" \
  -H "Authorization: Bearer sk-your-master-key"

# Global spend by model (specify date range)
curl "http://localhost:4000/global/spend/models?start_date=2026-06-01&end_date=2026-06-30" \
  -H "Authorization: Bearer sk-your-master-key"

Step 7: Configure Load Balancing and Fallbacks

LiteLLM load-balances automatically when multiple config entries share the same `model_name`. Requests to `gpt-5.2` distribute across all `gpt-5.2` entries using the routing strategy you set.

Load balance across multiple API keys

Use this to spread requests across several OpenAI API keys and stay within per-key rate limits:

model_list:
  - model_name: gpt-5.2
    litellm_params:
      model: openai/gpt-5.2
      api_key: os.environ/OPENAI_API_KEY_1

  - model_name: gpt-5.2
    litellm_params:
      model: openai/gpt-5.2
      api_key: os.environ/OPENAI_API_KEY_2

router_settings:
  routing_strategy: least-busy
  num_retries: 3
  retry_after: 5

The `least-busy` strategy sends each request to the entry with the fewest active in-flight requests. Other available strategies:

Fallback to a backup provider

Fallbacks automatically retry failed requests using a different model or provider:

router_settings:
  routing_strategy: least-busy
  num_retries: 3
  retry_after: 5
  fallbacks:
    - {"gpt-5.2": ["claude-sonnet"]}
    - {"claude-sonnet": ["llama-groq"]}

With this config, a 429 rate limit or 500 error from `gpt-5.2` after 3 retries automatically escalates to `claude-sonnet`. If that also fails, the request goes to `llama-groq`. Your application sees a normal successful response in all cases.

Context window fallbacks

For document processing where input length is unpredictable, fall back to a model with a larger context window:

router_settings:
  context_window_fallbacks:
    - {"gpt-5.2": ["claude-sonnet"]}

When a request exceeds GPT-5.2's context limit, LiteLLM automatically retries with Claude Sonnet, which supports a 200,000-token context window. The fallback model must already exist in your `model_list`.

Apply any config change by restarting the proxy:

docker compose restart litellm

Configuration Reference

Key environment variables

Provider-specific keys (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GROQ_API_KEY`, etc.) are read from environment variables by the `os.environ/VARIABLE_NAME` syntax in `litellm_config.yaml`.

Key config.yaml settings

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

  # Disable auth for local-only development (never use in production)
  no_auth: false

  # Global maximum spend across all keys
  max_budget: 100.0

  # Rate limit for all requests (requests per minute)
  rpm_limit: 200

litellm_settings:
  # Retries on provider errors before declaring failure
  num_retries: 3

  # Request timeout in seconds per provider call
  request_timeout: 120

  # Drop unsupported parameters instead of erroring (useful when mixing providers)
  drop_params: true

  # Enable verbose logging for request/response debugging
  set_verbose: false

Available provider prefixes (selected)

Full provider list and config options are at docs.litellm.ai/docs/providers.

Troubleshooting

Frequently Asked Questions

- model_name: claude-sonnet
  litellm_params:
    model: anthropic/claude-sonnet-4-6
    api_key: os.environ/ANTHROPIC_API_KEY

from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
    base_url="http://localhost:4000/v1",
    api_key="sk-your-master-key",
    model="gpt-5.2"
)

from llama_index.llms.openai import OpenAI
llm = OpenAI(base_url="http://localhost:4000/v1", api_key="sk-your-master-key", model="gpt-5.2")

model_list:
  - model_name: llama3
    litellm_params:
      model: ollama/llama3.3:8b
      api_base: http://host.docker.internal:11434

general_settings:
  master_key: sk-local-dev-key

docker compose pull litellm
docker compose up -d --force-recreate litellm

# Spend for a specific virtual key
curl "http://localhost:4000/key/info?key=sk-abc123" \
  -H "Authorization: Bearer sk-your-master-key"

# Global spend broken down by model over a date range
curl "http://localhost:4000/global/spend/models?start_date=2026-06-01&end_date=2026-06-30" \
  -H "Authorization: Bearer sk-your-master-key"