How to Run Ollama Locally: Complete Setup Guide (2026)

Install Ollama

Ollama provides a one-command installer for Linux and macOS, a Windows installer package, and a Docker image for containerised deployments.

Install on Linux or macOS

Run the official install script. It downloads the Ollama binary, sets up a systemd service on Linux, and adds the `ollama` command to your PATH.

curl -fsSL https://ollama.com/install.sh | sh

On macOS, Homebrew is also supported:

brew install ollama

After installation, verify it works:

ollama --version
# Expected output: ollama version 0.6.x

Install on Windows

Download the Windows installer from ollama.com/download. Run the `.exe` file and follow the prompts. Ollama installs to `%LOCALAPPDATA%\Programs\Ollama` and adds itself to PATH.

After installation, open a new Command Prompt or PowerShell window and verify:

ollama --version

Install with Docker

Docker is the recommended method for server deployments and production use. The official image is `ollama/ollama`.

# Pull the Ollama image
docker pull ollama/ollama

# Run Ollama in a container
# -d: run in background
# -v ollama:/root/.ollama: persist downloaded models to a named volume
# -p 11434:11434: expose the Ollama API on localhost port 11434
docker run -d \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

Verify the container is running:

docker ps
# Expected output:
# CONTAINER ID   IMAGE           COMMAND                  CREATED        STATUS        PORTS
# a1b2c3d4e5f6   ollama/ollama   "/bin/ollama serve"      5 seconds ago  Up 4 seconds  0.0.0.0:11434->11434/tcp

Pull and Run Your First Model

With Ollama installed, the next step is downloading a model. Ollama pulls models from the Ollama Library at ollama.com/library, which hosts 100+ pre-quantised models.

Recommended First Models

For most users with 8-16 GB RAM, start with one of these:

Step 1: Pull a Model

# Pull Llama 3.3 8B (recommended for beginners)
ollama pull llama3.3:8b

Expected output while downloading:

pulling manifest
pulling 970aa74c0a90... 100% ▕████████████████▏ 4.9 GB
pulling 0ba8f0e314b4... 100% ▕████████████████▏  12 KB
pulling 56bb8bd477a5... 100% ▕████████████████▏   96 B
pulling f02dd72bb242... 100% ▕████████████████▏  418 B
verifying sha256 digest
writing manifest
success

The download time depends on your connection speed. A 4.9 GB model takes roughly 3-4 minutes on a 200 Mbps connection.

Step 2: List Your Installed Models

ollama list

NAME                ID              SIZE      MODIFIED
llama3.3:8b         a6eb4748fd29    4.9 GB    2 minutes ago

Step 3: Run a Model

ollama run llama3.3:8b

Ollama loads the model and opens an interactive prompt:

>>> Send a message (/? for help)
>>> What is the capital of France?
The capital of France is Paris.

>>> /bye

Type `/bye` or press Ctrl+D to exit. Type `/?` for a list of available commands including `/set` to change context length or temperature.

Use the Ollama REST API

Ollama exposes a REST API on `http://localhost:11434` when the Ollama service is running. This API lets you integrate local LLM inference into your own scripts, applications, and tools.

Check the API is Running

curl http://localhost:11434
# Expected output: Ollama is running

Generate Text (POST /api/generate)

The `/api/generate` endpoint takes a model name and a prompt and returns the generated text.

curl http://localhost:11434/api/generate \
  -d '{
    "model": "llama3.3:8b",
    "prompt": "Write a Python function that checks if a number is prime",
    "stream": false
  }'

The response is a JSON object. The `"response"` field contains the generated text:

{
  "model": "llama3.3:8b",
  "created_at": "2026-03-16T10:00:00Z",
  "response": "def is_prime(n):
    if n < 2:
        return False
    for i in range(2, int(n**0.5) + 1):
        if n % i == 0:
            return False
    return True",
  "done": true,
  "total_duration": 3200000000,
  "eval_count": 87
}

Chat API (POST /api/chat)

The `/api/chat` endpoint supports multi-turn conversations with a `messages` array:

curl http://localhost:11434/api/chat \
  -d '{
    "model": "llama3.3:8b",
    "messages": [
      {"role": "user", "content": "What is Docker?"},
      {"role": "assistant", "content": "Docker is a containerisation platform."},
      {"role": "user", "content": "How does it differ from a VM?"}
    ],
    "stream": false
  }'

OpenAI-Compatible Endpoint

Ollama exposes an OpenAI-compatible API at `http://localhost:11434/v1`. Any application built for the OpenAI API can point to this endpoint instead, with no code changes other than the base URL and model name.

from openai import OpenAI

# Point the OpenAI client at your local Ollama instance
client = OpenAI(
    base_url='http://localhost:11434/v1',
    api_key='ollama'  # required by the client but not used by Ollama
)

response = client.chat.completions.create(
    model='llama3.3:8b',
    messages=[{'role': 'user', 'content': 'What is 2+2?'}]
)
print(response.choices[0].message.content)

Enable GPU Acceleration

CPU inference works but is slow for interactive use. A GPU reduces response latency from 20-60 seconds per response to 1-5 seconds for 7B-8B models. Ollama supports NVIDIA (CUDA), AMD (ROCm), and Apple Silicon (Metal).

Apple Silicon (M1/M2/M3/M4)

No setup needed. Ollama detects Apple Metal automatically when installed on macOS. All inference uses the GPU by default. Verify GPU usage with:

ollama run llama3.3:8b

Check the Ollama log for Metal confirmation:

# In a separate terminal while the model is running
tail -f ~/.ollama/logs/server.log | grep -i "metal|gpu"
# Expected: llm_load_print_meta: n_gpu_layers = 33

NVIDIA GPU (CUDA)

Ollama automatically detects NVIDIA GPUs when the CUDA toolkit is installed on the host. For Docker deployments, the NVIDIA Container Toolkit is required.

Install the NVIDIA Container Toolkit on Ubuntu:

# Add NVIDIA package repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then run the Ollama Docker container with GPU passthrough:

docker run -d \
  --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

Verify GPU is in use during inference:

# While a model is running
nvidia-smi
# The ollama process should appear with VRAM usage

AMD GPU (ROCm)

Ollama includes ROCm support for AMD GPUs. Use the ROCm-specific Docker image:

docker run -d \
  --device /dev/kfd \
  --device /dev/dri \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama:rocm

Checking How Many Layers Are on GPU

Ollama offloads model layers to GPU incrementally based on available VRAM. Run a model and check the log:

tail -f ~/.ollama/logs/server.log | grep "n_gpu_layers"
# Example output:
# llm_load_print_meta: n_gpu_layers = 33 (all layers on GPU)
# llm_load_print_meta: n_gpu_layers = 18 (partial offload, not enough VRAM)

If `n_gpu_layers` is 0, inference is on CPU. If it is less than the total layer count, only partial GPU offload occurred.

Key Configuration Options

Ollama reads configuration from environment variables. Set them before starting the service, either in your shell profile or as Docker `-e` flags.

Allow Remote Access

By default, Ollama binds to localhost only. To access the API from another machine (for example, from Open-WebUI running in Docker), expose the API on all interfaces:

# Set before starting Ollama (Linux systemd service)
sudo systemctl edit ollama

Add these lines in the editor:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Connect a Web Interface (Open-WebUI)

The Ollama CLI is useful for testing but most users prefer a browser-based chat interface. Open-WebUI provides a ChatGPT-style interface that connects directly to your local Ollama instance.

Quick Start (One Command)

If Ollama is running natively (not in Docker), this single command installs and starts Open-WebUI:

docker run -d \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

Open your browser at `http://localhost:3000`. Create an admin account on first visit. Open-WebUI auto-detects Ollama at `http://host.docker.internal:11434`.

For a full Docker Compose setup of Open-WebUI with Ollama, see the complete guide: How to Set Up Open-WebUI with Ollama.

Troubleshooting

Frequently Asked Questions

# Check status
systemctl status ollama

# Start the service
sudo systemctl start ollama

# Stop the service
sudo systemctl stop ollama

# Restart after configuration changes
sudo systemctl restart ollama

pip install ollama

import ollama

response = ollama.chat(
    model='llama3.3:8b',
    messages=[{'role': 'user', 'content': 'What is the capital of Germany?'}]
)
print(response['message']['content'])

npm install ollama

OLLAMA_MAX_LOADED_MODELS=2 ollama serve