How to Set Up AnythingLLM with Ollama (2026 Guide)

Install AnythingLLM

AnythingLLM offers two installation paths. The Desktop App is the fastest option for personal use on a single machine. Docker is the right choice for server deployments, always-on access, or when you want multiple users sharing the same instance.

Desktop App (Recommended for Single Users)

The Desktop App bundles everything into a standalone installer. No Docker or Node.js required.

Download the installer for your operating system from the official docs:

After installation, launch the app. It opens a browser window at `http://localhost:3001` automatically. The Desktop App includes its own embedded vector database and storage, so no external configuration is required for the app itself.

Docker (Recommended for Servers and Multi-User Access)

Create a directory for persistent storage, then run the official image:

# Create a storage directory on the host
mkdir -p ~/anythingllm/storage

# Run AnythingLLM
docker run -d \
  -p 3001:3001 \
  --cap-add SYS_ADMIN \
  -v ~/anythingllm/storage:/app/server/storage \
  -e STORAGE_DIR=/app/server/storage \
  --name anythingllm \
  --restart unless-stopped \
  mintplexlabs/anythingllm

The image is approximately 2 GB. Wait for the download and startup to complete, then verify:

docker ps
# Expected output:
# CONTAINER ID   IMAGE                        STATUS         PORTS
# a1b2c3d4e5f6   mintplexlabs/anythingllm    Up 2 minutes   0.0.0.0:3001->3001/tcp

Open `http://localhost:3001` in your browser.

Docker Compose (Optional)

If you prefer to manage everything in one file, create a `docker-compose.yml`:

version: '3.8'

volumes:
  anythingllm_storage:

services:
  anythingllm:
    image: mintplexlabs/anythingllm
    restart: unless-stopped
    ports:
      - "3001:3001"
    cap_add:
      - SYS_ADMIN
    volumes:
      - anythingllm_storage:/app/server/storage
    environment:
      - STORAGE_DIR=/app/server/storage

Start it with:

docker compose up -d

Pull Required Ollama Models

AnythingLLM needs two types of models from Ollama: a chat model that generates answers, and an embedding model that converts your documents into vectors for search.

Chat Models

Pull one of these depending on your hardware. For a deeper comparison of model quality and hardware requirements, see the best local LLM models guide.

Embedding Model (Required for Document Q&A)

The embedding model converts your uploaded documents into vectors that AnythingLLM searches at query time. Without it, document uploads fail silently.

# Pull the embedding model
ollama pull nomic-embed-text

# Expected output:
# pulling manifest
# pulling 970aa74c0a90... 100% 274 MB
# verifying sha256 digest
# success

`nomic-embed-text` is 274 MB and runs on CPU without GPU memory. It is the standard embedding model for Ollama-based setups and works well for English and multilingual documents.

Verify Both Models Are Available

ollama list
# Expected output (your versions may differ):
# NAME                    ID              SIZE    MODIFIED
# llama3.3:8b             a6eb4748fd29    4.9 GB  2 minutes ago
# nomic-embed-text:latest 0a109f422b47    274 MB  1 minute ago

Both models must appear in this list before you configure AnythingLLM.

Connect AnythingLLM to Ollama

The Ollama connection URL is the most common source of setup failures. The correct URL depends on how AnythingLLM is running.

Which URL to Use

Setup Wizard Steps

The first time you open `http://localhost:3001`, a setup wizard walks you through the configuration:

1. Create an admin account (username and password) 2. On the LLM configuration screen, select "Ollama" from the provider list 3. Enter the Ollama base URL for your deployment (see table above) 4. Select your chat model from the dropdown (it fetches the list from Ollama automatically) 5. On the embedding configuration screen, select "Ollama" again 6. Select `nomic-embed-text:latest` as the embedding model 7. Complete the wizard

Verify the Connection After Setup

In the AnythingLLM interface, go to Settings (gear icon) > LLM Provider. The status indicator next to the Ollama URL should show green. If it shows red, the URL is wrong or Ollama is not running.

Test Ollama directly to confirm it is reachable:

# If using Desktop App or checking from the host machine
curl http://localhost:11434
# Expected: Ollama is running

# If using Docker on Linux, test the bridge IP
curl http://172.17.0.1:11434
# Expected: Ollama is running

If Ollama is not running, start it:

# Linux (systemd service)
sudo systemctl start ollama

# macOS or manual start
ollama serve

Create a Workspace and Upload Documents

Workspaces are the core concept in AnythingLLM. Each workspace is a separate document container with its own vector store, chat history, and model settings. Documents uploaded to one workspace are not visible to other workspaces. If you came from Open-WebUI, think of workspaces as separate chat sessions that each have their own private document library.

Create Your First Workspace

1. Click the "+" button or "New Workspace" in the left sidebar 2. Enter a name for the workspace (for example, "Research Notes" or "Company Docs") 3. Click Create

Upload Documents

Click the paperclip icon in the chat input area, or use the Document Manager (the folder icon in the sidebar). Supported file types include:

• PDF, DOCX, TXT, Markdown, HTML
• CSV, XLSX, PPTX
• JSON files and most code file types (.py, .js, .ts, .go, etc.)
• GitHub repository URLs (imports the full repo content)
• YouTube video URLs (imports the transcript)

After selecting files, AnythingLLM processes each document and shows a progress indicator. Processing time depends on file size. A 100-page PDF typically takes 10-30 seconds on a modern CPU.

Processing: research-paper.pdf
  Chunking document...
  Embedding 47 chunks with nomic-embed-text...
  Done. 47 vectors stored.

Chat Mode vs Query Mode

Each workspace has two response modes, controlled by the toggle in the workspace settings:

Switch modes via the settings icon inside the workspace. For research and document Q&A, Query mode gives more focused answers and prevents the model from hallucinating information not in your files.

Test the Setup

After uploading a document, ask a question about its content in the chat input. A working RAG response includes a "Sources" section below the answer showing which document chunks were retrieved.

Enable Agents and MCP Tools

AnythingLLM includes an agent system that lets the model take actions beyond answering questions. Agents can browse the web, run code, call external APIs, and use MCP (Model Context Protocol) servers.

Activate an Agent

Type `@agent` at the start of any chat message to activate agent mode in that conversation:

@agent Search the web for the latest AI papers from May 2026 and summarize the top 3.

The agent works through the task step by step, showing each action in the chat window. Default built-in agent capabilities include:

• Web browsing and URL fetching
• Basic calculation and data analysis
• Reading documents already in the workspace

Enable or Disable Agent Capabilities

Go to Settings > Agents to see which capabilities are active. You can toggle individual capabilities on or off. For example, disable web browsing if you want agents restricted to local documents only.

Add MCP Servers

AnythingLLM has built-in MCP support as of 2026. To add an MCP server:

1. Go to Settings > Agent Tools > MCP Servers 2. Click "Add MCP Server" 3. Paste the server configuration in JSON format

Example: adding a local filesystem MCP server:

{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-filesystem",
        "/Users/yourname/Documents"
      ]
    }
  }
}

4. Click Save and restart AnythingLLM if prompted

After adding a server, the agent can use its tools automatically when relevant. You can also call a specific tool by naming it in your `@agent` message. For programmatic access to Ollama models outside the UI, see the Ollama Python guide.

Update AnythingLLM

For Docker deployments, pull the latest image and restart:

docker pull mintplexlabs/anythingllm
docker stop anythingllm
docker rm anythingllm

# Re-run with the same flags as the original install
docker run -d \
  -p 3001:3001 \
  --cap-add SYS_ADMIN \
  -v ~/anythingllm/storage:/app/server/storage \
  -e STORAGE_DIR=/app/server/storage \
  --name anythingllm \
  --restart unless-stopped \
  mintplexlabs/anythingllm

Your documents, chat history, and settings are preserved in the `~/anythingllm/storage` directory. The Docker container itself is stateless.

For the Desktop App, check for updates from the Help menu inside the app.

Troubleshooting

Frequently Asked Questions

docker pull mintplexlabs/anythingllm
docker stop anythingllm && docker rm anythingllm
# Re-run the original docker run command