Set Up Local LLMs With Ollama and Open WebUI

Running large language models locally has never been easier, and the combination of Ollama and Open WebUI delivers a production-quality setup in under 30 minutes. This guide walks you through every step — from installation to optimization — so you can ditch expensive API calls and keep your data entirely on your own machine.

Unlike cloud-based services like OpenAI's ChatGPT ($20/month for Plus) or Anthropic's Claude Pro ($20/month), a local LLM stack costs nothing beyond your existing hardware. With models like Llama 3.1, Mistral, and Gemma 2 now rivaling proprietary alternatives on many benchmarks, the case for local development has never been stronger.

Key Takeaways at a Glance

• Ollama simplifies model management to single-line terminal commands
• Open WebUI provides a ChatGPT-style browser interface for local models
• You can run capable 7B–13B parameter models on a machine with 16 GB of RAM
• The entire stack is 100% free and open source
• Your data never leaves your machine — ideal for sensitive or proprietary work
• Setup takes approximately 20–30 minutes on most systems

Why Run LLMs Locally in 2024

Privacy is the single biggest driver pushing developers toward local LLM setups. When you send prompts to OpenAI or Google, your data traverses third-party servers and may be used for training unless you explicitly opt out. A local stack eliminates that concern entirely.

Cost is the second factor. API pricing from leading providers adds up fast — OpenAI charges $5 per million input tokens for GPT-4o, and heavy development workflows can burn through hundreds of dollars monthly. Local inference costs only electricity.

There is also the matter of reliability and control. Cloud APIs experience outages, rate limits, and breaking changes. A local environment gives you deterministic, reproducible results without dependency on external services.

Finally, local setups enable offline development. Whether you are on a flight, in a restricted network environment, or simply prefer working without an internet connection, Ollama keeps running regardless.

Prerequisites and System Requirements

Before diving into installation, make sure your hardware meets the minimum requirements. The quality of your experience depends heavily on available RAM and, optionally, GPU acceleration.

Minimum Hardware

• RAM: 8 GB minimum (16 GB recommended for 7B models, 32 GB for 13B models)
• Storage: At least 10 GB free disk space per model
• CPU: Any modern x86_64 or Apple Silicon processor
• GPU (optional): NVIDIA GPU with 6+ GB VRAM for accelerated inference, or Apple M1/M2/M3 with unified memory

Supported Operating Systems

Ollama runs natively on macOS, Linux, and Windows. Open WebUI runs anywhere Docker is available, which covers all 3 platforms. Make sure you have Docker Desktop installed — you will need it for Open WebUI.

For the smoothest experience, Apple Silicon Macs (M1 or later) are arguably the best consumer hardware for local LLM inference, thanks to their unified memory architecture and Metal GPU acceleration.

Step 1: Install Ollama in Minutes

Ollama is a lightweight runtime that handles model downloading, quantization, and inference through a simple CLI. Installation varies by platform but is straightforward on all of them.

macOS and Linux

Open your terminal and run a single command:

curl -fsSL https://ollama.com/install.sh | sh

On macOS, you can alternatively download the desktop app from ollama.com and drag it into your Applications folder. The desktop app includes the CLI automatically.

Windows

Download the installer from ollama.com/download and run the .exe file. After installation, Ollama runs as a background service and is accessible from PowerShell or Command Prompt.

To verify your installation, run:

ollama --version

You should see a version number like 0.3.x or higher. Ollama automatically starts a local API server on port 11434, which Open WebUI will connect to later.

Step 2: Pull Your First Model

With Ollama installed, downloading a model requires just one command. The Ollama model library hosts dozens of open-source models in various sizes and quantization levels.

To download Meta's Llama 3.1 8B — one of the best open-source models available — run:

ollama pull llama3.1

This downloads approximately 4.7 GB of data. Once complete, you can test it immediately:

ollama run llama3.1

This launches an interactive chat session right in your terminal. Type any prompt and watch the model respond in real time.

Recommended Models to Try

• llama3.1 (8B) — Best general-purpose open model, strong reasoning and instruction following
• mistral (7B) — Excellent performance-to-size ratio, fast inference
• codellama (7B/13B) — Optimized for code generation and programming tasks
• gemma2 (9B) — Google's open model with strong multilingual capabilities
• phi3 (3.8B) — Microsoft's compact model, surprisingly capable for its size
• DeepSeek-coder-v2 (16B) — Top-tier coding model rivaling GPT-4 on code benchmarks

You can have multiple models installed simultaneously. Switch between them by simply running ollama run <model-name>. To see all installed models, use ollama list.

Step 3: Deploy Open WebUI for a Browser-Based Interface

Open WebUI (formerly known as Ollama WebUI) transforms your local Ollama setup into a polished, ChatGPT-like experience accessible from any browser. It supports conversation history, multiple chat sessions, model switching, document uploads, and even multi-modal interactions.

The fastest way to deploy it is with Docker. Run this single command:

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

This command does several things at once. It maps port 3000 on your host to the container's internal port 8080, creates a persistent volume for your data, and ensures the container restarts automatically on reboot.

Once the container is running, open your browser and navigate to http://localhost:3000. You will be prompted to create an admin account on first launch — this account is stored entirely locally.

After logging in, Open WebUI automatically detects your Ollama instance and displays all available models in a dropdown menu. Select a model, type a prompt, and you are running a fully private AI assistant.

Step 4: Configure and Optimize Your Setup

The default configuration works well, but a few tweaks can dramatically improve your experience.

Adjusting Context Window

By default, Ollama sets a 2048-token context window. For longer conversations or document analysis, increase it by creating a custom Modelfile:

FROM llama3.1
PARAMETER num_ctx 8192

Save this as 'Modelfile' and run ollama create llama3.1-long -f Modelfile. This creates a new model variant with an 8192-token context window.

GPU Acceleration

Ollama automatically detects and uses NVIDIA GPUs via CUDA and Apple Silicon GPUs via Metal. No manual configuration is needed in most cases. To verify GPU usage, check ollama ps while a model is running — it shows whether layers are loaded on GPU or CPU.

For NVIDIA users, ensure you have the latest drivers and the NVIDIA Container Toolkit installed if you plan to run Ollama inside Docker.

Open WebUI Power Features

Open WebUI includes several advanced features worth exploring:

• RAG (Retrieval-Augmented Generation): Upload PDFs and documents directly into chats for context-aware responses
• System Prompts: Set persistent instructions per model to customize behavior
• API Keys: Generate API keys to integrate your local stack into external applications
• Multi-user Support: Add team members with role-based access control
• Model Presets: Save temperature, top-p, and other parameters as reusable presets

How This Compares to Cloud Alternatives

A local Ollama + Open WebUI stack does not replace GPT-4o or Claude 3.5 Sonnet for every use case. Those frontier models still lead on complex reasoning, nuanced writing, and multimodal tasks.

However, for code generation, data extraction, summarization, and routine Q&A, models like Llama 3.1 8B perform remarkably well — often within 90% of GPT-4o quality at zero marginal cost. For teams processing sensitive data (legal, medical, financial), the privacy advantage alone justifies the setup.

The performance gap continues to narrow. Meta's Llama 3.1 405B model, when run on appropriate hardware, matches GPT-4 on several major benchmarks. Even the 8B variant outperforms GPT-3.5 Turbo on most tasks.

Looking Ahead: The Local AI Stack Is Maturing Fast

The local LLM ecosystem is evolving at breakneck speed. Ollama now supports tool calling, structured outputs, and vision models — features that were exclusive to cloud APIs just months ago.

Open WebUI ships updates weekly, adding capabilities like web search integration, image generation through Stable Diffusion backends, and function calling pipelines. The project has surpassed 35,000 GitHub stars and has become the de facto frontend for local LLM development.

Looking further out, upcoming hardware from NVIDIA (RTX 50-series with increased VRAM), Apple (M4 Ultra with up to 192 GB unified memory), and AMD is set to make local inference of even 70B+ parameter models practical on consumer desktops. Combined with increasingly efficient quantization techniques like GGUF and AWQ, the barrier to running state-of-the-art models locally drops with every quarter.

For developers and teams ready to take control of their AI stack, the Ollama and Open WebUI combination represents the most accessible entry point available today. The setup is free, the models are powerful, and your data stays exactly where it belongs — on your own hardware.