Why run AI locally?
Cloud AI providers charge recurring monthly fees, enforce strict rate limits, and transmit your queries, source code, and customer records across external networks.
For privacy-sensitive business workflows, pharmacy inventories, offline homelabs, or cost-conscious students, Local AI changes the equation. By running open-source model weights directly on your computer's RAM and GPU, you get:
- Zero Recurring API Bills: Pay once for hardware; run trillions of tokens for free.
- Total Privacy & Offline Operation: No internet connection required. Zero data leaves your machine.
- Low Latency & High Control: Stream tokens locally without cloud server queue delays.
Step 1: Install Ollama (Single Command)
On Linux or macOS, run:
curl -fsSL https://ollama.com/install.sh | sh
On Windows, download the official installer directly from ollama.com.
Once installed, verify the service is running:
$ ollama --version
ollama version 0.4.x
Step 2: Pick the right model for your RAM
Before pulling huge models, calculate your system memory constraints:
- 16 GB RAM (Standard Laptop):
llama3.2:3b(~2.0 GB VRAM/RAM): Lightning fast, great for formatting, summaries, and basic scripts.qwen2.5:7b(~4.7 GB VRAM/RAM): Exceptional coding, reasoning, and multi-language parsing.
- 32 GB+ RAM (Homelab / Workstation):
qwen2.5-coder:14b(~9.0 GB VRAM): High-tier autonomous code generation.
Run your first model instantly:
ollama run llama3.2:3b
You will see an interactive prompt appear directly in your shell. Type any question, code request, or text to translate.
Step 3: Connect via Python & REST API
Ollama exposes a local HTTP server at http://localhost:11434. You can query it from Python in just four lines:
import requests
response = requests.post(
"http://localhost:11434/api/generate",
json={"model": "llama3.2:3b", "prompt": "Explain DNS in one sentence.", "stream": False}
)
print(response.json()["response"])
Explore curated open-source tools
Looking for more local-first, self-hosted developer tools and automated scrapers? Browse the Open-Source Arsenal & Directory on THE SABBiR OS to discover verified local tools with 1-click install commands.