Local AI for Coding: Run Models on Your Own Hardware

Why run coding models locally

Running a model on your own hardware turns AI from a metered service into a tool you own. The benefits compound the more you use it:

• Your code stays private. Nothing is sent to a third party, which is often the deciding factor for companies with proprietary code or compliance requirements.
• No per-token bills. Agents can make hundreds of model calls per task. Locally, retries and experiments are free.
• It works offline, on a plane, on a train or behind a strict corporate firewall.
• Your setup stays stable. You pin the exact model version. No silent updates, deprecations or shutdowns.
• You can customize it. Fine-tune on your codebase, adjust sampling settings or run several models side by side.

Hardware: what actually matters

Memory comes first

The single most important number is how much memory your accelerator can use. The whole model, plus its context cache, should fit in fast memory:

• NVIDIA GPUs: VRAM is what counts. A 24 GB card, such as the RTX 3090 or 4090, is the sweet spot for hobbyists and runs 20B to 34B models at 4-bit comfortably.
• Apple Silicon Macs: unified memory is shared between the CPU and GPU, so a 32 GB or 64 GB Mac can run surprisingly large models. Leave some memory free for macOS itself.
• CPU-only: works for small models (1B to 7B) and is fine for experimenting, but expect slow generation for anything larger.

Then memory bandwidth

Token generation speed is mostly limited by how fast weights can be read from memory. That's why high-bandwidth GPUs and Apple's Max and Ultra chips generate much faster than CPUs with the same capacity.

MoE models are a special case

Mixture-of-experts models such as Qwen3-Coder and DeepSeek-Coder-V2 compute only a fraction of their weights per token, so they generate quickly if you can fit them. Their total size still sets the memory requirement.

Runtimes: how to serve a model

Runtime	Best for	Notes
Ollama	Getting started, laptops, Macs	One-command install, model library, OpenAI-compatible API
llama.cpp	Maximum control, CPUs, edge devices	The engine many tools build on, using GGUF quantized models
LM Studio	A desktop GUI	Browse, download and chat with models, and serve a local API
vLLM	GPU servers, teams	High-throughput serving with batching, ideal for sharing one endpoint
SGLang	GPU servers, agent workloads	Fast serving with strong support for structured and multi-turn workloads

Almost all of them expose an OpenAI-compatible API, so editors and agents that work with hosted APIs can usually be pointed at your local server by changing a base URL.

Quantization in practice

Quantization trades a little quality for a lot of memory. For coding:

• 8-bit: near-identical to full precision, at about half the memory
• 5-bit and 4-bit: the common sweet spot. Ollama's default 4-bit variants are a good starting point
• 3-bit and below: use only when nothing else fits, and test carefully, since code correctness tends to suffer first

If you have spare memory, moving from 4-bit to 5-bit or 8-bit is often a cheaper quality boost than moving to a bigger model.

Performance tips

1. Fit the model entirely in GPU memory. Partial offloading to system RAM can slow generation dramatically.
2. Right-size the context. Don't allocate a 128K context if your tasks use 16K, since the extra cache eats memory.
3. Use two models. A tiny model for autocomplete, where latency matters, and a larger one for chat and agents.
4. Keep models loaded. Reloading a large model can take a while. Most runtimes let you set how long a model stays in memory.
5. Batch on servers. If several people share one GPU, use vLLM or SGLang rather than a desktop runtime.

Security and privacy checklist

• Bind local servers to localhost unless you deliberately want network access, and put authentication in front of anything exposed beyond your machine.
• Download models from official sources, such as the publisher's Hugging Face organization or the Ollama library.
• Remember that agents can run commands. Use a branch or worktree, review diffs and sandbox agents that run without supervision.
• Local doesn't mean unlogged. Check what your editor extension or agent stores and where.

Start here

New to local AI? Follow How to Run an Open-Source Coding Model Locally with Ollama, then browse the model directory to find a model that fits your hardware. Posts tagged #local-ai are listed above.

Three example setups

The laptop setup (16 GB). A small coding model of about 1.5B parameters handles autocomplete and a 7B model handles chat, both served by Ollama. It's enough for everyday completion, explanations and small refactors, and it works offline on a plane.

The workstation setup (24 GB GPU). A 14B to 24B model at 4-bit handles chat and agent tasks, and a small model stays loaded for autocomplete. This is the sweet spot for running an open-source agent on real tasks at a comfortable speed.

The team server (48 GB+ or multiple GPUs). vLLM serves a 32B dense model or a mixture-of-experts model to the whole team over an OpenAI-compatible API, with authentication in front. Developers point their editors and agents at the shared endpoint, and their code stays inside the company network.

All three setups use the same tools and the same API, so you can start on a laptop and move up without changing your workflow.

Frequently asked questions

Is local AI really free?

The software and models are free to download, and there are no per-token charges. You pay for hardware and electricity. If you already own a recent Mac or a gaming GPU, the extra cost of running coding models is small compared with a heavy-usage API bill.

Can a laptop run a useful coding model?

Yes. A modern laptop with 16 GB of RAM runs 7B models well, which is plenty for autocomplete and everyday chat. A 32 GB or larger Apple Silicon laptop can run 20B to 30B models, including agent-capable ones like Devstral.

How does local quality compare with hosted frontier models?

The best hosted models still lead on the hardest problems. For routine work, such as autocomplete, explanations, refactors, tests and well-scoped bug fixes, modern open models of 14B and up are often good enough. Many developers use local models by default and reach for a hosted model only when needed.

Can my whole team share one local model?

Yes. Run vLLM or SGLang on a GPU server inside your network and give everyone the endpoint URL. That keeps code inside the company while sharing the hardware cost.

What's the difference between Ollama and llama.cpp?

llama.cpp is the low-level inference engine. Ollama builds on that kind of engine and adds a model library, simple commands and a background server. Start with Ollama for convenience, and use llama.cpp directly when you need fine-grained control or unusual hardware.