Ollama in Docker: budget resources and keep the endpoint private

Running Ollama in Docker packages the server; it does not provide free hardware, automatic GPU access, or a complete security boundary. For a low-cost local experiment, start with a private endpoint, persistent model storage, one representative workload, and measured memory use. Then decide whether the deployment deserves a larger resource allocation.

This guide concentrates on the operational choices that affect cost and exposure. It is not a promise that any model will fit any machine. Sources checked September 5, 2026. The electricity figures below are synthetic examples; substitute measurements and your actual tariff before making a spending decision.

A loopback-only starting configuration

The official Docker guide uses the ollama/ollama image and mounts model data at /root/.ollama. Its CPU example publishes port 11434. For an experiment accessed from the Docker host, our adaptation narrows that published address to loopback and uses a clearly named volume and container:

docker run -d --name mbc-ollama   -p 127.0.0.1:11434:11434   -v mbc-ollama-models:/root/.ollama   ollama/ollama

This command creates resources and may download the image; review it before running. It is a CPU starting example, not a GPU configuration. A pre-existing process on the same host port will conflict. Choose a different explicit host port if needed rather than stopping an unrelated service. For a reproducible deployment, record the tested image digest and update it through a deliberate maintenance process.

Docker's port-publishing documentation warns that publishing without a host address exposes the port beyond the host by default. It also notes a localhost exposure caveat in releases older than 28.0.0. Use a current supported installation and verify the listening address. Loopback narrows network access; it does not authenticate other processes on the same machine.

Choose a deployment boundary before adding clients

Requirement	Starting choice	What still needs checking
One desktop client	Loopback published port	Host processes and browser/client permissions
Another container	A scoped private container network	Only intended peers can reach the service
Remote team access	Authenticated access layer and encrypted transport	Identity, quotas, logs, and operational ownership
Public anonymous service	Not this minimal local setup	Abuse controls and a separate production design

Do not mount the Docker socket, your entire home directory, or a production credentials directory merely to make troubleshooting convenient. An inference server does not need general control over the host's containers. Start with the specific model volume and only add a device or directory when the selected deployment requires it and you understand its access implications.

A coding agent that talks to this server has its own permissions. Keeping the model in a container does not automatically contain shell commands executed by a separate agent running on the host. Draw that distinction in your runbook so a future operator does not mistake model isolation for whole-workflow isolation.

Persistence prevents repeated downloads, not data loss by magic

A named volume can preserve model data when the container is replaced. Docker's volume documentation distinguishes volume lifetime from container lifetime. Record which named volume belongs to this deployment before upgrades or cleanup. Removing the container and deleting its model volume are different operations with different recovery costs.

Keep an inventory of model names, image version, intended mount, and any configuration required to reproduce the setup. A volume is not a backup strategy on its own: disk failure or an explicit volume deletion can still remove it. Decide what is worth backing up and what can be downloaded again. Never run a broad prune command as a first response to a slow model.

Watch free disk space during a trial. A model download, an additional quantization, logs, and a replacement image can temporarily coexist. The useful storage budget is not just the size of the one model currently loaded. Maintain enough headroom to update safely and remove only identified obsolete assets after checking what still uses them.

GPU support is a separate prerequisite

Ollama's Docker guide has distinct paths for NVIDIA and AMD devices; the NVIDIA path requires the NVIDIA Container Toolkit. Do not infer GPU acceleration from the presence of the image alone. Verify the supported host/runtime combination and inspect the running model before comparing performance. A CPU-only run is valid, but it should be labelled CPU-only in your results.

Start a single representative request and observe load, response time, memory, and whether the result is acceptable. Record these before changing context length or concurrency. If two things change together, a slower run will not tell you which setting caused the regression. Keep a small repeatable prompt set so a version update can be compared against the same baseline.

Memory limits protect the host but cannot make a model smaller

Docker documents that containers have no resource constraints by default. Its resource guide also explains the risk of out-of-memory process termination. Establish resource requirements before setting explicit CPU and memory limits. A limit below the workload's actual requirement may simply cause failures, while no limit can allow the experiment to compete with important host applications.

Leave room for the operating system and your normal work. Evaluate a bounded context and one request before increasing parallelism. If a test fails under pressure, save the error category and observed memory state; stop the test before repeating it. Avoid disabling out-of-memory safeguards as a shortcut to declaring a model usable.

A transparent monthly electricity example

Assume, synthetically, that inference adds an average 80 watts above an otherwise-running machine's idle draw and runs four hours on twenty days. Incremental energy is 80 ÷ 1,000 × 4 × 20 = 6.4 kWh. At an illustrative $0.25 per kWh, that portion costs $1.60. Neither the wattage nor the tariff is a claim about your hardware or country.

If the machine stays on only for this service, incremental inference power understates the total: include the machine's baseline powered-on consumption as well. Equipment purchase or rental, maintenance time, storage, and failed runs belong in separate lines. This helps distinguish marginal operating cost from the cost of establishing the system in the first place.

Compare an accepted workload, not just energy against an API price. A local setup that misses deadlines may require a hosted fallback, and a cloud setup may avoid a hardware purchase. The Ollama Cloud calculator models hosted plan cash outlay, not Docker electricity or depreciation. Keep those categories separate until you assemble the final decision sheet.

Frequently asked questions

Will deleting a container remove every downloaded model?

Not necessarily when the models are stored in a separate named volume. Inspect the mount and volume name before acting. Do not rely on a remembered command from another deployment, and do not remove a volume until you know whether its contents are replaceable.

Can I expose port 11434 directly to the Internet?

This guide does not recommend that. A public endpoint needs a separate design for authentication, transport protection, request limits, and abuse handling. A working local curl request is not proof that such a public service is safe or economical.

Does choosing a cloud model still use Docker?

The client service may remain in the container while inference runs remotely. Review the selected model and account route, not just the container's location. Hosted usage belongs to its cloud billing ledger, even if the first network hop is local.

What should I record before increasing capacity?

Save the tested model, runtime version, effective context, concurrency, peak memory observation, elapsed task time, and acceptance result. For context-specific planning, continue with the context window explanation. Capacity changes should follow measured need, not a desire for a larger number.