Deploy Private LLMs on Your Own GPU Server: A Complete vLLM & Docker Guide

In today’s AI‑driven world, large language models (LLMs) have moved from research labs to production environments. While cloud‑based APIs are convenient, many organizations prefer to keep their models private, control costs, and guarantee low‑latency inference. Deploying a private LLM on a dedicated GPU server—using the high‑performance vLLM inference engine and containerized with Docker—offers the perfect blend of speed, scalability, and security.

Why Use vLLM and Docker for Private LLM Deployments?

• vLLM is built on top of PyTorch and leverages modern GPU scheduling, KV‑cache management, and tensor parallelism to deliver near‑optimal throughput for models ranging from 7 B to 70 B parameters.
• Docker isolates the runtime environment, guarantees reproducibility across development, staging, and production, and simplifies dependency management (CUDA, cuDNN, driver versions, Python packages).
• Running on a dedicated GPU server eliminates noisy‑neighbor effects, ensures consistent latency, and gives you full control over security policies and network topology.

Prerequisites: Hardware, Software, and Access

Hardware Requirements

1. GPU: At least one NVIDIA GPU with ≥ 16 GB VRAM (e.g., A100, RTX 4090). For models > 13 B parameters, consider multi‑GPU setups with NVLink or PCIe‑based scaling.
2. CPU & Memory: 8+ cores and 64 GB RAM to handle preprocessing, tokenization, and request queuing.
3. Storage: SSD with ≥ 200 GB free space for model checkpoints, Docker images, and logs.

Software Stack

• Ubuntu 22.04 LTS (or a compatible Linux distribution)
• Docker Engine 24.x or newer
• NVIDIA Container Toolkit (for GPU passthrough)
• CUDA 12.x & cuDNN 8.x (matching the driver version)
• Git, curl, and basic build tools (make, gcc)

Access & Permissions

Make sure you have sudo privileges to install system packages and to add your user to the docker group. You’ll also need a model checkpoint (e.g., LLaMA‑2‑13B, Mistral‑7B) that you are licensed to host.

Step‑by‑Step Setup: From Host Preparation to Running vLLM in Docker

1. Install NVIDIA Drivers and the Container Toolkit

# Update the OS and install dependencies
sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential dkms curl

# Add the NVIDIA driver repository
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

# Install driver (example for 525 series) and toolkit
sudo apt update
sudo apt install -y nvidia-driver-525 nvidia-docker2

# Restart Docker daemon
sudo systemctl restart docker

# Verify GPU visibility inside containers
docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi

2. Clone the vLLM Repository and Prepare a Dockerfile

While vLLM provides an official Docker image, creating a custom Dockerfile gives you full control over the model checkpoint location and any additional libraries (e.g., FastAPI, LangChain).

# Clone vLLM (optional – you can also use the published image)
git clone https://github.com/vllm-project/vllm.git
cd vllm

# Directory layout
# ├─ vllm/
# ├─ models/
# │   └─ llama2-13b/   (your checkpoint)
# └─ Dockerfile

Sample Dockerfile

FROM nvidia/cuda:12.2.0-runtime-ubuntu22.04

# Install Python & system deps
RUN apt-get update && apt-get install -y \
    python3.11 python3.11-venv python3-pip git curl && \
    rm -rf /var/lib/apt/lists/*

# Create a non‑root user
ARG USERNAME=appuser
ARG UID=1000
ARG GID=1000
RUN groupadd -g $GID $USERNAME && \
    useradd -m -u $UID -g $GID -s /bin/bash $USERNAME

# Switch to user
USER $USERNAME
WORKDIR /home/$USERNAME

# Install vLLM and optional server framework
RUN python3 -m venv venv && \
    . venv/bin/activate && \
    pip install --upgrade pip && \
    pip install vllm[all] fastapi uvicorn

# Copy model checkpoint (or mount at runtime)
COPY --chown=$USERNAME:$USERNAME models/ /home/$USERNAME/models/

# Expose the vLLM HTTP port
EXPOSE 8000

# Entry point – launch the vLLM server
CMD ["/bin/bash", "-c", "\
    . venv/bin/activate && \
    python -m vllm.entrypoints.openai.api_server \
      --model /home/appuser/models/llama2-13b \
      --port 8000 \
      --trust-remote-code \
    "]

3. Build and Tag the Docker Image

# From the repository root
docker build -t private-llm:vllm-13b .

During the build, Docker will automatically download the required PyTorch and CUDA wheels that match the base image’s CUDA version.

4. Run the Container with GPU Access and Persistent Storage

Mount the model directory as a volume (useful for swapping checkpoints without rebuilding) and expose the API port.

docker run -d \
  --name vllm_server \
  --gpus all \
  -p 8000:8000 \
  -v /srv/models/llama2-13b:/home/appuser/models/llama2-13b:ro \
  private-llm:vllm-13b

Verify that the server is up:

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "llama2-13b",
        "messages": [{"role":"user","content":"Explain quantum entanglement in simple terms."}]
      }' | jq .

5. Optional: Add a Reverse Proxy & TLS

For production use, place an Nginx or Traefik reverse proxy in front of the container. This gives you:

• HTTPS termination (Let’s Encrypt or internal PKI)
• Rate limiting and authentication (API keys, OAuth2)
• Load balancing if you later scale to multiple GPU nodes

Example Nginx snippet:

server {
    listen 443 ssl;
    server_name llm.mycompany.com;

    ssl_certificate /etc/letsencrypt/live/llm.mycompany.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/llm.mycompany.com/privkey.pem;

    location / {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Performance Tuning & Monitoring

GPU Utilization

vLLM automatically batches incoming requests to keep the GPU busy. However, you can fine‑tune:

• --max-num-batched-token – maximum tokens per batch (default 512).
• --max-model-len – adjust to the longest prompt you expect.
• Enable --enable-chunked-prefill for very long inputs.

Use nvidia-smi or the DCGM exporter to collect metrics and feed them into Prometheus/Grafana dashboards.

Memory Management

For multi‑GPU setups, launch vLLM with --tensor-parallel-size N where N equals the number of GPUs. The engine will shard the model across devices, reducing per‑GPU memory pressure.

Logging & Observability

• Redirect container logs to a central ELK stack: docker logs -f vllm_server > /var/log/vllm.log
• Instrument the FastAPI layer with OpenTelemetry for request tracing.
• Set VLLM_LOGGING_LEVEL=INFO (or DEBUG) via an environment variable for deeper insights.

Scaling Out: From a Single Server to a Cluster

When demand grows, you can extend the architecture without rewriting code:

1. Horizontal scaling: Deploy identical Docker containers on additional GPU nodes and place them behind a load‑balancing reverse proxy.
2. Model sharding: Use vLLM’s built‑in tensor parallelism across multiple GPUs in a single node for > 30 B‑parameter models.
3. Orchestration: Wrap the containers in Kubernetes with the nvidia.com/gpu resource request, allowing auto‑scaling based on queue length.

Because the API surface is OpenAI‑compatible, client code does not need to change when you move from a single‑GPU Docker container to a full‑scale inference fleet.

Conclusion

Deploying a private LLM on a dedicated GPU server using vLLM and Docker combines the best of performance, security, and operational simplicity. By following the steps outlined—preparing the host, building a reproducible container, exposing a robust OpenAI‑compatible endpoint, and adding production‑grade monitoring—you can deliver low‑latency, high‑throughput language services that stay under your organization’s control.

Whether you’re serving a 7 B chatbot for internal help desks or a 70 B research assistant for data‑intensive pipelines, the same workflow scales gracefully. Keep the Docker image lean, monitor GPU health, and leverage vLLM’s dynamic batching to squeeze every ounce of performance out of your hardware. Your private LLM is now ready to power the next generation of AI‑enabled products—securely, efficiently, and at scale.