vLLM vs. TGI vs. Ollama: Stress-Testing 500 Concurrent Requests on Dual RTX 4090s

When deploying local LLMs inside an organization, developer teams often start with Ollama because of its effortless single-command setup. But what happens when 50 engineers start querying the same internal coding assistant simultaneously?

In our lab, we benchmarked Ollama, vLLM (v0.6.3), and Hugging Face TGI (Text Generation Inference) under synthetic load ranging from 1 to 500 concurrent connections. The results highlight the massive architectural divide between local desktop inference and enterprise multi-tenant serving.

1. The 500-User Stress Test Setup

We hosted Llama-3.3-70B-Instruct (AWQ / Q4) on a server with dual NVIDIA RTX 4090s (48GB combined VRAM) and 128GB DDR5 RAM. Using a locust load-testing cluster, we sent prompts averaging 512 input tokens with 256 requested generation tokens.

2. Why PagedAttention Prevents Memory Fragmentation

Traditional serving engines allocate contiguous blocks of VRAM for the KV cache of each request based on the maximum possible sequence length. If a user asks a 100-token question in a 4096-token configured model, 97.5% of the allocated VRAM is wasted.

vLLM's PagedAttention breaks the KV cache into virtual memory pages (blocks of 16 or 32 tokens), bringing memory waste down from ~70% to under 4%. This allows vLLM to pack 8x to 12x more concurrent requests into the exact same 48GB VRAM pool.

4. Concurrency Benchmarks & P99 Latency

5. Production Docker & Engine Configurations

docker run --gpus all -p 8000:8000 \
  -v /data/models:/models \
  vllm/vllm-openai:latest \
  --model /models/Llama-3.3-70B-Instruct-AWQ \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.95 \
  --max-model-len 8192 \
  --enable-prefix-caching \
  --disable-log-stats