LLM Inference Optimization Part 4 — Production Serving

This is the final part of the series. Here we cover how to combine the Attention optimizations, KV Cache management, and Sparse Attention techniques from Parts 1–3 in a real production environment.

The key tools are vLLM and TGI (Text Generation Inference). We'll walk through how these two engines integrate the optimizations we've learned, and how to configure them in practice — with code.

vLLM vs TGI — At a Glance

Feature	vLLM	TGI (HuggingFace)
PagedAttention	Built-in	Built-in
Continuous Batching	Supported	Supported
Flash Attention	Supported	Supported
KV Cache Quantization	FP8 supported	Partial support
Model Quantization	AWQ, GPTQ, Marlin	AWQ, GPTQ, EETQ
Speculative Decoding	Supported	Supported
Multi-GPU (Tensor Parallel)	Supported	Supported
API Compatibility	OpenAI-compatible	Custom + OpenAI-compatible
Installation	pip install	Docker-based

Deploying vLLM in Practice

Basic Configuration

python

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    dtype="float16",

    # === Memory Management ===
    gpu_memory_utilization=0.90,   # Use 90% of GPU memory
    max_model_len=32768,            # Maximum context length

    # === KV Cache Optimization ===
    kv_cache_dtype="auto",          # "auto", "fp8_e5m2", "fp8_e4m3"
    # kv_cache_dtype="fp8_e5m2",    # FP8 KV Cache → 2x memory savings

    # === Quantization ===
    # quantization="awq",           # Model weight quantization

    # === Parallelism ===
    tensor_parallel_size=1,         # Number of GPUs
)

params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=1024,
    stop=["<|eot_id|>"],
)

output = llm.generate("Explain quantum computing.", params)
print(output[0].outputs[0].text)

OpenAI-Compatible API Server

bash

# Start vLLM server
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --dtype float16 \
    --gpu-memory-utilization 0.9 \
    --max-model-len 32768 \
    --kv-cache-dtype fp8_e5m2 \
    --port 8000

LLM Inference Optimization Part 4 — Production Serving

LLM Inference Optimization Part 4 — Production Serving

vLLM vs TGI — At a Glance

Deploying vLLM in Practice

Basic Configuration

OpenAI-Compatible API Server

Sign in to continue reading

Related Posts

LLM Inference Optimization Part 3 — Sparse Attention in Practice

LLM Inference Optimization Part 2 — KV Cache Optimization

LLM Inference Optimization Part 1 — Attention Mechanism Deep Dive