Deploying Open-Weight LLMs Locally with vLLM, AWQ & FlashAttention-2
A complete blueprint for running high-throughput, low-latency open-source models (Llama 3.3, DeepSeek, Qwen 2.5) on private infrastructure with AWQ 4-bit quantization.
Gourav Singh
Founder & AI Systems Architect
Why Open Weights Win for Enterprise Infrastructure
While proprietary cloud APIs offer fast prototyping, enterprise requirements around data sovereignty, regulatory compliance (HIPAA, GDPR, DPDP), and predictable unit economics make self-hosted open-weight models significantly more attractive at scale.
With models like Llama 3.3 70B and Qwen 2.5 72B matching proprietary performance, the engineering challenge shifts from model capability to high-throughput inference optimization.
VRAM Budgeting & AWQ 4-Bit Quantization
A standard 70B parameter model in 16-bit floating point precision requires approximately 140GB of GPU VRAM just to load model weights into memory, demanding multiple expensive A100/H100 GPUs.
Activation-aware Weight Quantization (AWQ) preserves the top 1% of salient weights that protect reasoning accuracy while compressing the remaining 99% into 4-bit integers. This fits a 70B model into under 38GB of VRAM with zero perceptible degradation on standard benchmark suites.
| Quantization Format | VRAM Footprint | Tokens/sec (Batch 1) | MMLU Accuracy |
|---|---|---|---|
| FP16 (Uncompressed) | 142 GB (2x A100 80GB) | 34 tok/s | 82.4% |
| GPTQ 4-bit | 41 GB (1x A100 80GB) | 52 tok/s | 81.1% |
| AWQ 4-bit (FlashAttention-2) | 38 GB (1x A100 / 2x L40S) | 78 tok/s | 82.2% |
| FP8 (Hopper Native) | 72 GB (1x H100 80GB) | 114 tok/s | 82.3% |
Quantization trade-offs for Llama-3-70B running on vLLM 0.6.x.
Production vLLM Server Deployment Config
Below is the verified production launch script using PagedAttention, FlashAttention-2, and CUDA graph capture for sub-30ms first-token latency:
#!/usr/bin/env bash
python3 -m vllm.entrypoints.openai.api_server \
--model casperhansen/llama-3.3-70b-instruct-awq \
--quantization awq \
--tensor-parallel-size 2 \
--max-model-len 8192 \
--gpu-memory-utilization 0.94 \
--max-num-seqs 256 \
--enable-chunked-prefill \
--dtype float16 \
--port 8000Private On-Premises LLM Clusters
Deploy state-of-the-art open weights inside your sovereign data boundary.