• The Role of an Inference Engine [00:02:16]:

  • Software layer operating between raw weights on disk and incoming HTTP user requests (e.g., vLLM, Ollama).
  • Performs five core responsibilities: loading model weights, maintaining context state, scheduling requests, executing matrix arithmetic, and serving an HTTP API.
  • Sits above compute kernels (low-level math operations) and below cluster orchestrators/control planes and API gateways.

• GPU Memory Dynamics (Weights vs. KV Cache) [00:05:16]:

  • Model Weights: Static memory cost determined by parameter count and precision (e.g., an 8B parameter model at 16-bit/2-byte precision occupies ~16 GB VRAM); remains fixed regardless of concurrent concurrency.
  • KV Cache: Dynamic working memory holding intermediate attention states across token generations; scales linearly with conversation depth and concurrent active users, directly competing with weights for remaining VRAM.

• Batching Strategies & Memory Management [00:07:39]:

  • Continuous Batching: Replaces naive static batching by evicting completed requests immediately and slotting new requests into the running batch, avoiding GPU idle cycles caused by differing generation lengths.
  • PagedAttention / Paging: Allocates KV cache in small, non-contiguous fixed-size blocks on demand rather than reserving maximum possible context windows per request, eliminating severe internal memory fragmentation.

• Weight Quantization [00:09:47]:

  • Reduces precision per parameter (e.g., from FP16 to INT8/INT4), slashing fixed model footprint.
  • The primary benefit is freeing VRAM for a larger KV cache pool, directly scaling concurrent user throughput on cost-effective GPUs.

• Prefix Caching [00:10:58]:

  • Caches identical token sequences across requests (e.g., static system instructions, tool definitions, recurring multi-turn histories).
  • Bypasses redundant prompt reprocessing, providing massive speed and efficiency gains for agentic workflows.

• Prefill vs. Decode & Speculative Decoding [00:12:28]:

  • Prefill Phase: Parallel ingestion of prompt tokens; strictly compute-bound as the GPU operates at peak arithmetic saturation.
  • Decode Phase: Autoregressive, strictly sequential token generation; heavily memory-bandwidth bound due to fetching full weights per generated token.
  • Speculative Decoding: Employs an ultra-fast draft model to predict token sequences validated by the target model in a single parallel pass, reducing end-to-end decode latency for individual users.

• Orchestration, Routing, and Model Scaling [00:14:46]:

  • Control Plane & Gateways: Standard round-robin load balancing fails for LLMs; gateways require cache-aware routing to send requests to replicas with pre-warmed KV caches.
  • Model Sharding: Distributes massive parameter architectures across multiple GPUs via Tensor Parallelism (intra-layer splitting) or Pipeline Parallelism (inter-layer pipelining).
  • LLM Autoscaling: Relies on inference-specific metrics (queue depth, KV cache utilization) rather than CPU/GPU utilization; hindered by substantial cold-start penalties when pulling images and warming weights.
  • Prefill/Decode Disaggregation: Decouples the prefill (compute-intensive) and decode (bandwidth-intensive) workloads across specialized hardware topologies for optimal hardware utilization.

This precisely quantifies why longer context windows are expensive beyond just model size: KV cache grows quadratically with context, and current GPU memory can't keep pace. Each eviction or recomputation directly inflates cost-per-token — making KV cache the hidden tax on long-context AI workloads.