The optimization target changed

Moving a private inference stack from llama.cpp to vLLM looked simple at first: replace the server, point the API at it, and keep going. In practice, the project touched model formats, quantization, speculative decoding, cache capacity, heterogeneous GPU behavior, routing, service isolation, monitoring, credential handling, and retirement of obsolete infrastructure.

llama.cpp had taken the original environment a long way. A quantized 27B Qwen-family model could generate at roughly 50 to 57 tokens per second for a single user. But the environment was no longer one model server for one person. Interactive chat, API consumers, and agentic development work now shared the same underlying inference resources.

The relevant question for me became less about the speed of one request and more about scheduling several requests, allocating KV cache, observing queue behavior, and keeping applications independent from a particular GPU or inference engine.

The routing layer made the migration possible

Applications talk to LiteLLM rather than a specific inference host. LiteLLM exposes one logical model name and routes it to multiple backends. That abstraction meant the chat interface and agent platform could preserve their model contract while the backend changed from a GGUF/llama.cpp deployment to mixed-precision Safetensors served by vLLM.

It is the kind of separation that can seem unnecessary before a migration and essential during one. I was able to change hardware behavior, quantization, context configuration, and the inference engine without turning the migration into an application rewrite. I could test a backend directly, validate it through the router, and roll the calling applications forward only after their existing contract still worked.

I verified more than a successful completion response at this layer. The contract had to preserve authentication, logical model selection, streamed responses, practical request limits, and useful errors when a backend was not eligible to serve a request. That is where an infrastructure migration becomes visible to users: not in which process owns GPU memory, but in whether the interface they depend on keeps behaving predictably.

The logical model name was intentionally more stable than any individual host. It gives me room to test a replacement node, adjust an engine configuration, or take a node out of rotation without teaching every downstream client how the platform is assembled. The routing layer is therefore both a migration mechanism and a long-term control point.

Performance mattered, but it was not the only result

The vLLM configuration standardized a 131,072-token context window and used native multi-token prediction. On the faster inference node, two-token speculation produced observed generation rates around 79 to 88 tokens per second, compared with approximately 57 tokens per second in the previous configuration. A second node reached about 76 tokens per second under a lower power and memory-clock envelope.

The more important capability change for me was concurrency. vLLM's scheduler and paged KV cache give me operating primitives for managing mixed interactive and agent workloads, while its native metrics expose the state I need to reason about that behavior. I can see when work is running or waiting, whether cache is becoming a constraint, and how the engine is behaving rather than treating a model server as a black box.

I designed around contention, not an isolated prompt

The clients sharing this platform behave differently. Interactive chat benefits from low perceived latency. API callers need an explicit, reliable contract. Agentic work can create long prompts, repeated tool loops, and bursts of generation that look very different from a chat exchange. I needed an engine whose scheduler and cache model I could observe while those demands overlap.

That made KV cache a first-class capacity concern. A 131K context window is useful only when I can understand what it does to concurrent requests. I evaluated long-context behavior alongside ordinary requests and used active and waiting request counts, prompt and generation throughput, and cache utilization to reason about the workload. The goal was not to claim a maximum theoretical context size; it was to keep the service intelligible when demand is mixed.

Configuration changes form a system

The checkpoint representation, context limit, prefix cache, and speculative-decoding setting are coupled choices. More context changes cache pressure. Cache pressure changes concurrency. Speculative decoding can improve output speed, but it also introduces acceptance behavior worth measuring. Routing metadata must accurately state what the backend can accept or callers will make decisions using a fictional capability.

I made those relationships explicit in the serving configuration and monitoring. That was a more useful operating model than treating each flag as a disconnected performance tweak. When I adjust one dimension later, I have a clear set of dependent signals and application behaviors to validate.

I measured a deployment, not a universal benchmark

I measured repeated generation behavior on my own nodes, with the same model family and configuration but different power and memory-clock limits. The stronger node reached the higher observed throughput; the lower-power node did not. That was useful because speculative-decoding acceptance remained effectively comparable between them, which kept me from blaming the serving configuration for a difference that was primarily physical.

The measurement was only one input to the migration decision. It does not claim that vLLM is always faster, that every model benefits by the same amount, or that the same result will hold on another hardware profile. What it did show was that this deployment had more headroom and a better operating model for its actual mixed workload.

Treat observability and privilege as part of the cutover

The migration also required rebuilding the operational envelope. A model server that responds to a prompt is not necessarily ready for production use. The final system used a dedicated unprivileged service identity, explicit cache ownership, rotated application credentials, and engine-native monitoring.

The most instructive failure was a restart under the new service account. GPU access, memory allocation, model access, shared memory, and cache locations all appeared healthy. The actual issue was a stale just-in-time compilation artifact created earlier by root. The fix was not to grant more access; it was to correct the ownership boundary of the cache that the process legitimately needed.

The monitoring migration followed the same principle. I redesigned my existing dashboards around vLLM's request, token, cache, and speculative-decoding signals. A renamed job label would have hidden the fact that the underlying telemetry model had changed. I wanted the dashboard to answer operational questions: Is work backing up? Is token generation healthy? Is cache pressure limiting concurrency? Is speculative decoding contributing? Those answers needed native engine signals, not a cosmetic rename.

I also tested the monitoring path as a dependency of the service rather than as documentation after the fact. A successful scrape, representative engine metrics, dashboard queries, and alert semantics all needed to agree. Otherwise I would have created a faster platform with a weaker ability to diagnose it.

The validation sequence I used

I validated the migration from the inside out, then back through the user-facing path:

  1. I confirmed that each engine could load the intended checkpoint and complete representative requests.
  2. I checked the service under its intended unprivileged identity, including cache ownership and GPU access.
  3. I exercised the logical model through LiteLLM rather than relying only on a direct backend test.
  4. I confirmed chat and agent clients could use their existing credentials and model contract.
  5. I verified the Prometheus targets, engine-native metrics, dashboards, and alert inputs.
  6. I compared observed behavior across both nodes before retiring the legacy path.

That sequence was deliberately conservative. It separates a model-load failure from an identity failure, a routing failure, an application contract failure, and an observability failure. It also gives me a traceable place to start if a later regression appears after a model, engine, or hardware change.

A private AI platform is still infrastructure

The final request path is deliberately simple: chat and agent workloads reach a routing layer, which selects from vLLM inference nodes serving the same logical model. The applications see one model contract. The serving layer handles scheduling and cache management. Observability exposes the engine state needed to operate it.

The lasting lesson for me was not merely that one runtime generated tokens faster. It was that a private inference environment becomes substantially easier to evolve when I separate the model-serving problem from the systems that consume it. That is the design I would carry into the next engine change, hardware refresh, context decision, or additional inference node.