A serving migration is a platform migration

The original inference stack served a 27B Qwen-family model through llama.cpp. It was stable and delivered strong single-user performance, but the environment had grown beyond a single model server. Interactive chat, API consumers, and an agentic development platform were now sharing the same inference layer.

My objective was not simply to replace one executable. I wanted to improve concurrent serving, retain a large context window, expose engine-native observability, and preserve a stable application contract while the serving backend changed. I approached it as an infrastructure migration: define what callers must not notice, make the operational evidence explicit, then retire the old path only after the whole request chain is proven.

Keep the applications outside the change

LiteLLM provided the important abstraction boundary. Applications continued to request the same logical model name while LiteLLM routed requests to the serving backends. That meant LibreChat and the agent platform did not need to know whether an individual request was served by llama.cpp or vLLM, or which physical inference node handled it.

That separation reduced migration risk. The serving engine, model representation, scheduling behavior, and physical hardware could change behind a stable application-facing interface.

I treated that boundary as a compatibility contract, not merely a convenient proxy. Before moving traffic, I verified the model alias, authentication behavior, streaming responses, request limits, and failure behavior through the same route used by the chat and agent clients. A direct request to a new engine proves only that the engine responds. It does not prove that the application can select it, interpret its responses, or recover when an upstream node is unavailable.

The router also gave me a useful operational seam. I could place one backend in the candidate role, test it directly and through the logical model name, then decide when it was ready for ordinary traffic. That makes rollback a routing decision instead of an emergency application change.

Why I chose vLLM for the new workload

The new production layer uses vLLM with a mixed-precision Safetensors checkpoint, prefix caching, a 131,072-token context window, and multi-token prediction. The intent was to trade a single-request-oriented serving model for one better suited to concurrent interactive and agent workloads.

vLLM exposed scheduler state, request queues, token throughput, speculative-decoding behavior, and KV-cache utilization as first-class operating signals. Those are more useful controls for a shared AI platform than a fixed number of model slots.

On the higher-performing node, I observed generation throughput increase from approximately 57 tokens per second under the prior configuration to approximately 88 tokens per second after the migration and two-token speculative decoding. The lower-performing node reached approximately 76 tokens per second. I treat these as deployment observations, not a general benchmark: the two otherwise similar nodes operate with different power and memory-clock limits.

I also avoided treating that result as the sole decision criterion. A faster isolated decode is useful, but a platform serving chat and agent requests needs to show me what is running, what is waiting, what is occupying cache, and whether the routing layer is accurately representing what the engines can support.

What I changed deliberately

I kept the model contract stable while changing the layer below it. I moved from a uniform quantized model representation to a mixed-precision checkpoint, standardized the production context window at 131,072 tokens, and enabled prefix caching and multi-token prediction. Those were not independent tuning knobs. Context length affects available cache; cache affects concurrent workload capacity; speculative decoding changes both observed throughput and the metrics I need to watch.

I also corrected the routing metadata to match the actual serving limit. An abstraction layer is only useful if it represents the capability of the systems behind it. Leaving an older, larger advertised context limit in place would have made the control plane more convenient to read and less trustworthy to operate.

Context length is a capacity decision

I chose a 131,072-token limit because the workloads need room for long conversations, source inspection, and tool-oriented tasks. That number is not free capacity. Every active request reserves KV-cache space, and the serving engine has to balance prompt ingestion, generation, and concurrent demand against the GPU memory available to it. A large advertised window that regularly starves other requests is not a feature.

I therefore treated the configured limit, cache behavior, and concurrency as one operating envelope. During validation, I checked long-context requests alongside ordinary interactive requests rather than testing each in isolation. The question I wanted answered was whether the system could continue serving a mixed workload predictably—not whether a single prompt could be made exceptionally large.

Speculative decoding needed evidence, not optimism

Multi-token prediction can improve generation throughput when the draft path is accepted often enough to justify its work. It can also add complexity without producing a useful gain for a particular model and workload. I enabled two-token speculation, measured the observed generation behavior on both nodes, and retained the engine's acceptance and throughput signals as part of the operating view.

That let me distinguish three different outcomes: a backend that is simply healthy, a backend that is busy, and a backend whose speculative configuration is materially contributing to output. I do not need every optimization to be permanent; I need enough telemetry to decide whether it earns the resources and operational attention it consumes.

Production work begins after the model responds

The most valuable part of my migration was the hardening work around the model server. I moved the service to a dedicated unprivileged account with explicit GPU access, neutral runtime locations, and dedicated caches. A failed non-root restart initially looked like a CUDA issue, but my targeted testing showed the GPU path was healthy. The actual problem was a root-owned just-in-time compiler cache left behind during migration testing.

That distinction mattered. Broadly loosening device permissions would have concealed the cause rather than correcting it. Fixing the cache ownership and validating the service account's complete runtime path produced a durable boundary for both inference nodes.

Observability required its own migration. My existing Prometheus jobs, Grafana dashboards, and alert rules were built around llama.cpp metric names. Renaming a scrape job would have left empty panels and misleading alerts. Instead, I mapped each monitoring function to a vLLM-native signal: active and waiting requests, prompt and generation throughput, speculative-decoding acceptance, KV-cache utilization, and tokens processed per engine step.

I rebuilt the questions behind the dashboards

The old dashboard answers were not automatically useful after the engine change. I rebuilt the panels and alerts around questions I would need during an incident: Are requests queueing faster than they complete? Is prompt processing the bottleneck, or is generation? Is the KV cache preventing useful concurrency? Is a throughput change related to hardware state, demand, or the serving configuration?

That framing also avoids a common observability mistake: preserving a familiar chart while losing the meaning behind it. A metric name can be carried forward while its semantics change. I wanted the new monitoring layer to reflect the engine I was actually operating and to make capacity decisions based on evidence rather than a single aggregate tokens-per-second number.

Failure modes I planned for

I expected the migration to fail at seams rather than at the obvious happy path. The model could load while the router advertised the wrong context limit. The service could start as an administrator while failing under its intended identity. A dashboard could render while scraping no useful engine data. A node could generate well on a direct test while behaving differently under shared demand.

For each of those boundaries, I used a check that matched the failure mode: application-path requests for routing, non-root restarts for identity and cache ownership, target health and representative metrics for observability, and repeated mixed-workload checks for capacity behavior. That approach made the cutover less about declaring a server “up” and more about proving that the platform could be run.

Validation followed the request path

My final check was not a direct prompt to a GPU. I exercised the complete application path: client workload, LiteLLM routing, vLLM serving, model response, and monitoring health. I validated both inference nodes, the routing layer, the chat application credential, and the observability targets before retiring the legacy runtime.

The result was more than a faster local model endpoint. It was a serving layer with clearer ownership, stable application contracts, native operating signals, and a practical path for supporting both people and autonomous workloads.