Performance Testing of LLM Inference Endpoints

LLM inference is the runtime process of generating model responses from prompts, and performance testing LLM inference endpoints is now a core reliability discipline for AI products. Latency testing is the measurement of response delay under controlled workload conditions. Token throughput is the rate at which a system produces or processes tokens per second. Model serving is the infrastructure layer that hosts, schedules, batches, and returns model outputs through an API.

Why LLM Inference Performance Testing Differs From Standard API Load Testing

LLM inference performance testing is different because every request has variable computational work, variable response length, and often a long-lived streaming connection. A conventional API load testing script that checks only status codes and average response time will miss the behavior users actually feel.

Traditional REST endpoints usually perform bounded work: read data, compute a response, and return a payload. LLM inference endpoints perform prefill work over the prompt, then decode one token at a time until a stop condition, maximum token limit, or timeout occurs.

This distinction makes averages dangerous. A test can show a 2.5 second mean response time while hiding 20 second tail latency for long prompts, slow first tokens, or queuing when GPU memory is saturated.

Teams that mature their LLM performance suites commonly report 30 to 50 percent faster capacity planning cycles because they stop debating anecdotal chat behavior and start comparing controlled token-level measurements. The practical goal is not to find one magic throughput number; it is to map service behavior across the realistic prompt and generation patterns your users create.

How does token generation change the definition of latency?

Token generation changes latency because an LLM response is produced incrementally rather than all at once. Time to first token is the delay between submitting a request and receiving the first generated token, while inter-token latency is the delay between subsequent generated tokens.

For a streaming chatbot, time to first token often matters more than total completion time because users perceive responsiveness as soon as content begins. For a batch extraction workflow, total completion time and tokens per second may dominate because no user is watching the stream.

A useful latency testing plan separates prefill latency, queue latency, decode latency, and network latency. Without that separation, teams may optimize the model while the real bottleneck is request scheduling, reverse proxy buffering, or client-side stream handling.

When should you test full response latency versus streaming latency?

You should test full response latency when the consuming system waits for the entire answer, and you should test streaming latency when a user or downstream agent can act on partial output. The wrong choice can make a healthy endpoint look slow or an unusable endpoint look acceptable.

Full response latency is appropriate for summarization jobs, document classification, data transformation, and automated evaluation pipelines. Streaming latency is appropriate for assistants, coding copilots, search answer interfaces, and agent orchestration where partial output is part of the experience.

Many production systems need both views. A support assistant, for example, may stream text to the user but also wait for final structured metadata before committing the conversation state.

Core Metrics for LLM Inference, Latency Testing, and Token Throughput

The most reliable LLM performance scorecard combines user-visible latency, token throughput, capacity saturation, and quality-of-service stability. No single metric can represent model serving performance because prompt length, generation length, concurrency, and batching interact nonlinearly.

Time to first token is the most common user-perceived responsiveness metric for streamed LLMs. End-to-end latency is the duration from request submission to final token or final response body.

Output token throughput is the number of generated tokens per second, often measured per request, per model replica, or per GPU. Input token throughput is the number of prompt tokens processed per second during prefill, and it becomes critical for retrieval-augmented generation where prompts may contain thousands of context tokens.

Queue time is the time a request waits before model execution begins. Queue time often grows sharply once concurrency exceeds the scheduler, batcher, GPU memory, or rate-limit capacity.

Metric	What it reveals	Common failure signal	Primary owner
Time to first token	Perceived responsiveness for streaming users	Users wait several seconds before any visible answer	QA, platform, model serving
Inter-token latency	Smoothness of streamed output	Responses arrive in bursts or pause mid-sentence	Model serving, networking
Output tokens per second	Decode capacity and generation speed	Throughput collapses under modest concurrency	ML platform
Input tokens per second	Prompt prefill efficiency	Long-context requests delay all users	ML platform, application team
Queue time	Scheduler pressure and saturation	Latency spikes while GPU utilization is high	Platform engineering
Error and cancellation rate	Reliability under load	Timeouts, dropped streams, provider throttling	SRE, QA
Cost per response	Economic viability of serving configuration	Performance target is met only at unsustainable cost	Engineering leadership

Tail latency deserves special attention. For interactive AI products, p95 and p99 time to first token usually correlate better with customer complaints than average latency.

A practical service-level target might specify p95 time to first token below 1.5 seconds, p95 inter-token gap below 150 milliseconds, output throughput above 35 tokens per second per active request, and stream error rate below 0.5 percent. These values are not universal, but they force explicit negotiation between user experience, cost, and model quality.

Designing Workloads That Represent Real Model Serving Demand

A representative workload for model serving must model prompt distributions, output distributions, user arrival patterns, and endpoint features such as streaming, tools, and retrieval context. Synthetic concurrency alone is not enough because two users can consume radically different amounts of compute.

Start by segmenting production or expected traffic into workload classes. Typical classes include short chat turns, long retrieval-augmented prompts, structured extraction, code generation, agent tool planning, and batch summarization.

For each class, define input token ranges, maximum output tokens, temperature, top-p, stop sequences, streaming mode, and expected cancellation behavior. If you test every request with a 50-token prompt and 100-token output, your results will not survive contact with production traffic.

Arrival rate also matters. Closed-loop tests keep a fixed number of virtual users active, while open-loop tests send requests at a target rate regardless of response time. Open-loop testing is better for measuring backpressure and queue growth, while closed-loop testing is useful for user journey realism.

How should prompt and output length distributions be modeled?

Prompt and output length distributions should be modeled with percentiles, not a single average. A realistic test might include 50 percent short prompts, 35 percent medium prompts, 10 percent long-context prompts, and 5 percent extreme prompts that exercise maximum context limits.

Output lengths should be controlled separately from prompt lengths. A long prompt with a short answer stresses prefill, while a short prompt with a long answer stresses decode throughput.

Teams often discover that 5 percent of long-context requests consume 30 to 60 percent of total GPU time. That insight can justify request routing, context compression, token budgets, or separate model pools for premium workloads.

How does streaming alter virtual user behavior?

Streaming alters virtual user behavior because the client remains connected while tokens arrive, and users may cancel once they have enough information. A load test that waits for complete responses but never models cancellation can overstate backend cost and understate frontend connection pressure.

For chat products, include think time between turns, mid-stream cancellation, and multi-turn context growth. For API products, include client timeouts, retry policies, and consumption speed if clients parse streamed events slowly.

Streaming tests should record first-byte arrival, first-token arrival, token cadence, stream completion, and abnormal termination. This exposes problems such as proxy buffering, server-sent event framing issues, idle timeout mismatches, and connection pool exhaustion.

Tooling Options for LLM Inference Endpoint Benchmarks

The best tooling depends on whether you need protocol realism, token-level observability, GPU correlation, or broad CI integration. General load testing tools are useful, but LLM-specific harnesses reduce custom code when measuring token throughput and model serving internals.

k6 is strong for API-level scenarios, thresholds, and CI-friendly scripts. Locust is effective when workload behavior is easier to express in Python, especially for multi-step conversations or agent flows.

vLLM benchmark utilities, NVIDIA Triton tooling, and custom harnesses are better when you own the serving stack and need scheduler, batching, KV cache, or GPU-level insight. Hosted model providers require black-box tests, so you may need client-side metrics plus vendor rate-limit and retry telemetry.

Approach	Best fit	Strength	Trade-off
k6 HTTP streaming scripts	CI performance gates for API endpoints	Fast execution, clear thresholds, strong dashboards	Token parsing needs custom code
Locust user models	Conversation flows and adaptive behavior	Python flexibility and realistic user pacing	Requires careful distributed setup at high scale
vLLM benchmarks	Self-hosted OpenAI-compatible serving	Native token throughput and scheduler visibility	Less representative of full application path
Triton Inference Server metrics	GPU-backed enterprise model serving	Deep inference and hardware telemetry	Requires platform ownership and setup discipline
Custom harness	Special protocols, tools, or agents	Exact workload modeling	Higher maintenance and validation burden

The highest-signal setups combine a traffic generator, application metrics, model server metrics, and infrastructure telemetry. For example, a k6 script may generate streamed chat traffic while Prometheus captures GPU utilization, queue depth, request batch sizes, and memory pressure.

Do not treat tool output as truth until you validate that the client can generate load faster than the system under test. In LLM testing, client CPU, network bandwidth, event-stream parsing, and TLS connection reuse can become hidden bottlenecks.

Example k6 Script for Streaming LLM Latency Testing

A minimal but useful LLM latency testing script should capture status, time to first byte, full response duration, approximate token count, and request parameters. The example below uses k6 to send OpenAI-compatible chat completions and enforce basic performance thresholds.

This script is intentionally API-level rather than model-server-specific. It is suitable for smoke performance checks in continuous performance testing, while deeper benchmark runs should add token-by-token parsing and backend telemetry correlation.

import http from 'k6/http';
import { check, sleep } from 'k6';
import { Trend, Rate } from 'k6/metrics';

const firstByte = new Trend('llm_time_to_first_byte_ms');
const totalLatency = new Trend('llm_total_latency_ms');
const successRate = new Rate('llm_success_rate');

export const options = {
  scenarios: {
    chat_load: {
      executor: 'ramping-arrival-rate',
      startRate: 2,
      timeUnit: '1s',
      preAllocatedVUs: 50,
      maxVUs: 300,
      stages: [
        { duration: '5m', target: 10 },
        { duration: '10m', target: 40 },
        { duration: '5m', target: 80 },
        { duration: '5m', target: 0 }
      ]
    }
  },
  thresholds: {
    llm_success_rate: ['rate greater than 0.995'],
    llm_time_to_first_byte_ms: ['p(95) less than 1500'],
    llm_total_latency_ms: ['p(95) less than 12000']
  }
};

const prompts = [
  'Summarize this customer complaint in three concise bullets.',
  'Draft a support reply for a delayed enterprise integration.',
  'Extract the action items from a meeting transcript excerpt.'
];

export default function () {
  const prompt = prompts[Math.floor(Math.random() * prompts.length)];
  const payload = JSON.stringify({
    model: 'production-chat-model',
    messages: [{ role: 'user', content: prompt }],
    temperature: 0.2,
    max_tokens: 350,
    stream: false
  });

  const params = {
    headers: {
      'Content-Type': 'application/json',
      'Authorization': 'Bearer ' + __ENV.LLM_API_TOKEN
    },
    timeout: '30s'
  };

  const started = Date.now();
  const res = http.post(__ENV.LLM_ENDPOINT + '/v1/chat/completions', payload, params);
  const ended = Date.now();

  firstByte.add(res.timings.waiting);
  totalLatency.add(ended - started);
  successRate.add(res.status === 200);

  check(res, {
    'response status is 200': r => r.status === 200,
    'response has completion body': r => r.body && r.body.length > 100
  });

  sleep(Math.random() * 3 + 1);
}

For streamed responses, extend the harness to parse server-sent events and record first token rather than first byte. First byte can be misleading if the server sends headers quickly but delays meaningful tokens.

Run small calibration tests before high-load tests. If a single request has high variance on an idle system, increasing concurrency will only amplify noise and make root-cause analysis harder.

Capacity Planning With Concurrency, Batching, and GPU Saturation

Capacity planning for LLM inference requires measuring the saturation curve, not just the highest successful request rate. The useful operating point is usually below maximum throughput because tail latency and error rates rise steeply near full GPU or scheduler saturation.

Continuous batching improves utilization by combining active token generation steps across requests. However, batching can also increase queue time, create fairness issues, and penalize short requests behind long generations if scheduler settings are poorly tuned.

KV cache memory is often the hidden limiter. KV cache is the memory used to store attention keys and values from previous tokens so the model can generate subsequent tokens efficiently.

At moderate traffic, GPU compute may be the bottleneck. At long context lengths or high concurrency, memory pressure can become dominant, causing rejected requests, preemption, paging, or severe latency spikes.

How do you find the safe operating limit?

You find the safe operating limit by ramping load until one or more service-level indicators degrade, then backing off to a stable margin. A common target is to operate at 60 to 75 percent of the concurrency level where p95 latency or error rate begins its sharp climb.

Plot arrival rate, active requests, queue depth, time to first token, output tokens per second, GPU utilization, GPU memory, and errors on the same timeline. The first metric to inflect usually identifies the bottleneck.

For example, if GPU utilization is 55 percent but queue time rises, the serving scheduler or per-replica concurrency limit may be constraining the system. If utilization is 95 percent and output tokens per second flattens, compute saturation is likely.

When should model routing be part of the performance strategy?

Model routing should be part of the performance strategy when requests have different latency, quality, or context requirements. Routing short low-risk tasks to smaller models can protect premium model capacity and reduce cost per successful response.

Many teams see 20 to 40 percent infrastructure cost reduction after separating simple classification, rewrite, and extraction calls from complex reasoning calls. The performance gain comes from matching workload shape to serving capacity rather than forcing every prompt through the largest model.

Routing also supports graceful degradation. During a traffic spike, noncritical tasks can move to cheaper models, shorter context windows, or asynchronous processing while interactive user flows keep their latency budget.

Common Mistakes That Break LLM Performance Benchmarks

Most failed LLM benchmarks are invalid because they simplify away the workload characteristics that dominate production performance. The common pattern is a clean chart that answers the wrong question.

The first mistake is testing with one prompt and one max-token value. This produces a repeatable benchmark but not a representative benchmark.

The second mistake is ignoring warm-up and cache effects. Model weights, tokenizer caches, CUDA kernels, routing paths, and provider-side caches can make early or repeated runs look better or worse than steady-state reality.

The third mistake is mixing retries into latency without labeling them. Retries may improve success rate while hiding throttling, overload, or poor backoff design in resilience testing.

• Do not benchmark only average latency. p95 and p99 metrics expose queueing, long prompts, and noisy neighbors that averages hide.
• Do not compare models with different output lengths. A model that writes shorter answers may appear faster while doing less work.
• Do not ignore tokenizer differences. The same text can produce different token counts across model families.
• Do not overload the client generator. A saturated load generator creates artificially low server pressure and distorted timings.
• Do not forget cancellation behavior. User-aborted streams affect capacity and should appear in the workload if they happen in production.

Another subtle failure is measuring the model endpoint in isolation when the product path includes retrieval, moderation, guardrails, tool calls, and post-processing. Isolation tests are valuable, but release decisions need at least one end-to-end scenario that includes the full AI application test strategy.

Release Gates and Regression Benchmarks for Model Serving Changes

LLM inference endpoints need performance release gates because small configuration changes can create large latency and cost regressions. A practical gate combines fast CI checks, scheduled benchmark suites, and pre-release soak tests.

Fast CI checks should detect obvious regressions in latency, status codes, stream behavior, and response schema. They should finish in 10 to 20 minutes and use stable prompts with controlled token budgets.

Scheduled benchmark suites should run broader prompt distributions, longer duration, and multiple concurrency levels. These tests are where token throughput, queueing behavior, and saturation curves become visible.

Pre-release soak tests should exercise the endpoint for several hours with production-like arrival patterns. Soak testing is especially important for memory fragmentation, connection leaks, autoscaling instability, and provider-side rate-limit drift.

A mature gate might fail a release if p95 time to first token regresses by more than 15 percent, stream error rate exceeds 0.5 percent, output tokens per second falls below the baseline by 10 percent, or cost per thousand successful responses rises beyond the agreed budget. These thresholds work best when stored with the model version, serving image, hardware profile, prompt mix, and tokenizer version.

Can quality evaluation and performance testing run together?

Quality evaluation and performance testing can run together, but they should remain analytically separate. Combining them is useful when faster decoding settings, smaller models, quantization, or routing changes may alter answer quality.

Track performance metrics alongside task success, groundedness, refusal correctness, and format compliance. If a change improves token throughput by 35 percent but doubles hallucinated citations, the release is not a performance win for the product.

For high-stakes workflows, keep a fixed evaluation set that runs before and after each performance benchmark. This prevents teams from optimizing for speed while silently degrading business outcomes.

Observability Signals Needed During LLM Inference Load Tests

LLM load tests need observability at the client, gateway, application, model server, and hardware layers. Without correlated telemetry, bottleneck diagnosis becomes guesswork.

Client metrics should include request start time, time to first byte, time to first token, stream completion time, token counts, status, retry count, and cancellation reason. Gateway metrics should include connection counts, upstream latency, buffering, timeout events, and rate-limit responses.

Application metrics should record prompt construction time, retrieval latency, guardrail latency, tool-call latency, and post-processing time. Model server metrics should include queue depth, batch size, prefill time, decode time, active sequences, cache utilization, and rejected requests.

Hardware metrics should include GPU utilization, GPU memory usage, memory bandwidth, CPU utilization, network throughput, and container throttling. If you use autoscaling, capture scale-out triggers and cold-start time because new replicas may not be ready when traffic arrives.

The strongest benchmark reports align all these signals on a single timeline. That alignment turns a vague statement like the model was slow into a precise finding such as long-context requests pushed KV cache memory above 90 percent, queue time increased after 42 requests per second, and p95 time to first token crossed the threshold within three minutes.

Key Takeaways

• LLM inference performance testing must measure token-level behavior, not only HTTP response time.
• Time to first token, inter-token latency, token throughput, queue time, and error rate are the core signals for interactive model serving.
• Representative benchmarks require realistic prompt lengths, output lengths, streaming behavior, concurrency, and cancellation patterns.
• Maximum throughput is not the same as safe capacity; reliable systems operate below the saturation point where tail latency climbs sharply.
• General tools such as k6 and Locust work well when paired with LLM-specific token metrics and backend observability.
• Common benchmark failures include single-prompt tests, ignored cache effects, overloaded clients, and unlabeled retries.
• Performance gates should protect both speed and answer quality because faster inference is not valuable if model output degrades.