AIPerf: Benchmark LLM Inference on Kubernetes

💡 Quick Answer: pip install aiperf, then from a pod inside the cluster run aiperf profile --model <served-model> --tokenizer <hf-model> --endpoint-type chat --streaming --url http://<svc>:8000 --concurrency 16 --request-count 200. AIPerf (ai-dynamo/aiperf) is NVIDIA’s successor to GenAI-Perf: it reports TTFT, time to second token, inter-token latency, request latency and throughput, and adds request-rate/arrival patterns, public datasets, goodput SLOs, trace replay, GPU telemetry and a live dashboard. In Jobs use --ui simple or --ui none.

Why AIPerf

HTTP load tools (wrk, hey, k6) don’t understand streamed tokens. AIPerf measures what LLM users feel — first-token delay and streaming smoothness — under controlled load, and replaces GenAI-Perf, which NVIDIA is phasing out. It runs as a set of cooperating processes over a ZeroMQ bus, so a single client can drive high concurrency without becoming the bottleneck.

Install

pip install aiperf
aiperf profile --help

Run it from a pod in the cluster (same namespace or at least the same network path as real clients) so ingress, VPN and laptop Wi-Fi don’t pollute the numbers.

Quick Benchmark

aiperf profile \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --tokenizer meta-llama/Llama-3.1-8B-Instruct \
  --endpoint-type chat \
  --streaming \
  --url http://vllm-service.ai-inference:8000 \
  --concurrency 16 \
  --request-count 200 \
  --warmup-request-count 10

• --model must equal the server’s model name (curl http://vllm-service:8000/v1/models).
• --tokenizer must match the model, or every token-based metric is wrong. For air-gapped clusters pass a local path (--tokenizer /models/tokenizers/llama3).
• The same command works for NIM, SGLang, TGI and Triton’s OpenAI-compatible frontend — change --url/--model.
• Authenticated endpoints: -H "Authorization: Bearer $TOKEN".

Run as a Kubernetes Job

apiVersion: batch/v1
kind: Job
metadata:
  name: aiperf-benchmark
  namespace: ai-inference
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: aiperf
          image: python:3.12-slim
          command: ["/bin/bash", "-c"]
          args:
            - |
              set -euo pipefail
              pip install -q aiperf
              aiperf profile \
                --model meta-llama/Llama-3.1-8B-Instruct \
                --tokenizer meta-llama/Llama-3.1-8B-Instruct \
                --endpoint-type chat --streaming \
                --url http://vllm-service:8000 \
                --concurrency 1,4,16,64 \
                --request-count 500 \
                --warmup-request-count 10 \
                --synthetic-input-tokens-mean 512 \
                --output-tokens-mean 256 \
                --random-seed 42 \
                --ui simple \
                --artifact-dir /results/llama8b
          env:
            - name: HF_TOKEN                # gated tokenizers
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token
          resources:
            requests:
              cpu: "4"
              memory: 4Gi
          volumeMounts:
            - name: results
              mountPath: /results
      volumes:
        - name: results
          persistentVolumeClaim:
            claimName: benchmark-results

A comma-separated --concurrency list runs a sweep in one invocation; see AIPerf concurrency sweeps for analysis. AIPerf also ships a JobSet-based Kubernetes operator for managed, repeatable runs (see the upstream docs/kubernetes guide).

Load Shapes

# Fixed concurrency (closed loop): N requests always in flight
aiperf profile ... --concurrency 32 --request-count 500

# Request rate (open loop), Poisson arrivals — the default arrival pattern
aiperf profile ... --request-rate 10 --request-count 500

# Rate with a concurrency cap, constant or bursty (gamma) arrivals
aiperf profile ... --request-rate 10 --concurrency 50 --arrival-pattern constant
aiperf profile ... --request-rate 10 --arrival-pattern gamma --arrival-smoothness 0.5

# Ramp concurrency from 1 to target over 60 s, or run for a fixed time
aiperf profile ... --concurrency 100 --concurrency-ramp-duration 60 --request-count 1000
aiperf profile ... --concurrency 32 --benchmark-duration 300

Closed-loop concurrency answers “how much can one replica sustain”; open-loop request rate answers “what happens to latency at X req/s” and exposes queueing that concurrency tests hide.

Datasets

# Synthetic prompts with controlled input/output sequence lengths
aiperf profile ... --synthetic-input-tokens-mean 1024 --synthetic-input-tokens-stddev 64 \
  --output-tokens-mean 256 --extra-inputs max_tokens:256

# Public conversational dataset
aiperf profile ... --public-dataset sharegpt

# Your own prompts (JSONL)
aiperf profile ... --input-file prompts.jsonl --custom-dataset-type single_turn

Timestamped production traces (e.g. Mooncake format) can be replayed with their original timing — see AIPerf trace replay. If output lengths come back shorter than requested, the server stopped at EOS: set max_tokens via --extra-inputs (and ignore_eos:true on vLLM for fixed-length tests).

Goodput (SLO-Based Throughput)

Count only requests that meet latency targets. Keys are metric tags, values in display units (ms):

aiperf profile ... --concurrency 50 --request-count 500 \
  --goodput "time_to_first_token:500 inter_token_latency:50"

Deep dive: AIPerf goodput.

Multiple Replicas, Embeddings and Rankings

# Round-robin across replicas (e.g. behind a headless Service)
aiperf profile ... --url http://vllm-0.vllm-headless:8000 --url http://vllm-1.vllm-headless:8000

# Embeddings (OpenAI-style) and NIM rerankers
aiperf profile --model nvidia/nv-embedqa-e5-v5 --endpoint-type embeddings \
  --url http://embedding-service:8000 --concurrency 50 --request-count 1000
aiperf profile --model nvidia/nv-rerankqa-mistral-4b-v3 --endpoint-type nim_rankings \
  --url http://ranking-service:8000 --concurrency 20 --request-count 500

Other endpoint types include completions, responses, nim_embeddings, audio_transcription and image_generation; multimodal (vision) requests go through chat. Run aiperf profile --help for your version’s list. Benchmarking several models at once: AIPerf multi-model.

GPU Telemetry and Server Metrics

aiperf profile ... --gpu-telemetry http://nvidia-dcgm-exporter.gpu-operator:9400/metrics

AIPerf defaults to DCGM on localhost:9400/9401, so in Kubernetes pass the DCGM Exporter Service URL. It also scrapes the inference server’s own Prometheus endpoint (<url>/metrics) by default; add more with --server-metrics, or disable with --no-server-metrics.

Reading the Output

            NVIDIA AIPerf | LLM Metrics
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃                      Metric ┃     avg ┃    min ┃     max ┃     p99 ┃     p50 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│        Request Latency (ms) │ 1234.56 │ 987.34 │ 1567.89 │ 1534.23 │ 1223.45 │
│    Time to First Token (ms) │  234.56 │ 189.23 │  298.45 │  289.34 │  231.12 │
│    Inter Token Latency (ms) │   15.67 │  12.34 │   19.45 │   19.01 │   15.45 │
│ Output Token Count (tokens) │  150.00 │ 120.00 │  180.00 │  178.90 │  149.00 │
│  Request Throughput (req/s) │   31.45 │      - │       - │       - │       - │
└─────────────────────────────┴─────────┴────────┴─────────┴─────────┴─────────┘

Results go to --artifact-dir (default artifacts/<model>-<endpoint>-<load>/): profile_export_aiperf.json and .csv summaries plus per-request profile_export.jsonl. --num-profile-runs 3 repeats the run and reports confidence intervals.

Common Issues

Frequently Asked Questions

What is NVIDIA AIPerf?

An open-source (Apache 2.0) benchmarking tool from NVIDIA’s Dynamo project for generative AI inference. It drives OpenAI-compatible and other endpoints with synthetic, public or recorded workloads and reports LLM metrics such as TTFT, ITL, per-user throughput and goodput.

AIPerf vs GenAI-Perf — which should I use?

AIPerf. GenAI-Perf is being phased out with no new features; AIPerf keeps a similar profile CLI and adds arrival patterns, goodput, trace replay, multi-URL load balancing and live dashboards. Keep GenAI-Perf only where you need continuity with existing baselines — see GenAI-Perf.

How do I benchmark vLLM on Kubernetes with AIPerf?

Run AIPerf in a pod or Job in the cluster against the vLLM Service: aiperf profile --model <served-name> --tokenizer <hf-model> --endpoint-type chat --streaming --url http://<vllm-svc>:8000 --concurrency 16. For offline/batch throughput see AIPerf with vLLM offline.

Should I use concurrency or request rate?

Use concurrency to find the maximum sustainable load per replica, and request rate (Poisson arrivals) to see latency under realistic traffic at a given req/s. Production sizing usually needs both.

How do I measure SLO compliance?

Use --goodput with metric-tag thresholds, for example "time_to_first_token:500 inter_token_latency:50"; AIPerf reports the throughput of requests that met every target.