Canonical page: https://taskbull.si/AIJobs/guides/inference-engineering-portfolio-benchmark/

[TaskBull](https://taskbull.si/) / [AIJobs](https://taskbull.si/AIJobs/) / [Guides](https://taskbull.si/AIJobs/guides/) / An Inference Engineering Portfolio Project: Build a Reproducible Benchmark INFERENCE ENGINEERING · PORTFOLIO FIELD NOTES

# An Inference Engineering Portfolio Project: Build a Reproducible Benchmark

 Plan an auditable inference benchmark: define the workload, separate latency from throughput, control cache effects and build an honest portfolio evidence packet.

 By TaskBull Editorial · November 2, 2026 · 7 min read

 AI-generated fictional workstation; the screens do not represent measured benchmark results. A useful inference-engineering portfolio does more than display a tokens-per-second number. It lets another person understand what you measured, reproduce the conditions and challenge the conclusion . A modest experiment with that evidence can be more informative than an impressive score with no workload attached.

 This project guide takes you from a bounded question to a benchmark report. It includes an example command and a result template, not measured GPU results: TaskBull has not run the serving benchmark described here. Use an endpoint and hardware you already have permission and resources to test.

 KEEP THESE IN MIND

## Key takeaways

- Define one workload and one decision before tuning a serving system. 
- Record model, tokenizer, software, hardware, request shape and cache policy with every run. 
- Compare client latency and system throughput separately; metric names can mean different things across tools. 
- Keep attempted, successful and failed requests visible, and preserve raw evidence rather than only a headline chart. 
- A portfolio benchmark is a bounded experiment, not a production capacity promise or hiring guarantee. 

## In this guide

- [Choose a question that your experiment can answer](https://taskbull.si/AIJobs/guides/inference-engineering-portfolio-benchmark/#choose-a-question) 
- [Write a reproducibility contract before running](https://taskbull.si/AIJobs/guides/inference-engineering-portfolio-benchmark/#write-the-run-contract) 
- [Define the metrics and their measurement boundaries](https://taskbull.si/AIJobs/guides/inference-engineering-portfolio-benchmark/#define-the-metrics) 
- [Follow a five-stage benchmark workflow](https://taskbull.si/AIJobs/guides/inference-engineering-portfolio-benchmark/#benchmark-workflow) 
- [Run a documented baseline, then vary one condition](https://taskbull.si/AIJobs/guides/inference-engineering-portfolio-benchmark/#example-command) 
- [Control cache state, warmup and repeated runs](https://taskbull.si/AIJobs/guides/inference-engineering-portfolio-benchmark/#cache-warmup-and-repeats) 
- [Build an honest result table and comparison](https://taskbull.si/AIJobs/guides/inference-engineering-portfolio-benchmark/#report-without-inventing-results) 
- [Turn the experiment into a portfolio evidence packet](https://taskbull.si/AIJobs/guides/inference-engineering-portfolio-benchmark/#portfolio-evidence) 
- [Sources & editorial notes](https://taskbull.si/AIJobs/guides/inference-engineering-portfolio-benchmark/#sources) 

## Choose a question that your experiment can answer

 Start with a sentence such as: “On this single local serving setup, how does increasing allowed concurrency affect completion rate and client-perceived delay for one fixed synthetic workload?” That is testable. “Which inference stack is best?” is not a useful first project unless you can control the many differences between stacks.

 Choose one endpoint, model revision and workload family. Write what stays fixed and what varies. If you change precision, hardware, prompt mix and serving version at once, you may observe a difference, but you cannot confidently attribute it to one optimisation.

 vLLM’s benchmark CLI supports a learning/regression exercise; its documentation recommends GuideLLM for production-server benchmarking. Choose the tool that fits the question. [Read the project’s tooling guidance](https://docs.vllm.ai/en/latest/benchmarking/cli/).

## Write a reproducibility contract before running

 Create a run manifest alongside the code. Include the exact model and tokenizer revisions; serving and client versions or commit IDs; OS, driver and accelerator/runtime details; device count and memory; precision/quantisation; context settings; decoding options; prefix-cache policy; endpoint mode; and the benchmark command. Record whether client and server share a machine and whether other workloads were active.

 Use synthetic or legitimately shareable inputs. Save their generation method, seed and a content hash so a reviewer can identify the same workload. A seed is useful evidence, but it does not by itself promise identical outputs across every hardware and software combination. Keep model licensing and any access requirements explicit; this guide does not require buying compute or publishing restricted weights.

 Make a small run matrix, not an uncontrolled search. The values below are chosen project settings, not measured results or recommended production limits .

 Run group What stays fixed What you vary Question 

 Short synthetic baseline Same model, 128 input-token target and 64 output-token budget Concurrency cap 1, then 4 How does this workload react to more outstanding work? 

 Longer-input extension Same model, output budget and load protocol Input-token target 1,024 instead of 128 Does the input-size change alter the latency profile? 

 Cache-control extension Same request records and load protocol Documented cold/reset versus intentional reuse condition Which part of the result depends on cache state? 

 Record actual input/output token counts as well as requested settings. An output budget is not proof that every request generated that many tokens. If a run changes more than the intended variable, separate it rather than pooling it into the comparison.

## Define the metrics and their measurement boundaries

 NVIDIA’s benchmarking guide distinguishes waiting for the first output, receiving the complete response and overall system throughput. It also explains why the measurement window matters. Specify both the units and the start/end boundaries behind each number. [NVIDIA’s primary explanation of inference metrics](https://developer.nvidia.com/blog/llm-benchmarking-fundamental-concepts/).

- Client TTFT: how long the benchmark client waits for the first streamed output. 
- End-to-end latency: the client’s time to receive the completed response under the chosen tool’s definition. 
- TPOT and ITL: keep the tool’s definitions distinct. Current vLLM documentation separates a per-request output-token average from gaps between streamed outputs; speculative decoding can make them differ. 
- Completed requests and output tokens per second: label the aggregation window and denominator; these are not the same as one user’s output speed. Use the actual installed tool’s metric definitions when reading results. Do not paste an ITL figure from one implementation beside a TPOT figure from another and call it an equivalent comparison. [vLLM’s client-side latency definitions](https://docs.vllm.ai/en/latest/benchmarking/cli/).

 Keep attempted, successful, failed and timed-out requests in the report. A fast latency distribution over surviving requests can conceal an unusable failure rate. Percentiles also need a sample count: a high percentile from a tiny run is an observation with limited stability, not a robust service guarantee.

## Follow a five-stage benchmark workflow

 The infographic is an original project checklist. It connects the evidence you need at each stage rather than proposing universal performance targets. Keep the manifest, run IDs and raw output together so every comparison can be traced back to an actual run.

 THE DECISION, AT A GLANCE

### From a workload question to auditable evidence

 Five evidence gates for a bounded inference portfolio project. The workflow contains no invented benchmark scores.

- 01 

### Define the contract

 One question; fixed model, environment, workload and cache policy.

- 02 

### Check a baseline

 Confirm endpoint readiness, valid outputs and the load generator’s behaviour.

- 03 

### Run controlled comparisons

 Vary one condition, repeat and retain successful and failed runs.

- 04 

### Explain the trade-off

 Keep latency, throughput, actual work and errors visible together.

- 05 

### Package the evidence

 Publish shareable inputs, run IDs, raw output, analysis and limitations.

 Read the infographic as text First write the benchmark question and a fixed run contract. Second check a small valid baseline and understand how the client generates load. Third compare one changed condition with repeats while retaining errors. Fourth explain the latency, throughput and correctness trade-off using the actual run evidence. Fifth package commands, shareable input records, raw results and limitations. These are TaskBull editorial evidence gates, not an official benchmark certification or measured results.

 TaskBull Editorial synthesis. [Source 1](https://docs.vllm.ai/en/latest/cli/bench/serve/) · [Source 2](https://docs.vllm.ai/en/latest/benchmarking/cli/) · [Source 3](https://developer.nvidia.com/blog/llm-benchmarking-fundamental-concepts/) · [Source 4](https://docs.pytorch.org/docs/2.15/benchmark_utils.html)

## Run a documented baseline, then vary one condition

 Documented command example—not executed for this article. It assumes an already configured, compatible local completion endpoint and a served model whose identifier you supply. Verify your installed release’s vllm bench serve --help ; the linked latest reference can change. The numeric options are small illustrative project settings, not capacity claims.

 MODEL_ID='your-served-model-id'
vllm bench serve \
 --backend openai --model "$MODEL_ID" \
 --host 127.0.0.1 --port 8000 --endpoint /v1/completions \
 --dataset-name random --random-input-len 128 --random-output-len 64 \
 --num-prompts 100 --seed 7 --request-rate 1 --max-concurrency 1 \
 --num-warmups 5 --percentile-metrics ttft,tpot,itl,e2el \
 --metric-percentiles 50,95,99 \
 --save-result --save-detailed --result-dir ./results/c1 The CLI distinguishes offered request rate from the cap on outstanding work. Holding the rate at one request per second while raising the cap may leave the cap unused if responses finish quickly; label that outcome honestly. For a saturation experiment, design and record a separate load protocol instead of claiming the cap alone creates four simultaneous users. [Current benchmark CLI arguments](https://docs.vllm.ai/en/latest/cli/bench/serve/).

 Save each repetition into a new result directory with its run ID. Check readiness and outputs before increasing load. Stop on unexpected failures or resource limits, retain failed runs and diagnose the cause. Do not run stress tests against someone else’s public endpoint or infer that a documented example was validated on your hardware.

## Control cache state, warmup and repeated runs

 Prefix caching can reuse work for a shared prompt prefix. vLLM’s feature documentation places that benefit in prompt processing rather than the generation of new output tokens. A repeated-prompt experiment can therefore ask a different question from a fresh-prompt experiment. [How automatic prefix caching works](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/).

 Define the cache condition for every group, including warmup. Restart or reset the test server between uncached comparison runs using your installed version’s documented procedure; do not assume a fixed random seed produces a cold workload. If reuse is intentional, label it and explain why the shared-prefix pattern fits the question. Current vLLM benchmarking guidance explicitly warns about results inflated by cache reuse. [Read the repeated-run cache warning](https://docs.vllm.ai/en/latest/benchmarking/cli/).

 Separate initialisation from the steady-state condition you intend to study, and repeat measured runs. If investigating local PyTorch operations, its Timer handles warmup and accelerator synchronisation and emphasises replicates. Such local timing can help diagnose a component; it is not a replacement for client-observed endpoint timing. [PyTorch’s timing and replication guidance](https://docs.pytorch.org/docs/2.15/benchmark_utils.html).

 Choose a repeat protocol before seeing the results. Preserve every run, report variation and investigate outliers rather than removing inconvenient records without explanation. Keep diagnostics with extra instrumentation distinct from headline runs if they change the measurement conditions.

## Build an honest result table and comparison

 Start with the empty template below. A dash means no measurement has been made; it does not mean zero latency, no errors or zero throughput. Replace the dashes only with results from your own retained runs.

 Run ID Attempts / success / failures Client TTFT p50 / p95 (ms) Completion throughput (requests/s) Cache condition / evidence 

 baseline-repeat-1 — / — / — — / — — To be recorded 

 variant-repeat-1 — / — / — — / — — To be recorded 

 Write the conclusion in terms of the actual trade-off. If completed throughput rises but tail latency worsens, show both. If shorter outputs explain a faster result, do not attribute the whole change to the serving engine. If failures increase, do not quietly compare successful-request averages as though the workload completed equally well.

 Add a small quality check with expected outputs for your task, separate from the random performance workload. Performance alone does not show that a changed model or precision setting preserves useful answers. MLPerf’s official suite has specified scenarios, metrics and quality targets; this personal project is not an MLPerf-compliant or validated result. [MLCommons’ benchmark definitions and scope](https://mlcommons.org/benchmarks/inference-datacenter/).

## Turn the experiment into a portfolio evidence packet

 A reviewer should be able to find the following without guessing:

- The decision and boundary: one paragraph describing the workload, user concern, controlled variable and what the experiment cannot answer. 
- The run contract: exact environment, model/configuration, shareable input method, cache policy and executable commands. 
- The evidence: immutable run IDs, raw output, success/failure accounting, actual token counts and the script or notebook used to summarise them. 
- The analysis: a small labelled plot or table showing the latency/throughput trade-off, repeat variation and any quality check. 
- The limitation and next experiment: what you would test before applying the result to another workload, hardware setup or production service. If you do not have access to compatible serving hardware, publish the experiment design and a clearly labelled synthetic analysis exercise. Keep the measurement fields empty and explain what remains unrun. That is useful planning evidence, but it must not be presented as hands-on serving performance.

 When comparing vacancies, use this packet to explain the work you actually did. [Browse inference-engineering roles](https://taskbull.si/AIJobs/inference-engineer-jobs/) and read each employer’s requirements; a portfolio does not replace work-authorisation, seniority or role-specific criteria. For a broader delivery comparison, see [forward deployed versus applied AI engineering](https://taskbull.si/AIJobs/guides/forward-deployed-vs-applied-ai-engineer/).

 RESEARCH YOU CAN FOLLOW

## Sources & editorial notes

 Primary sources checked 2026-11-02. 

- [vLLM: bench serve CLI reference](https://docs.vllm.ai/en/latest/cli/bench/serve/) Current arguments; pin and check the installed version.

- [vLLM: Benchmark CLI](https://docs.vllm.ai/en/latest/benchmarking/cli/) Tool scope, metric boundaries, cache warning and alternative production tooling.

- [NVIDIA: LLM Inference Benchmarking—Fundamental Concepts](https://developer.nvidia.com/blog/llm-benchmarking-fundamental-concepts/) Original April 2, 2025 explanation of latency, throughput and workload parameters.

- [vLLM: Automatic Prefix Caching](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/) Shared-prefix computation and its limits.

- [PyTorch 2.15: Benchmark utilities](https://docs.pytorch.org/docs/2.15/benchmark_utils.html) Local timing, warmup, synchronisation and replication.

- [MLCommons: MLPerf Inference Datacenter](https://mlcommons.org/benchmarks/inference-datacenter/) Official scenarios, quality targets and submission context; this tutorial is not a validated submission.

 This is an original project-design tutorial researched on November 2, 2026. The serving command was checked against documentation but not run here. Numeric workload settings are illustrative choices; result cells are intentionally unmeasured. No GPU performance, production capacity, employer endorsement or hiring outcome is claimed.

 YOUR NEXT STEP

## Explore inference engineering work

 Read the role requirements, location eligibility and original employer source before applying.

[Browse inference roles ↗](https://taskbull.si/AIJobs/inference-engineer-jobs/) 

 PART OF TASKBULL AIJOBS

## Understand the work.

 Explore the specialist collection behind this guide.

[Inference engineering jobs ↗](https://taskbull.si/AIJobs/inference-engineer-jobs/)[All guides ↗](https://taskbull.si/AIJobs/guides/)[Applied AI jobs ↗](https://taskbull.si/AIJobs/applied-ai-jobs/)