Compare local LLM serving setups with traceable benchmark receipts
At a glance
Alex Ellis uses RigMark to replace isolated token-speed claims with disclosed workloads, settings, and saved result receipts. This workflow is listed under Local AI, Benchmarking, Reproducibility.
SourceBuilding a tool for these workflows? See weekly rates.
What they did
1) Prepare Python 3.10+, Git, and a quiet OpenAI-compatible model endpoint. Clone https://github.com/alexellis/rigmark, enter the checkout, run ./rigmark check, then ./rigmark configure to fill metadata.json with your actual hardware and serving configuration. 2) Run ./rigmark bench --base-url http://SERVER:8000 --model auto --label my-appliance --comparison-id sweep-1 --metadata metadata.json, replacing the endpoint. Supply authentication through OPENAI_API_KEY if required. 3) Inspect the generated results/LABEL-TIMESTAMP.json and matching .card.txt, including output gates and incomplete or unverified phases. Keep the same comparison ID and request settings for the next setup. 4) Run ./rigmark compare --card results/setup-a.json results/setup-b.json. Resolve reported mismatches before claiming a matched comparison, and retain both receipts with the comparison card.
Worth knowing
For servers without vLLM's /tokenize or token-ID completion input, add --skip-prefill and label the suite incomplete. Output gates do not prove code correctness or reasoning quality; capped concurrency throughput does not measure completed agent tasks.