--- name: inference description: Set inference service-level targets before serving, then measure time to first token, inter-token latency, tokens per second, throughput and error rate at the expected concurrency, size capacity with Little's law, and work out cost per million tokens from the person's own hourly cost. Use when a task sets, measures or checks inference latency, throughput, error rates, capacity or cost per token. license: MIT metadata: version: "1.0" homepage: https://inference.md api: https://inference.md/api/v1 (optional, no key, stores nothing) terms: https://inference.md/terms --- # Inference Provided as is, at your own risk: figures are estimates from the numbers you use and may be wrong, and nothing here is engineering, capacity-planning, financial or legal advice. Terms: https://inference.md/terms. Write the targets down before you serve or test anything. Measure at the load you expect, after a warm-up, and report pass or fail per target with the margin. Never test an endpoint the person doesn't own or have written permission to test. When this file has a "Your targets" or "Your test rules" section, those win over the defaults below. Settings you drafted or changed yourself count only after the person says yes. ## 1. Set targets first - Agree the targets with the person before any test: time to first token (TTFT) at p50 and p95, inter-token latency (ITL) at p95, tokens per second per stream, total throughput, error rate, and the concurrency they must hold at. - Use percentiles, not averages. A good mean can hide a slow tail. - Write down the request shape the targets apply to: average input tokens, max output tokens, and whether responses stream. - If the person has no targets, offer a preset from https://inference.md (interactive chat, agent steps, offline batch, long context) and wait for a yes. ## 2. Get permission before any test - Test only endpoints the person owns, or has written permission to test. Being able to reach an endpoint is not permission. - Read the endpoint's terms and rate limits first, and stay inside them. Never try to get around a rate limit, a quota or an abuse control. - Work out the worst-case cost before you start: requests x (input tokens + max output tokens) x the person's own per-token price, or test time x their hourly cost. If it is over the test spend cap, stop and ask. - Start small (the default is 20 requests at concurrency 2). Ask before going above the request or concurrency caps. - Never put an API key in a prompt, a log, a report or a URL. Read it from the environment. ## 3. Measure - Warm up first: send a few requests and leave them out of the results. Cold starts and cache fills skew the first numbers. - Stream every request and record, per request: TTFT (send to first token), each gap between tokens (ITL), output tokens, total time, and whether it failed. - Tokens per second per stream = (output tokens - 1) / (time of last token - time of first token). Total throughput = all output tokens / wall-clock time of the measured phase. - Count as errors: non-success status codes, network errors, timeouts and streams that end early. - Measure at the concurrency you plan to serve. Results at concurrency 2 say little about concurrency 64. - Keep prompts and output lengths the same across runs you compare, and note the time of day. - Optional local probe, run on the person's machine only: `https://inference.md/probe/inference-probe.mjs`. Read it before running it. It sends nothing to inference.md. ## 4. Turn samples into numbers - p50, p90, p95 and p99 by linear interpolation between closest ranks. A p95 needs about 100 samples and a p99 about 500 to be more than rough. - Optional tool: `POST https://inference.md/api/v1/latency_stats` with `{ "samples": [212, 240, ...], "drop_first": 2 }`. ## 5. Compare with the targets - For each target: pass or fail, the measured value, and the margin (target minus measured for maximums, measured minus target for minimums). - With few requests, a low error rate proves little: with 0 errors in n requests, the 95% upper bound is about 3 / n (the rule of three). - Never change a target, drop a slow sample or rerun until it passes without telling the person. - Optional tool: `POST https://inference.md/api/v1/slo_check` with `{ "measured": {...}, "targets": {...} }`. The probe report's `measured` block and the settings keys fit it directly. ## 6. Size capacity - Little's law: requests in flight = arrival rate (requests/s) x time each request takes (s). Size for the peak, not the average. - Random arrivals bunch up: plan for the p99 in flight, not the average. - Replicas = the larger of (output tokens/s needed / (measured per-replica tokens/s x utilization)) and (p99 in flight / (measured per-replica concurrency x utilization)), plus spares for failover. Leave headroom: many teams plan for 60 to 80% utilization; the person picks the number. - Batching raises total throughput but usually slows each stream. Check per-stream speed at the batch size you choose. - Autoscale on requests in flight or queue length, not CPU. Scaling up takes time, so keep headroom for that gap. - Optional tool: `POST https://inference.md/api/v1/capacity_plan`. ## 7. Cost per million tokens - Cost per million tokens = hourly cost / (tokens/s x 3600 x utilization) x 1,000,000. Use the person's own all-in hourly cost and measured throughput only. - Break-even utilization against a per-token price the person enters = cost at full load / that price. - Never look up or quote prices yourself. Ask the person for them. - Optional tool: `POST https://inference.md/api/v1/cost_per_mtok`. ## 8. Plan for failure - Set a timeout on every request and retry only idempotent requests, with backoff and a cap. - Have a fallback: a second endpoint the person owns or is allowed to use, a smaller output limit, or a queue. Decide it in advance. - Shed load before the error target breaks: reject or queue over the max concurrency rather than letting every request slow down. ## 9. Report - Six lines at most: the setup (concurrency, requests, request shape), each target with pass or fail and margin, what wasn't checked, and the formula or tool you used. - Say that results are a measurement at one time and load, not a guarantee.