INFERENCE.md

# ABOUT INFERENCE.MD

INFERENCE.md is a working idea: a plain markdown file that says what the model endpoints you run or rent should deliver, and how an AI agent checks. Inference is a model answering a request. The file names each endpoint, sets targets for time to first token, tokens per second, the slow tail at p95 and p99, and errors. It says how to measure them with a probe you run against endpoints you own, and how to work out capacity and cost from your own numbers. It serves no models and publishes no benchmarks.

## WHAT IT COULD COVER

• - endpoints: each one by name, its model, its runtime and what it serves
• - time to first token: how long until the answer starts
• - output speed: tokens per second once it streams
• - the tail: p95 and p99, because an average hides the slow requests
• - errors: timeouts, rate limits and server errors, and the share you accept
• - load: requests at once, and the busiest hour
• - capacity: replicas needed for the peak, with room to spare
• - cost: per million tokens, from your own bill and your own measurements
• - when it slips: shift traffic, shed load, or call a person

## HOW IT COULD WORK

• - the probe: a short script on your own machine sends a fixed set of prompts and records time to first token, total time and tokens out
• - the runtime: serving engines such as vLLM, SGLang and Hugging Face TGI can expose metrics for Prometheus to collect
• - traces: OpenTelemetry spans on each call, so a slow answer can be followed through retrieval, the model and its tools
• - the math: requests in flight equal requests per second times seconds per request, and that sets the replica count
• - the agent: runs the probe on schedule, compares, and proposes a change; a person approves anything that costs more

## TERMS AND TOOLS IT COULD NAME

• - TTFT: time to first token, what a person waiting on a chat feels
• - p95 and p99: the times that 95 and 99 out of 100 requests beat
• - Little's law: items in a system equal arrival rate times time in the system
• - continuous batching: engines add new requests to a running batch, trading each request's speed against total throughput
• - KV cache: memory that holds the context of requests in flight; it often limits how many fit at once
• - Prometheus and Grafana: open-source tools that collect and graph metrics
• - OpenTelemetry: an open standard for traces, metrics and logs, with draft conventions for model calls
• - MLPerf Inference: a benchmark suite from MLCommons; INFERENCE.md points to your own numbers instead

## WHO IT COULD HELP

• - teams running an open-weight model: set targets before launch, and see when a new version is slower
• - platform engineers: size replicas for the peak hour from measured numbers
• - product teams with a chat feature: keep time to first token short enough that people stay
• - agencies running agents for clients: a dated record of what each endpoint delivered
• - finance and operations: cost per million tokens from the real bill and real throughput

An agent waits on a model many times for one task, so the slow tail adds up. If each call has a 1 in 100 chance of being slower than p99, a chain of 20 calls hits at least one of those close to one time in five.

## A SKETCH

```md
---
inference: draft
---
# INFERENCE.md
- endpoint: chat, our 8B model on vLLM, made-up example
- first_token: 0.8 s at p95, 2 s at p99
- speed: 30 tokens per second or more per request
- errors: under 1 in 100 over any hour
- peak: 12 requests per second, 4 pm to 6 pm
- capacity: 3 replicas, 2 carry the peak, 1 spare
- probe: 200 fixed prompts, Sundays 3 am, our endpoints only
- cost: our monthly bill divided by tokens served
- slip: shift to the spare, tell on-call, no retry loops
- scale: inside the 3 replicas alone, any added one needs a yes
- never: load-test an endpoint we do not own
```

## DISCLAIMER

INFERENCE.md does not run, host or test models, and it publishes no benchmarks or vendor comparisons. Numbers in a file like this are targets you set, not measurements or promises from any provider, and the ones in the sketch are made up. The file states targets; your own monitoring, capacity and contracts are what deliver them. Test only endpoints you own or have permission to test, and stay inside each provider's terms and rate limits. Products named here are described for reference only, with no affiliation or endorsement.

INFERENCE.md is an informational page. It is provided as is, without warranty of any kind, and may be incomplete, inaccurate or out of date. Nothing here is financial, investment, legal, tax, insurance, medical, engineering, safety or other professional advice, or an offer or solicitation of any kind. The sketch is an illustration, not a standard or a recommendation. Names of third-party products or companies, if any, are used only to describe them; no affiliation or endorsement is implied. Use at your own risk. This page sets no cookies and runs no scripts or analytics; like any website, its host keeps standard request logs.