How we measure response time

For a website or an API, that means the time from starting the request to receiving the complete response. For an AI model, it means the time until the first visible word of the answer appears — the moment a person watching a chat window stops waiting.

What we deliberately do not measure:

• • Quality. We ask every model the same trivial question and never judge the answer. A fast wrong answer scores the same as a fast right one.
• • Throughput. We measure the first word, not tokens per second over a long generation. A provider that starts fast can finish slow.
• • Price. Never a factor in any ranking on this site.
• • Uptime. A benchmark run is a snapshot of minutes, not a monitoring record. We report what failed during our run, which is not the same as a service's availability.

Amsterdam, San Francisco, Montreal, Singapore and Tokyo. Each is a rented server in a commercial data centre, and each request in a test is made from the city it is reported under — we never estimate one city's number from another's.

This is a network with excellent connectivity, which is the honest caveat: our Amsterdam number is what a well-connected server in Amsterdam sees, not what someone on mobile data in Amsterdam sees. Treat our numbers as the floor your users' experience is built on. The real thing is always slower.

Within a location we run one or two requests at a time, never a burst, so we are measuring a service rather than stress-testing it. Requests identify themselves honestly in the User-Agent, so anyone can find them in their logs. If you have seen our traffic and want it to stop, that is on the page for site operators.

We make an ordinary GET request and record how long each stage took. The names on the left are what we call them; the names in brackets are what your browser's developer tools call the same thing.

Finding the server (DNS)

Turning the hostname into an IP address.

Reaching the server (TCP)

Opening the connection. This is the part that is mostly distance — the speed of light through fibre sets a floor you cannot buy your way under.

Setting up security (TLS)

Negotiating the encrypted channel.

Waiting for the server (TTFB)

The server thinking. On most pages this is the largest piece, and it is the piece you control.

Receiving the response

Transferring the bytes once they start flowing.

Two details that matter when reading the breakdown chart on a benchmark page. First, we resolve DNS once per location before the requests run, so it sits outside the bars rather than being counted in every request — that is what a real browser does too, after the first visit. Second, the phases shown come from the single median request, so they add up to that request's total. We could show each phase's own worst case instead, but then the parts would not sum to anything real.

We send the same trivial prompt to every model: Say 'ok' and nothing else. The prompt is deliberately near-empty so that what we measure is the service's overhead — connection, queueing, scheduling, model start-up — rather than how long it takes to write an essay.

The request streams, and we parse the stream ourselves, timestamping each event as it decodes. That gives three moments: when the response opened, when the first visible word arrived, and when the answer finished. The headline number is always the middle one.

Only visible text counts. Reasoning models emit internal "thinking" tokens before they say anything a user can read. Those never start the clock. A model that thinks for two seconds and then speaks took two seconds, no matter how much traffic it sent in the meantime — because that is what the person waiting experienced. A call that streams no visible text at all is a failure, never a zero.

Reasoning effort is set to the lowest setting each model offers, and held identical across providers so the comparison is fair. Real workloads at higher effort will be slower than what we publish. We discard one warm-up call per model before the measured ones. We pay for every call with our own metered keys, and we never store the model's output.

Gateways like OpenRouter let several companies serve the same model. Comparing them is only meaningful if you can prove which company actually answered, so we do three things:

• • Pin the provider. Each request names exactly one provider and disables fallbacks. If that provider cannot serve the request, we want the failure, not a silent handoff to someone else.
• • Verify every single request. The gateway reports which upstream served each response. Any request not confirmed as coming from the provider we pinned is thrown out, and if that check fails across a location we withhold the whole cell rather than publish a number under the wrong company's name.
• • Disclose quantization. Providers may run the same model at different numeric precision, which affects both speed and quality. We publish what the gateway's catalogue lists per endpoint on the run date — including "unknown" where it lists nothing.

Why some cells have no ranked number:

Unavailable

The pinned provider errored or streamed nothing from this location. With fallbacks off, that is a real availability result — often more decision-relevant than the milliseconds.

Unverified

We could not confirm the response came from the provider we pinned, so its numbers are withheld.

Greyed-out numbers

A verified measurement from too few answered requests to rank — fewer than five. The value is shown because it is real, but an n=1 "typical" is not a typical. It never decides "fastest".

One distinction we hold to strictly: the network evidence we collect describes the route to the gateway. The upstream provider's name is only what the gateway told us. We never infer where a provider's hardware physically sits from the route to the gateway in front of it.

Typical

Half the requests were faster than this, half slower. The median — what a normal moment looks like. (Technically: p50.)

On a bad day

One request in twenty was slower than this. (Technically: p95.)

Slowest seen

The worst response we actually recorded. Rare, but it happened to someone.

We publish no averages. An average blends the ordinary case and the disaster into a number that describes neither. Nineteen fast requests and one five-second stall average out to something that looks fine, and the user who hit the stall is invisible in it. The pair — typical and bad day — tells you both what usually happens and what your users complain about.

The single headline number on a benchmark card is the middle of the five locations' typical times — a typical-location view. We use it rather than pooling every request together so that one slow city cannot drag the summary, and so that a test which ran more requests in one place is not weighted toward it. It is a summary for scanning a list; the per-location table is the real answer.

Percentiles need requests behind them. A standard benchmark run is 20 requests per location. Below five verified requests we show the value but exclude it from every ranking, and label it as such.

Infrastructure speed moves by hour and by week. A provider that wins our Tuesday run may lose the Thursday one, and if we called that a recommendation we would be publishing noise with a confident face on it.

So a per-location winner on a benchmark page is always labelled fastest this run — an observation with a date attached, not advice. We upgrade that to a standing recommendation only when the same provider wins repeated runs by a margin wider than the run-to-run variation we see. Until then the wording stays hedged on purpose, and we would rather sound less certain than be wrong in a way that costs you a migration.

Failed requests are reported per location and never quietly dropped — a service that answers fast 80% of the time and refuses the rest is not an 80%-as-good service, and averaging that away would hide the most important thing on the page.

When a target returns 401, 403 or 429, we treat it as a finding and stop rather than retrying into it. Two consecutive blocked runs pause that target entirely for a few days.

The honest caveat: a burst of unanswered requests may be a limit on our account rather than anything about the provider's capacity. We cannot always tell the two apart from outside. Where we cannot, we say so on the page and leave the cell unranked instead of publishing a number that would read as a verdict on the provider.

• • One vantage point per city. A single server on a single network. Another host in the same city, on a different network, may see something different.
• • Data-centre network, not consumer network. No mobile radio, no home Wi-Fi, no congested last mile. Real users are slower than our numbers, sometimes much slower.
• • A snapshot, not a trend. Every page carries the date it was measured. A number from a month ago describes a month ago.
• • The front door only. We measure one URL from outside. We cannot see which of your services was slow, and we do not claim to.

One caveat before you compare: run these from your laptop and you are measuring your own connection, not a data centre — see what this can't tell you. Expect your numbers to be slower than ours. What should match is the shape: which stage dominates, and how much the slow requests differ from the typical one.

One request, timed by stage

This is section 3's five phases, straight out of curl.

curl -sS -o /dev/null \
  -w 'dns %{time_namelookup}\nconnect %{time_connect}\ntls %{time_appconnect}\nwait %{time_starttransfer}\ntotal %{time_total}\n' \
  https://example.com/ \
| awk '{t[$1]=$2*1000} END {
    printf "Finding the server     %d ms\n", t["dns"]
    printf "Reaching the server    %d ms\n", t["connect"]-t["dns"]
    printf "Setting up security    %d ms\n", t["tls"]-t["connect"]
    printf "Waiting for the server %d ms\n", t["wait"]-t["tls"]
    printf "Receiving the response %d ms\n", t["total"]-t["wait"]
  }'

Twenty requests, our exact percentiles

We use the nearest-rank method, which is why "slowest seen" equals the worst request at 20 requests rather than an interpolated value. This awk reproduces it exactly.

URL="https://example.com/"

for i in $(seq 1 20); do
  curl -sS -o /dev/null -w '%{time_total}\n' "$URL"
done | sort -n | awk '
  function nearest(p, n,   i) { i = int(p * n); if (i < p * n) i++; return i }
  { v[NR] = $1 * 1000 }
  END {
    printf "typical      %d ms\n", v[nearest(0.50, NR)]
    printf "on a bad day %d ms\n", v[nearest(0.95, NR)]
    printf "slowest seen %d ms\n", v[nearest(0.99, NR)]
  }'

Time to first visible token

The part that matters is which event stops the clock. A reasoning delta never does; neither does an empty content delta. Only real visible text counts, and a stream that produces none is a failure rather than a zero.

import json, os, time, urllib.request

BODY = {
    "model": "gpt-5.6-sol",
    "stream": True,
    "max_completion_tokens": 256,
    "messages": [{"role": "user", "content": "Say 'ok' and nothing else."}],
}

request = urllib.request.Request(
    "https://api.openai.com/v1/chat/completions",
    data=json.dumps(BODY).encode(),
    headers={
        "Authorization": "Bearer " + os.environ["OPENAI_API_KEY"],
        "Content-Type": "application/json",
    },
)

started = time.monotonic()
ttft = None
with urllib.request.urlopen(request) as stream:
    for raw in stream:
        line = raw.decode("utf-8", "replace").strip()
        if not line.startswith("data:"):
            continue
        payload = line[5:].strip()
        if payload == "[DONE]":
            break
        try:
            delta = json.loads(payload)["choices"][0]["delta"]
        except (ValueError, KeyError, IndexError):
            continue
        # Reasoning and empty deltas are not visible output.
        if delta.get("content"):
            ttft = (time.monotonic() - started) * 1000
            break

print(f"first visible token: {ttft:.0f} ms" if ttft else "no visible output — failure")

Run that twenty times, feed the numbers through the awk above, and you have the same two numbers we publish. To run it from five cities instead of one, that is what our API does.