LLM Tool Rate Limiting: Per-Client Limits for MCP Servers

At the MCP server that runs the tools, keyed by who is calling. In Promptise Foundry, add AuthMiddleware first, so every call has a caller, then RateLimitMiddleware with a token bucket. Give an expensive tool its own, stricter limit on the decorator:

# Who is calling: one key per customer's agent. Auth runs first, so the rate limiter knows the caller.
server.add_middleware(
    AuthMiddleware(
        APIKeyAuth(
            keys={
                os.environ["ACME_KEY"]: {"client_id": "acme-bot", "tenant_id": "acme"},
                os.environ["GLOBEX_KEY"]: {"client_id": "globex-bot", "tenant_id": "globex"},
                os.environ["INITECH_KEY"]: {"client_id": "initech-bot", "tenant_id": "initech"},
            }
        )
    )
)
# How often: each client gets its own bucket of 20 calls, refilled at 120 calls a minute.
server.add_middleware(RateLimitMiddleware(TokenBucketLimiter(rate_per_minute=120, burst=20)))


@server.tool(read_only_hint=True, rate_limit="5/min")
async def get_sales(sku: str) -> dict:
    """Units sold in the last 30 days for one product. Runs a data warehouse query."""

Every client now has its own budget. In the load test below, a client stuck in a loop sent over 3,000 calls in ten seconds and got 39 answered, while two other clients sharing the server didn't lose a single call:

acme-bot        10 calls    10 ok      0 rate limited  median  36.6 ms
globex-bot      10 calls    10 ok      0 rate limited  median  37.4 ms
initech-bot   3030 calls    39 ok   2991 rate limited  median   9.3 ms

The rest of the guide builds this server, breaks it in the two ways that quietly turn per-client limits into one shared limit, and shows how agents and scripts should react to a refusal. Caching, timeouts and circuit breakers, the other half of protecting a server, are in the MCP server caching and circuit breaker guide.

A token bucket holds up to burst tokens and refills at a steady rate. Every call takes one token; a call that finds the bucket empty is refused, and told how long until the next token arrives. So TokenBucketLimiter(rate_per_minute=120, burst=20) lets a client make 20 calls at once, then two a second. A declared rate_limit="5/min" is a bucket of 5 that gains one token every 12 seconds. A refused call doesn't take a token, so a client that keeps hammering still gets its steady rate and nothing more.

Promptise has three kinds of limit, and they answer different questions:

The key decides whose budget a call spends. By default it's the tenant plus the client_id, and both come from authentication. A call without an identified caller is keyed as global, and every such call shares that one bucket. That single rule is behind both mistakes in step 4.

• Python 3.10 or newer.

• Promptise Foundry 1.3.3 or newer from PyPI. Since 1.3.3, the server warns at startup when per-client limits would collapse into one shared bucket (step 4).

• An OpenAI API key, for the agent in the last part. Other providers work by changing the model string; see Models & Providers.

pip install "promptise>=1.3.3"
export OPENAI_API_KEY="sk-..."

The example is a product catalog that three customers' agents share: acme, globex and initech. One tool is cheap, the other runs a data warehouse query that you don't want called more than a few times a minute.

import asyncio
import os

from promptise.mcp.server import APIKeyAuth, AuthMiddleware, MCPServer, RateLimitMiddleware, TokenBucketLimiter

server = MCPServer("catalog", require_auth=True)  # every tool needs a caller identity

# Who is calling: one key per customer's agent. Auth runs first, so the rate limiter knows the caller.
server.add_middleware(
    AuthMiddleware(
        APIKeyAuth(
            keys={
                os.environ["ACME_KEY"]: {"client_id": "acme-bot", "tenant_id": "acme"},
                os.environ["GLOBEX_KEY"]: {"client_id": "globex-bot", "tenant_id": "globex"},
                os.environ["INITECH_KEY"]: {"client_id": "initech-bot", "tenant_id": "initech"},
            }
        )
    )
)
# How often: each client gets its own bucket of 20 calls, refilled at 120 calls a minute.
server.add_middleware(RateLimitMiddleware(TokenBucketLimiter(rate_per_minute=120, burst=20)))

STOCK = {f"SKU-{n}": (n * 37) % 120 for n in range(1, 101)}
SALES = {f"SKU-{n}": (n * 53) % 40 for n in range(1, 101)}


@server.tool(read_only_hint=True)
async def get_stock(sku: str) -> dict:
    """Units on hand for one product, for example "SKU-7"."""
    await asyncio.sleep(0.02)  # a database lookup
    return {"sku": sku, "on_hand": STOCK.get(sku, 0)}


@server.tool(read_only_hint=True, rate_limit="5/min")
async def get_sales(sku: str) -> dict:
    """Units sold in the last 30 days for one product. Runs a data warehouse query."""
    await asyncio.sleep(0.5)  # an expensive query
    return {"sku": sku, "sold_30d": SALES.get(sku, 0)}


if __name__ == "__main__":
    server.run(transport="http", host="127.0.0.1", port=9460)

• Auth first, then the limiter. Middleware runs in the order you add it, so AuthMiddleware has set the caller before RateLimitMiddleware picks a bucket.

• `require_auth=True` turns on authentication for every tool and refuses requests without a valid key with 401. Without it, AuthMiddleware only identifies callers of tools declared with auth=True.

• `tenant_id` on each key puts the customer into the bucket key, so two customers can never share a bucket, even if their client IDs collide. With JWTs, the tenant comes from a claim instead; see Multi-Tenancy.

• `rate_limit="5/min"` gives get_sales its own bucket per client, on top of the server-wide one. Promptise adds the middleware that enforces it when the server starts, and a typo such as "5/mins" raises ValueError at registration instead of silently never limiting.

export ACME_KEY="test-acme-key" GLOBEX_KEY="test-globex-key" INITECH_KEY="test-initech-key"
python catalog_server.py

A load test should look like your real traffic. Here, acme and globex behave: one lookup a second each. initech's agent is stuck in a loop: four loops calling as fast as the server answers. All three run at the same time, for ten seconds, against the server over HTTP:

"""Three customers' agents share one MCP server. initech's agent is stuck in a loop."""
import asyncio
import json
import os
import statistics
import time
from collections import Counter

from promptise.mcp.client import MCPClient

URL = "http://127.0.0.1:9460/mcp"
SECONDS = 10


def outcome(result) -> str:
    return json.loads(result.content[0].text)["error"]["code"] if result.isError else "ok"


async def steady(name, results, latency):
    """A well-behaved agent: one lookup a second."""
    async with MCPClient(url=URL, api_key=os.environ[f"{name.upper()}_KEY"]) as client:
        end, n = time.monotonic() + SECONDS, 0
        while time.monotonic() < end:
            start = time.perf_counter()
            result = await client.call_tool("get_stock", {"sku": f"SKU-{n % 100 + 1}"})
            latency[name].append(time.perf_counter() - start)
            results[name][outcome(result)] += 1
            n += 1
            await asyncio.sleep(1)


async def runaway(name, results, latency, loops=4):
    """A stuck agent: four loops calling the same tool as fast as the server answers."""
    async with MCPClient(url=URL, api_key=os.environ[f"{name.upper()}_KEY"]) as client:
        end = time.monotonic() + SECONDS

        async def loop():
            while time.monotonic() < end:
                start = time.perf_counter()
                result = await client.call_tool("get_stock", {"sku": "SKU-7"})
                latency[name].append(time.perf_counter() - start)
                results[name][outcome(result)] += 1

        await asyncio.gather(*(loop() for _ in range(loops)))


async def main():
    names = ("acme", "globex", "initech")
    results = {name: Counter() for name in names}
    latency = {name: [] for name in names}
    await asyncio.gather(steady("acme", results, latency), steady("globex", results, latency), runaway("initech", results, latency))
    for name in names:
        counts = results[name]
        print(
            f"{name + '-bot':12} {sum(counts.values()):5} calls  {counts['ok']:4} ok  "
            f"{counts['RATE_LIMIT_EXCEEDED']:5} rate limited  median {statistics.median(latency[name]) * 1000:5.1f} ms"
        )


asyncio.run(main())

acme-bot        10 calls    10 ok      0 rate limited  median  36.6 ms
globex-bot      10 calls    10 ok      0 rate limited  median  37.4 ms
initech-bot   3030 calls    39 ok   2991 rate limited  median   9.3 ms

initech got 39 answers: the burst of 20, plus two a second for ten seconds. Its other 2,991 calls were refused before they reached the tool, which is why its median is 9.3 ms. acme and globex got every answer, at their usual latency. Three runs gave the same picture: initech got 39 answers each time, from 1,982 to 3,312 calls, and acme and globex got 10 out of 10 every time, with medians between 36.6 and 46.6 ms. The laptop was busy with other work during these runs, which is why the runaway client managed only a few thousand calls.

There's no HTTP 429 here. An MCP tool call is a JSON-RPC request, and a refusal is a tool result with isError: true. This script sends six calls to get_sales over plain HTTP and prints the sixth:

"""Six calls to get_sales over plain HTTP. What does the sixth one look like on the wire?"""
import os

import httpx

URL = "http://127.0.0.1:9460/mcp"
headers = {"x-api-key": os.environ["ACME_KEY"], "accept": "application/json, text/event-stream"}

with httpx.Client(headers=headers, timeout=30) as http:
    init = http.post(URL, json={"jsonrpc": "2.0", "id": 0, "method": "initialize", "params": {
        "protocolVersion": "2025-06-18", "capabilities": {}, "clientInfo": {"name": "raw", "version": "0"}}})
    http.headers["mcp-session-id"] = init.headers["mcp-session-id"]
    http.post(URL, json={"jsonrpc": "2.0", "method": "notifications/initialized"})
    for n in range(1, 7):
        response = http.post(URL, json={"jsonrpc": "2.0", "id": n, "method": "tools/call",
                                        "params": {"name": "get_sales", "arguments": {"sku": f"SKU-{n}"}}})
    print("HTTP", response.status_code, "| Retry-After header:", response.headers.get("retry-after"))
    print(response.text.strip())

HTTP 200 | Retry-After header: None
event: message
data: {"jsonrpc":"2.0","id":6,"result":{"content":[{"type":"text","text":"{\n  \"error\": {\n    \"code\": \"RATE_LIMIT_EXCEEDED\",\n    \"message\": \"Rate limit exceeded for tool 'get_sales' (declared limit: 5/min)\",\n    \"retryable\": true,\n    \"suggestion\": \"Wait 10 seconds before retrying.\",\n    \"details\": {\n      \"retry_after_seconds\": 9.019\n    }\n  }\n}"}],"isError":true}}

The HTTP status is 200 and there's no Retry-After header. Everything a client needs is in the error envelope: the code RATE_LIMIT_EXCEEDED, retryable: true, a suggestion rounded up to whole seconds, and the exact retry_after_seconds. So a gateway or a client library that only looks at status codes won't notice rate limiting at all. With Promptise's MCPClient, result.isError is True and the envelope is the text of result.content[0].

Swap the two add_middleware calls, so the limiter runs before authentication:

# WRONG ORDER: the limiter runs before auth, so it can't tell clients apart.
# How often: each client gets its own bucket of 20 calls, refilled at 120 calls a minute.
server.add_middleware(RateLimitMiddleware(TokenBucketLimiter(rate_per_minute=120, burst=20)))
# Who is calling: one key per customer's agent. Auth runs first, so the rate limiter knows the caller.
server.add_middleware(
    AuthMiddleware(
        APIKeyAuth(
            keys={
                os.environ["ACME_KEY"]: {"client_id": "acme-bot", "tenant_id": "acme"},
                os.environ["GLOBEX_KEY"]: {"client_id": "globex-bot", "tenant_id": "globex"},
                os.environ["INITECH_KEY"]: {"client_id": "initech-bot", "tenant_id": "initech"},
            }
        )
    )
)

Nothing fails, and the server still starts. Since 1.3.3, it logs one warning at startup that names the tools affected:

MCPServer 'catalog': RateLimitMiddleware runs before AuthMiddleware for tools ['get_sales', 'get_stock']: no caller is identified yet when it picks a bucket, so every caller shares one 'global' bucket. Add AuthMiddleware first (middleware runs in the order it was added), or pass key_func= to RateLimitMiddleware if one shared bucket is intended.

Earlier versions said nothing. Here is what the warning is about, from the same load test:

acme-bot        10 calls     2 ok      8 rate limited  median  12.8 ms
globex-bot      10 calls     1 ok      9 rate limited  median  13.7 ms
initech-bot   3561 calls    36 ok   3525 rate limited  median   7.6 ms

When the limiter runs, no caller has been identified yet, so every call goes into the one global bucket. initech's loop empties it, and acme and globex got 1 or 2 answers out of 10 each, in all three runs. With the middleware in the right order, as in step 1, the server logs no such warning.

The same thing happens with the right order if a tool isn't authenticated: on a server without require_auth=True, a tool declared without auth=True is never authenticated, so it has no client_id either. That case gets its own warning, on the first call through TestClient or at startup over HTTP:

MCPServer 'catalog': RateLimitMiddleware cannot tell callers of tools ['get_stock'] apart: they are not auth=True (and the server is not require_auth=True), so AuthMiddleware does not identify the caller and every caller shares one 'global' bucket. Mark them auth=True (or use MCPServer(require_auth=True)) to rate-limit each client separately, or pass key_func= to RateLimitMiddleware to choose the key yourself.

A check with two API keys and a bucket of three showed it: the second key's first call was refused after the first key's three. Two cases stay quiet on purpose. A key_func tells the server the key is your choice, so it never warns; and a server with no AuthMiddleware at all has no callers to tell apart. So read your startup logs, and if your limits are meant to be per client, still test them with two clients.

A script or a backend job that calls tools directly should honour retry_after_seconds. This helper retries only what the server marks retryable, waits as long as it's told, and gives up after a few tries:

import asyncio
import json
import random


async def call_with_retry(client, tool: str, args: dict, attempts: int = 6):
    """Call an MCP tool. On a retryable error, wait as long as the server asks, then try again."""
    for attempt in range(1, attempts + 1):
        result = await client.call_tool(tool, args)
        if not result.isError or attempt == attempts:
            return result
        try:
            error = json.loads(result.content[0].text)["error"]
        except (ValueError, KeyError, TypeError):
            return result  # not a Promptise error envelope: don't guess
        if not error.get("retryable"):
            return result
        wait = error.get("details", {}).get("retry_after_seconds", 0.5 * 2**attempt)
        await asyncio.sleep(wait + random.uniform(0, 0.2))  # a little jitter, so callers don't line up

To see what it changes, a nightly job looked up 60 products with initech's key, three ways, each against a freshly started server: plain calls in a loop, the same loop with call_with_retry, and all 60 at once with asyncio.gather and call_with_retry:

"""A nightly job looks up 60 products. How it calls decides how many calls it wastes."""
import asyncio
import json
import os
import sys
import time

from promptise.mcp.client import MCPClient

from retry import call_with_retry

SKUS = [f"SKU-{n}" for n in range(1, 61)]
MODE = sys.argv[1]  # "plain", "retry" or "retry-parallel"


class Counting:
    """Wraps the client to count every call that reaches the server."""

    def __init__(self, client):
        self.client, self.calls = client, 0

    async def call_tool(self, name, args):
        self.calls += 1
        return await self.client.call_tool(name, args)


async def main():
    async with MCPClient(url="http://127.0.0.1:9460/mcp", api_key=os.environ["INITECH_KEY"]) as client:
        counting = Counting(client)
        start = time.perf_counter()
        if MODE == "plain":
            results = [await counting.call_tool("get_stock", {"sku": sku}) for sku in SKUS]
        elif MODE == "retry":
            results = [await call_with_retry(counting, "get_stock", {"sku": sku}) for sku in SKUS]
        else:
            results = await asyncio.gather(*(call_with_retry(counting, "get_stock", {"sku": sku}) for sku in SKUS))
        ok = sum(not r.isError for r in results)
        print(f"{MODE:15} {ok}/60 answered  {counting.calls:4} calls sent  {time.perf_counter() - start:5.1f} s")


asyncio.run(main())

python batch_sync.py plain    # and: retry, retry-parallel

plain           21/60 answered    60 calls sent    0.7 s
retry           60/60 answered    97 calls sent   20.2 s
retry-parallel  25/60 answered   250 calls sent    2.7 s

The loop with retries got every answer: the burst first, then two a second, as the server asked. It sent 97 calls for 60 answers: after the burst, almost every product took one refused call and one that waited as told. To skip the refused calls, pace yourself to the limit. The parallel version did worse than no retries at all. All 40 refused calls were told the same thing, about half a second, so they all came back at the same moment, one of them won, and the rest were refused again, six times over: 250 calls for 25 answers. retry_after_seconds tells one caller when its next token arrives. If you run many callers on one key, queue them, or pace them to the limit yourself.

Per client is the default, but a customer with three agents then gets three budgets. To give each customer one budget, sized by the plan it pays for, pass a key_func and a limiter of your own. RateLimitMiddleware hands the limiter only the key, so the key carries the plan:

import json

from promptise.mcp.server import RequestContext, TokenBucketLimiter


class PlanLimiter:
    """One bucket size per plan. RateLimitMiddleware hands the limiter only the key, so the key names the plan."""

    def __init__(self, plans: dict[str, TokenBucketLimiter]):
        self.plans = plans

    def consume(self, key: str) -> tuple[bool, float]:
        plan = json.loads(key)[0]
        return self.plans[plan].consume(key)


def tenant_plan_key(ctx: RequestContext) -> str:
    """One budget per customer, shared by all of its agents, sized by its plan."""
    plan = "pro" if ctx.client.has_role("pro") else "free"
    return json.dumps([plan, ctx.client.tenant_id or ctx.client_id])

To compare the two, plans_demo.py builds the catalog server in process with three API keys, two acme agents on the pro plan and one globex agent on the free plan, and sends 40 quick calls from each through TestClient. It does that twice, with these two limiters:

print("Per client, 20 each (the default key):")
asyncio.run(burst(make_server(RateLimitMiddleware(TokenBucketLimiter(rate_per_minute=120, burst=20)))))
print("Per customer, sized by plan:")
plans = {"free": TokenBucketLimiter(rate_per_minute=30, burst=5), "pro": TokenBucketLimiter(rate_per_minute=600, burst=50)}
asyncio.run(burst(make_server(RateLimitMiddleware(PlanLimiter(plans), key_func=tenant_plan_key))))

Per client, 20 each (the default key):
  acme-bot      (acme, pro): 20/40 answered
  acme-reports  (acme, pro): 20/40 answered
  globex-bot    (globex, free): 20/40 answered
Per customer, sized by plan:
  acme-bot      (acme, pro): 40/40 answered
  acme-reports  (acme, pro): 10/40 answered
  globex-bot    (globex, free):  5/40 answered

With per-client keys, every agent got its own 20. With the customer key, acme's two agents shared the pro bucket of 50, and globex got the free plan's 5. Any object with a consume(key) method that returns (allowed, retry_after_seconds) works as a limiter.

Rate limits count calls over time. A slow tool needs a limit on calls in flight too, and Promptise's max_concurrent and ConcurrencyLimiter are shared by every client: one client can hold every slot. A small middleware gives each client its own cap:

from collections import defaultdict

from promptise.mcp.server import ConcurrencyLimitError, RequestContext


class PerClientConcurrency:
    """At most `limit` calls in flight per client, so one client can't take every slot."""

    def __init__(self, limit: int):
        self.limit = limit
        self.in_flight: dict[tuple, int] = defaultdict(int)

    async def __call__(self, ctx: RequestContext, call_next):
        key = (ctx.client.tenant_id, ctx.client_id)
        if self.in_flight[key] >= self.limit:
            raise ConcurrencyLimitError(f"{ctx.client_id} already has {self.limit} calls in flight")
        self.in_flight[key] += 1
        try:
            return await call_next(ctx)
        finally:
            self.in_flight[key] -= 1

Add it after AuthMiddleware. Here, initech starts six one-second exports on a tool with max_concurrent=4, and acme starts one a tenth of a second later:

max_concurrent=4 on the tool:
  initech-bot: ['CONCURRENCY_LIMIT_EXCEEDED', 'CONCURRENCY_LIMIT_EXCEEDED', 'ok', 'ok', 'ok', 'ok']
  acme-bot:    CONCURRENCY_LIMIT_EXCEEDED
plus PerClientConcurrency(limit=2):
  initech-bot: ['CONCURRENCY_LIMIT_EXCEEDED', 'CONCURRENCY_LIMIT_EXCEEDED', 'CONCURRENCY_LIMIT_EXCEEDED', 'CONCURRENCY_LIMIT_EXCEEDED', 'ok', 'ok']
  acme-bot:    ok

With only max_concurrent, initech took all four slots and acme was turned away. With a cap of two per client, initech got two and acme got its export. Both refusals use CONCURRENCY_LIMIT_EXCEEDED, so a caller can tell "busy, try again in a moment" from "over quota". The per-tool limit itself is covered in the caching and circuit breaker guide.

A script can sleep. An agent can only decide what to do next. This agent gets a question that needs eight calls to get_sales, whose limit is 5 a minute:

import asyncio
import os
import sys

from promptise import build_agent
from promptise.config import HTTPServerSpec

QUESTION = "Which of SKU-11 to SKU-18 sold fewer than 10 units in the last 30 days?"
INSTRUCTIONS = "You answer inventory questions for the warehouse team. Use the catalog tools."
RATE_LIMIT_RULE = (
    " If a tool answers RATE_LIMIT_EXCEEDED, don't call it again in this turn. Answer with what you have,"
    " name the products you couldn't check, and say when to ask again."
)


async def main(careful: bool):
    agent = await build_agent(
        model="openai:gpt-5-mini",
        servers={"catalog": HTTPServerSpec(url="http://127.0.0.1:9460/mcp", api_key=os.environ["ACME_KEY"])},
        instructions=INSTRUCTIONS + (RATE_LIMIT_RULE if careful else ""),
        trace_tools=True,
    )
    try:
        result = await agent.ainvoke({"messages": [{"role": "user", "content": QUESTION}]})
        print(">>>", result["messages"][-1].content)
    finally:
        await agent.shutdown()


asyncio.run(main(careful="--careful" in sys.argv))

python restock_agent.py

→ Invoking tool: get_sales with {'sku': 'SKU-11'}
→ Invoking tool: get_sales with {'sku': 'SKU-12'}
→ Invoking tool: get_sales with {'sku': 'SKU-13'}
→ Invoking tool: get_sales with {'sku': 'SKU-14'}
→ Invoking tool: get_sales with {'sku': 'SKU-15'}
→ Invoking tool: get_sales with {'sku': 'SKU-16'}
→ Invoking tool: get_sales with {'sku': 'SKU-17'}
→ Invoking tool: get_sales with {'sku': 'SKU-18'}
✖ get_sales error: RATE_LIMIT_EXCEEDED: Rate limit exceeded for tool 'get_sales' (declared limit: 5/min)
✖ get_sales error: RATE_LIMIT_EXCEEDED: Rate limit exceeded for tool 'get_sales' (declared limit: 5/min)
✖ get_sales error: RATE_LIMIT_EXCEEDED: Rate limit exceeded for tool 'get_sales' (declared limit: 5/min)
✔ Tool result from get_sales: {"sku": "SKU-11", "sold_30d": 23}
✔ Tool result from get_sales: {"sku": "SKU-12", "sold_30d": 36}
✔ Tool result from get_sales: {"sku": "SKU-13", "sold_30d": 9}
✔ Tool result from get_sales: {"sku": "SKU-14", "sold_30d": 22}
✔ Tool result from get_sales: {"sku": "SKU-15", "sold_30d": 35}
→ Invoking tool: get_sales with {'sku': 'SKU-16'}
→ Invoking tool: get_sales with {'sku': 'SKU-17'}
→ Invoking tool: get_sales with {'sku': 'SKU-18'}
✖ get_sales error: RATE_LIMIT_EXCEEDED: Rate limit exceeded for tool 'get_sales' (declared limit: 5/min)
✖ get_sales error: RATE_LIMIT_EXCEEDED: Rate limit exceeded for tool 'get_sales' (declared limit: 5/min)
✖ get_sales error: RATE_LIMIT_EXCEEDED: Rate limit exceeded for tool 'get_sales' (declared limit: 5/min)
>>> Only SKU-13 sold fewer than 10 units (9 units in the last 30 days).

Known 30-day sales:
- SKU-11: 23
- SKU-12: 36
- SKU-13: 9
- SKU-14: 22
- SKU-15: 35

Sales for SKU-16, SKU-17, and SKU-18 were not retrieved due to a rate-limit error. Would you like me to retry those now?

The model asked for all eight at once, five were answered, and three were refused. The trace shortens the error, but the model gets the whole envelope: in a separate run, the refused call's ToolMessage had status error and the JSON from step 3, with "suggestion": "Wait 12 seconds before retrying.". The model retried the three in its next step, and they were refused again: a new token takes 12 seconds. Then it answered with what it had, named the products it couldn't check and offered to retry. Its first line still answers as if the list were complete: SKU-16, which it never saw, sold 8.

The retry wasted three tool calls and a model call, so the rule in RATE_LIMIT_RULE tells the model not to retry within the turn. Each version ran ten times, every run against a freshly started server, so every run started with full buckets:

Nine of ten default runs retried, one of them twice, and no retry ever succeeded. The rule removed the retries, one model call and about 37% of the tokens per run, without changing what the agent found, and every answer with the rule told the user to ask again in about 12 seconds. Both versions were honest about the gap: every answer named the three products it couldn't check, and none made up numbers for them. Most answers also framed the result as partial ("From the data I have", "Of the SKUs checked"), but one default answer opened with "Answer: SKU-13 only.", and the run shown above opened with "Only SKU-13 sold fewer than 10 units", as did a separate run with the rule. That's wrong, and a reader who stops at the first line won't see why. The 20 runs cost $0.039 with gpt-5-mini.

What this means for limits that agents will hit:

• Size the burst to the fan-out. Agents send their tool calls in parallel, so a question that needs eight lookups needs eight tokens at once. A limit like "5/min" refuses that question every time, however slowly the user asks. Look at how many calls one turn makes before you pick a number.

• Tell the model what to do when it's refused. Without a rule, the model usually retries at once (nine runs of ten here), and with per-minute limits that never works. One sentence in the instructions stopped it in all ten runs.

• Don't count on the model to wait. It reads retry_after_seconds and passes it on to the user, but it has no way to sleep. If an answer must be complete, retry in your code, or put the work in a background job; see the long-running tasks guide.

These hold for Promptise Foundry 1.3.3, checked against its source and the runs above:

• Buckets live in one process. TokenBucketLimiter keeps its buckets in memory, so every worker or replica has its own, and a restart refills them. With four replicas behind a load balancer, a client gets up to four times its limit. There's no shared limiter in the SDK.

• A limiter's `consume()` is synchronous. RateLimitMiddleware calls it inline on the event loop, so a limiter that asks Redis over the network would block every other request while it waits. For a limit shared across replicas, enforce it at your gateway, or write an async middleware of your own.

• The two rate limits stack. A call to get_sales spends a token from the server-wide bucket and one from its own. The server-wide limiter runs first, so a call the per-tool limit refuses has already spent its server-wide token.

• Concurrency limits are shared by all clients. max_concurrent and ConcurrencyLimiter count every caller together, as step 7 showed. A per-client cap is your own middleware.

• Refusals aren't HTTP 429s. Proxies, API gateways and dashboards that count status codes see a 200. Count RATE_LIMIT_EXCEEDED from the response, or in a middleware placed before the limiter; the MCP server monitoring guide explains how middleware order decides what gets counted.

• Resource reads and prompts count too. They run through the same middleware, so they spend tokens from the server-wide bucket. rate_limit= also works on @server.resource and @server.prompt.

• The shared-bucket warning is a log line. It doesn't stop the server, it isn't raised for a server without AuthMiddleware or a limiter with a key_func, and it's logged once. A test with two clients is still the proof.

• `burst` defaults to the rate. TokenBucketLimiter(rate_per_minute=600) allows 600 calls at once, and a declared "100/min" allows 100 at once. Set burst when you want something smoother.

Should an MCP server return HTTP 429 when a client is rate limited?

Not for a tool call. The request reached the server and got an answer: the answer is that this call was refused, which the MCP protocol expresses as a tool result with isError: true. That's also what lets the model read the reason and react. A 429 makes sense in front of the server, at a gateway that limits raw HTTP requests before they reach MCP at all.

How is this different from OpenAI or Anthropic rate limits?

Those limit your calls to the model provider: requests and tokens per minute on your API key. Tool rate limiting protects the systems your tools call, and splits their capacity fairly between your own customers. An agent can hit either one. For the model side, see the semantic cache and fallback guide.

Token bucket or fixed window?

A token bucket suits agents, because they work in bursts: a turn sends several calls at once, then the agent thinks. The bucket allows the burst and holds the long-run rate. A fixed window, say 100 calls per clock minute, lets a client spend the whole minute's budget in its first second, and again in the first second of the next minute.

How do I rate limit per user rather than per agent?

The key comes from the identity on the call, which is the agent's unless the agent passes on the user's. If each user's requests reach the server with their own token, the default key is already per user. Otherwise, write a key_func that reads whatever identifies the user on the request. See the agent identity guide for passing a user's token through an agent.

Can I share one rate limit across several MCP server replicas?

Not with the built-in limiter, whose buckets live in each process. Use sticky routing per client, so each client always hits the same replica, divide the limit by the number of replicas, or enforce the shared limit at a gateway in front of them.

• Caching & Performance: Rate Limiting: RateLimitMiddleware, TokenBucketLimiter and declared per-tool limits.

• Caching & Performance: Concurrency Limits: max_concurrent and ConcurrencyLimiter.

• Multi-Tenancy: where tenant_id comes from and everything it scopes.

• Routers & Middleware: writing middleware like the per-client concurrency cap.

• MCP server caching and circuit breakers: caching, retries, timeouts and a breaker for the API behind your tools.

• Multi-tenant AI agents: keeping customers apart across memory, caches and tools.

• Build a production MCP server: auth, roles, rate limits and an audit log in one server.