Too Many MCP Tools? LLM Tool Selection, Measured

Three things. Every model call carries every tool definition, so cost and latency grow with the tool count: 122 tools came to about 9,700 input tokens per call before the model read a word of the task. Past a limit the call fails: OpenAI rejects requests with more than 128 tools. And look-alike tools (suspend_user and delete_user, create_domain and create_dns_record) give the model more ways to pick the wrong one. The fix is to send each request only the tools that fit it. With Promptise Foundry, that's one argument:

    agent = await build_agent(
        model="openai:gpt-5-mini",
        servers={"platform": StdioServerSpec(command=sys.executable, args=["platform_server.py"])},
        optimize_tools="semantic",  # each request gets only the tools that fit it
    )

Before each model call, the agent compares the recent conversation with every tool description using a small local embedding model, and offers the model the 8 best matches, plus a request_more_tools fallback. Here's one of the recorded runs, "dep_9f2c is building the wrong branch. Stop it.":

[semantic] task 6, expected cancel_deployment
  offered: ['create_project', 'delete_project', 'archive_project', 'transfer_project', 'cancel_deployment', 'redeploy', 'unset_environment_variable', 'update_organization_settings', 'request_more_tools']
  called:  ['cancel_deployment']
  answer:  Done — I've stopped deployment dep_9f2c.

Across the experiment, the median first call dropped from about 9,700 input tokens to about 800. It isn't free, though. With default settings, semantic selection left the right tool out of the first call's 8 for 4 of the 20 tasks. The model can ask for more with request_more_tools, and for some tasks it did; for one, it reached for a heavier look-alike instead. The rest of this guide shows the measurements, where it fails, and how to tune it.

optimize_tools works in two layers. The static layer runs once, when the agent is built: it strips parameter descriptions from the schemas and shortens long tool descriptions. The semantic layer runs on every request:

The three presets, from Promptise's source (promptise/tool_optimization.py):

Two details matter later. The text that gets embedded is name: description after shortening, so the semantic preset matches requests against the first 100 characters of each description. And since 1.3.0 the selection is made again before every model call, from the latest user message, the reply it answers, the turn's tool calls and up to three earlier user turns (semantic_context_turns); tools already called in the turn stay.

• Python 3.10 or newer.

• Promptise Foundry 1.3.2 or newer with sentence-transformers, which semantic selection uses for embeddings. It isn't part of the base install; the promptise[tool-optimization] extra brings it. Without it, build_agent(optimize_tools="semantic") raises an ImportError that names the extra, before it connects to any server. It pulls in PyTorch, so expect a large download. This guide used sentence-transformers 6.1.0.

• An OpenAI API key. The default embedding model, all-MiniLM-L6-v2, runs locally. Promptise downloads it from Hugging Face on first use and caches it in ~/.cache/huggingface/.

pip install "promptise[tool-optimization]>=1.3.2"
export OPENAI_API_KEY="sk-..."

You need a server that looks like the real thing: many resources, the usual list, get, create, update and delete for each, plus the special actions every admin API grows. Writing 122 functions by hand would be silly, so this helper generates them from tables. Each tool just echoes its arguments back, which is all you need to see which one the model picked:

def make_tool(server, name: str, description: str, params: dict[str, str]):
    """Register one tool. Parameters whose description starts with "Optional" may be left out."""

    async def handler(**kwargs):
        return {"ok": True, "tool": name, "arguments": kwargs}

    parameters, annotations = [], {}
    for p, d in params.items():
        optional = d.startswith("Optional")
        annotation = Annotated[str | None if optional else str, Field(description=d)]
        parameters.append(inspect.Parameter(p, inspect.Parameter.KEYWORD_ONLY, annotation=annotation,
                                            default=None if optional else inspect.Parameter.empty))
        annotations[p] = annotation
    handler.__signature__ = inspect.Signature(parameters)
    handler.__annotations__ = annotations
    handler.__name__ = name
    server.tool(name=name, description=description)(handler)


def add_tools(server, resources: dict, actions: dict):
    """Five standard tools per resource, plus one tool per action."""
    for res, (plural, example, what, fields) in resources.items():
        noun = res.replace("_", " ")
        id_param = {f"{res}_id": f'The {noun} ID, for example "{example}".'}
        make_tool(server, f"list_{res}s",
                  f"List {plural} in the organization, newest first. A {noun} is {what}. "
                  f"Returns IDs you can pass to get_{res}.",
                  {"filter": "Optional text to filter by.", "limit": "Optional maximum number of results."})

The server file is mostly data: 14 resources and 52 actions, each with descriptions written the way a careful API team would, including when to use one tool rather than its look-alike:

server = MCPServer("acme-platform")

# resource: (plural, ID example, what it is, fields you can set on create and update)
RESOURCES = {
    "user": ("users", "u_42", "a person with a login in your organization",
             {"email": "Work email address.", "role": "One of owner, admin, member, billing or viewer."}),
    # ... 13 more resources
}

# Actions beyond list, get, create, update and delete. name: (description, {parameter: description})
ACTIONS = {
    "suspend_user": ("Suspend a user so they can't sign in. Their data and memberships are kept and can be restored. "
                     "Use this, not delete_user, when someone leaves the company.",
                     {"user_id": "The user to suspend.", "reason": "Why, recorded in the audit log."}),
    "list_audit_events": ("Search the audit log: who did what, and when, across the organization. "
                          "Use it to find out who changed or deleted something.",
                          {"action": "Action to filter by, for example database.delete.", "since_days": "How far back."}),
    # ... 50 more actions
}

add_tools(server, RESOURCES, ACTIONS)

The excerpt above shows the pattern; the full server repeats it for all 122 tools. A second server, customer_server.py, holds 26 billing and support tools; you'll need it in a moment.

Before spending anything on model calls, look at the payload. This script builds an agent at each level and estimates the tokens of the tool definitions, serialized the way LangChain sends them to OpenAI. It's an offline estimate; the API's own count, in Step 5, came out lower.

"""How big are the tool definitions the model receives, at each optimization level?

An offline estimate with tiktoken, which langchain-openai brings; no model is called.

    python definitions.py
"""
import asyncio
import json
import sys

import tiktoken
from langchain_core.utils.function_calling import convert_to_openai_tool

from promptise import build_agent
from promptise.config import StdioServerSpec

SERVERS = {"platform": StdioServerSpec(command=sys.executable, args=["platform_server.py"])}
enc = tiktoken.get_encoding("o200k_base")


def definition_tokens(tools) -> int:
    payload = [convert_to_openai_tool(t) for t in tools]
    return len(enc.encode(json.dumps(payload)))


async def main():
    for level in [None, "minimal", "standard", "semantic"]:
        agent = await build_agent(model="openai:gpt-5-mini", servers=SERVERS, optimize_tools=level)
        try:
            tools = agent.tools
            suspend = next(t for t in tools if t.name == "suspend_user")
            print(f"--- optimize_tools={level!r}: {len(tools)} tools, {definition_tokens(tools)} tokens")
            print(json.dumps(convert_to_openai_tool(suspend)["function"]))
        finally:
            await agent.shutdown()


asyncio.run(main())

--- optimize_tools=None: 122 tools, 12432 tokens
{"name": "suspend_user", "description": "Suspend a user so they can't sign in. Their data and memberships are kept and can be restored. Use this, not delete_user, when someone leaves the company.", "parameters": {"properties": {"user_id": {"description": "The user to suspend.", "type": "string"}, "reason": {"description": "Why, recorded in the audit log.", "type": "string"}}, "required": ["user_id", "reason"], "type": "object"}}
--- optimize_tools='minimal': 122 tools, 10022 tokens
{"name": "suspend_user", "description": "Suspend a user so they can't sign in. Their data and memberships are kept and can be restored. Use this, not delete_user, when someone leaves the company.", "parameters": {"properties": {"user_id": {"type": "string"}, "reason": {"type": "string"}}, "required": ["user_id", "reason"], "type": "object"}}
--- optimize_tools='standard': 122 tools, 9927 tokens
{"name": "suspend_user", "description": "Suspend a user so they can't sign in. Their data and memberships are kept and can be restored. Use this, not delete_user, when someone leaves the...", "parameters": {"properties": {"user_id": {"type": "string"}, "reason": {"type": "string"}}, "required": ["user_id", "reason"], "type": "object"}}

--- optimize_tools='semantic': 123 tools, 9931 tokens
{"name": "suspend_user", "description": "Suspend a user so they can't sign in. Their data and memberships are kept and can be restored.", "parameters": {"properties": {"user_id": {"type": "string"}, "reason": {"type": "string"}}, "required": ["user_id", "reason"], "type": "object"}}

Watch what happens to suspend_user. minimal drops the parameter descriptions. standard cuts the description mid-sentence. semantic cuts it before the one sentence that says when to use it. The semantic line counts all 123 tools (122 plus a fallback tool), but the model only ever sees 8 of them per model call, plus the fallback, as Step 4 shows.

The cut ends on a whole sentence when that keeps at least two thirds of the limit, which is why the semantic definition stops cleanly after "restored." while standard, with 150 characters to fill, still cuts mid-sentence.

Twenty requests, phrased the way people actually ask, each with the one tool that does the job. Several are deliberately close to a dangerous look-alike:

"""Twenty admin requests, phrased the way people ask, each with the one tool that does the job."""

TASKS = [
    ("Jane (u_42) left the company yesterday. Make sure she can't get in any more.", "suspend_user"),
    ("The last deploy of prj_web_shop broke checkout. Put production back the way it was.", "rollback_deployment"),
    ("Our CI key key_8812 leaked in a public repo. Give us a fresh secret for it with the same permissions.", "rotate_api_key"),
    ("Point www.acme.io at acme-shop.pages.dev with a CNAME.", "create_dns_record"),
    ("Somebody removed db_staging last week. Find out who.", "list_audit_events"),
    ("dep_9f2c is building the wrong branch. Stop it.", "cancel_deployment"),
    ("How much was key_8812 used this week?", "get_api_key_usage"),
    ("Bring back last night's backup of db_staging.", "restore_database_backup"),
    ("Add Omar (u_77) to the payments team, t_7.", "add_team_member"),
    ("inc_12 is fixed, the database is healthy again. Close it out.", "resolve_incident"),
    ("Turn new_checkout on for 10% of our users.", "set_feature_flag_rollout"),
    ("Priya's laptop (u_51) was stolen. Log her out of everything right now; she still works here.", "revoke_user_sessions"),
    ("Ship the preview build dep_7a1e to production as it is.", "promote_deployment"),
    ("db_staging keeps running out of memory. Put it on the large plan.", "resize_database"),
    ("Our endpoint was down on Tuesday and missed delivery dlv_501. Send it again.", "retry_webhook_delivery"),
    ("We need more than 20 projects. Can you get our limit raised to 50?", "request_quota_increase"),
    ("Sam (sam@acme.io) is joining finance. Get them into the org with the billing role.", "invite_user"),
    ("Is acme.io verified yet? We added the TXT record an hour ago.", "verify_domain"),
    ("ar_5 keeps paging everyone during the migration tonight. Shut it up for two hours.", "mute_alert_rule"),
    ("Set CHECKOUT_REGION to eu-west-1 on env_prod.", "set_environment_variable"),
]

The experiment builds one agent per setting and runs every task against each, interleaved so they share the same network conditions. A LangChain callback records what each model call was given, which tools it called and what it cost. invocation_params holds the tools actually sent, so for semantic selection you see the ones chosen for that call, not the full list:

SETTINGS = {
    "all tools": None,
    "minimal": "minimal",
    "standard": "standard",
    "semantic": "semantic",
    "semantic, full descriptions": ToolOptimizationConfig(
        level=OptimizationLevel.SEMANTIC, max_description_length=400
    ),
}


class ModelCalls(AsyncCallbackHandler):
    """Records the tools offered, the tools called, token usage and duration of every model call."""

    def __init__(self):
        self.calls = []

    async def on_chat_model_start(self, serialized, messages, **kwargs):
        tools = (kwargs.get("invocation_params") or {}).get("tools") or []
        self.calls.append({"tools": [t["function"]["name"] for t in tools], "start": time.perf_counter()})

    async def on_llm_end(self, response, **kwargs):
        call = self.calls[-1]
        message = response.generations[0][0].message
        usage = message.usage_metadata or {}
        call["seconds"] = time.perf_counter() - call.pop("start")
        call["called"] = [c["name"] for c in message.tool_calls]
        call["input_tokens"] = usage.get("input_tokens", 0)
        call["cached_tokens"] = (usage.get("input_token_details") or {}).get("cache_read", 0)
        call["output_tokens"] = usage.get("output_tokens", 0)

Pass it per request with agent.ainvoke(..., config={"callbacks": [recorder]}). A task counts as solved when the expected tool was called at any point in the run.

Some runs don't end with an answer. The model can keep calling tools until the ReAct agent's reason node reaches its limit of 15 iterations, and since Promptise 1.3.2 such a run raises GraphIterationLimitError instead of returning the last tool result as if it were the answer. The runner catches it and keeps the run, with the tools called so far and no answer, so it counts like any other: solved only if the expected tool was called before the limit.

    try:
        result = await agent.ainvoke(
            {"messages": [{"role": "user", "content": text}]}, config={"callbacks": [recorder]}
        )
        row["answer"] = result["messages"][-1].content
    except GraphIterationLimitError as error:
        # The model was still calling tools when the limit stopped the run: no answer.
        row["answer"] = None
        row["iteration_limit"] = str(error)
        row["pending_tools"] = error.pending_tools
    except Exception as error:  # the run never got going, e.g. the provider refused the request
        return {"error": f"{type(error).__name__}: {error}", **row}

Each of the 20 tasks ran three times per setting with openai:gpt-5-mini on Promptise Foundry 1.3.2, from the code in the repository: 60 runs per setting, 300 in total. "Right tool offered" counts the runs whose first model call included the expected tool, and "Stopped at the limit" the runs that ended in GraphIterationLimitError. Medians are per run; tokens are what the OpenAI API reported. Earlier versions of this guide reported a 200-run experiment on a different 122-tool server and task set whose code wasn't published; these numbers come from the published code and aren't comparable with those.

What the numbers say:

• gpt-5-mini handled 122 tools well. With every tool in view it got 57 of 60 right, the best of the five settings, and its only misses were the backup restore below. Too many tools didn't break accuracy here; it made every call carry almost 10,000 tokens.

• The static presets saved 16 to 17% of input tokens and lost accuracy. The likely reason is the parameter descriptions they strip, which carry hints such as The environment ID, for example "env_prod". Without them, eight runs went looking for an ID the request already gave, calling list_environments, list_projects or list_databases until the iteration limit, and two took ar_5, an alert rule, for a user. Two of standard's misses used a tool that also does the job: update_database with the large plan instead of resize_database, and create_user instead of invite_user.

• Semantic selection cut first-call input by 92%, from 9,686 to 798 tokens, and both semantic settings were the fastest. The right tool was missing from the first call in 12 of the preset's 60 runs and in 9 with full descriptions. The model got it back through request_more_tools or the next call's selection in 8 and 6 of those runs, not in the rest, as the next section shows.

• Caching narrows the cost gap. OpenAI served 96% of the all-tools input from its prompt cache, because the 122 definitions are the same at the start of every request. Cached tokens are billed at a discount, so the real cost difference is smaller than the raw token counts suggest. The semantic prompts got few cache hits, 10 to 13%: they're short, and OpenAI only caches prompts above a minimum length.

Of the 300 runs, 27 stopped at the iteration limit. Almost all of them repeated a call this echo server can't answer usefully, such as a list_* tool that lists nothing or get_api_key_usage, which returns no numbers. Nine of them had called the expected tool before that and count as solved; the other 18 count as misses.

One task failed under every setting, and it isn't about tool count. "Bring back last night's backup" failed in all 15 runs: list_database_backups returns no backups to choose from on this test server, and a restore overwrites data, so the agent kept looking, seven times until the iteration limit, or asked for confirmation instead of restoring. That's the test, not the agent. The CNAME request is different: the settings that saw all 122 tools created the record in 8 of 9 runs, while all six semantic runs failed, three listing domains until the limit and three asking whether and where to make the change.

Semantic selection lost runs on three tasks that all tools solved every time: the CNAME record above, a session revoke and an alert rule. In the last two, the right tool wasn't on the first call. The costliest case was the request to sign a user out everywhere, from someone who says she still works there. Here it is under both semantic settings, the first of three runs each:

[semantic] task 12, expected revoke_user_sessions
  offered: ['delete_user', 'delete_team', 'delete_database', 'delete_incident', 'suspend_user', 'reactivate_user', 'reset_user_mfa', 'transfer_ownership', 'request_more_tools']
  called:  ['suspend_user']
  answer:  Done — Priya (u_51) has been suspended so she can't sign in. I used the reason: "Company laptop stolen — immediate suspension to prevent unauthorized access. User still employed; do not delete account." Do you want me to also reset her MFA devices, transfer ownership if she was an owner, or remove h

[semantic, full descriptions] task 12, expected revoke_user_sessions
  offered: ['delete_user', 'delete_team', 'delete_database', 'delete_incident', 'suspend_user', 'reactivate_user', 'reset_user_mfa', 'transfer_ownership', 'request_more_tools']
  called:  ['suspend_user']
  answer:  Done — I suspended Priya's account (u_51) immediately. That prevents any sign-in and ends active sessions so the stolen laptop can no longer access the account. I realize you said she still works here. Suspension prevents her from signing in while suspended. Options and recommended next steps: - If

"Log her out of everything" is revoke_user_sessions, but the index ranked the user tools around it higher, and the first call got suspend_user instead. All six semantic runs suspended Priya first. Two then reactivated her and revoked her sessions, which is what was asked; two revoked her sessions but left her suspended; and two stopped at the suspension and offered next steps. The three settings that saw all 122 tools revoked her sessions in all nine runs and suspended no one. Descriptions matter in the other direction as well: "Jane left the company" doesn't look much like "Suspend a user so they can't sign in". The sentence that does match, "when someone leaves the company", sits after character 100, so the preset's first call missed suspend_user in all three runs and full descriptions got it every time. The preset recovered there: twice the model looked Jane up and the next call's selection, which reads the turn's tool calls, offered suspend_user, and once it asked request_more_tools for it.

That's the failure to take seriously. With the right tool missing, the model didn't ask for more tools or stop to check; it used the closest tool it had. Suspension can be undone, but the same reflex with a destructive look-alike would not be. Keep approval on tools that change things, and run your riskiest requests through the retrieval check below.

The alert rule failed more safely. "ar_5 keeps paging everyone… Shut it up for two hours" matched incidents, not alert rules, so mute_alert_rule was missing from the first call in all six semantic runs. Five times the model asked what ar_5 was instead of acting; once it called request_more_tools, got mute_alert_rule and muted the rule. Asking is the better failure, but the task wasn't done.

The audit question missed the first call under both semantic settings. "Somebody removed db_staging last week. Find out who" is about deleting databases, word for word, so the index offered delete_database and restore_database_backup and not list_audit_events. Embedding models match topics, not intent. Here the fallback earned its place: in all six runs the model called request_more_tools, got list_audit_events and used it.

Test retrieval before you pay for model calls

Every miss above can be found without calling the model. The agent's index is ToolIndex in promptise.tool_optimization. It's what build_agent uses, and since 1.3.0 the docs describe it, with build_selection_query, for exactly this kind of offline check. Build it over the agent's own tools and check each task's expected tool against the top 8:

"""Before you pay for a single model call: would semantic selection hand the model the right tool?

Builds the same index the agent builds (same tools, same truncated descriptions,
same embedding model) and checks each task's expected tool against the top 8.
"""
import asyncio
import sys

from promptise import OptimizationLevel, ToolOptimizationConfig, build_agent
from promptise.config import StdioServerSpec
from promptise.tool_optimization import ToolIndex

from tasks import TASKS

SERVERS = {"platform": StdioServerSpec(command=sys.executable, args=["platform_server.py"])}

SETUPS = {
    "semantic preset": (ToolOptimizationConfig(level=OptimizationLevel.SEMANTIC), "all-MiniLM-L6-v2"),
    "full descriptions": (ToolOptimizationConfig(level=OptimizationLevel.SEMANTIC, max_description_length=400),
                          "all-MiniLM-L6-v2"),
    "full descriptions, bge-small": (ToolOptimizationConfig(level=OptimizationLevel.SEMANTIC, max_description_length=400),
                                     "BAAI/bge-small-en-v1.5"),
}


async def main():
    for label, (config, embedding_model) in SETUPS.items():
        agent = await build_agent(model="openai:gpt-5-mini", servers=SERVERS, optimize_tools=config)
        try:
            tools = [t for t in agent.tools if t.name != "request_more_tools"]
            index = ToolIndex(tools, model_name_or_path=embedding_model)
            misses = []
            for text, expected in TASKS:
                names = [t.name for t in index.select(text, top_k=8)]
                if expected not in names:
                    misses.append(f"{expected} (got {', '.join(names[:3])}, ...)")
            print(f"{label}: right tool in the top 8 for {len(TASKS) - len(misses)}/{len(TASKS)} tasks")
            for miss in misses:
                print("  missed", miss)
        finally:
            await agent.shutdown()


asyncio.run(main())

semantic preset: right tool in the top 8 for 16/20 tasks
  missed suspend_user (got delete_team, list_incidents, reactivate_user, ...)
  missed list_audit_events (got delete_database, create_database, restore_database_backup, ...)
  missed revoke_user_sessions (got reset_user_mfa, reactivate_user, delete_user, ...)
  missed mute_alert_rule (got resolve_incident, transfer_project, escalate_incident, ...)
full descriptions: right tool in the top 8 for 17/20 tasks
  missed list_audit_events (got delete_database, create_database, restore_database_backup, ...)
  missed revoke_user_sessions (got suspend_user, reset_user_mfa, reactivate_user, ...)
  missed mute_alert_rule (got resolve_incident, transfer_project, escalate_incident, ...)
full descriptions, bge-small: right tool in the top 8 for 19/20 tasks
  missed create_dns_record (got request_quota_increase, verify_domain, search_logs, ...)

This offline check predicted the live runs' first calls exactly: 16 of 20 for the preset, 17 of 20 with full descriptions, the same misses, at no model cost. Swapping the embedding model for BAAI/bge-small-en-v1.5 fixed the audit question, the session revoke and the alert, and broke the DNS one, so a different model isn't automatically a better one. Run your own real requests through this check before you switch semantic selection on.

How to tune it

• Keep full descriptions. ToolOptimizationConfig(level=OptimizationLevel.SEMANTIC, max_description_length=400) put the right tool in the first call more often, 51 times in 60 against 48, with the same input tokens per call. In this run it didn't change the end result: both semantic settings solved 50 of 60.

• Put the "when to use it" first. The index reads name: description. A description that opens with the situations people describe ("someone left", "find out who changed something") matches requests better than one that opens with what the endpoint does.

• Try another embedding model with embedding_model=, a Hugging Face name or a local path, and keep it only if your retrieval check improves.

• Use `semantic_top_k`, `preserve_tools` and the fallback. semantic_top_k sets how many tools each call gets, preserve_tools adds tools to every call on top of those, with their full descriptions, and the model can call request_more_tools with a query or tool_names when the right tool is missing. In a test, semantic_top_k=3 offered 3 tools plus the fallback, and on "Stop dep_9f2c" the model called archive_project instead of asking for more, so don't set it too low; two preserved tools were added to the 8 with their full descriptions and parameter descriptions.

Put all 148 tools, platform and customer, in one server (all_in_one_server.py in the repository), and with all tools sent OpenAI refuses every call:

{"error": "GraphExecutionError: Graph 'react' failed at node 'reason': OpenAIInvalidRequestError: Error code: 400 - {'error': {'message': \"Invalid 'tools': array too long. Expected an array with maximum length 128, but got an array with length 148 instead.\", 'type': 'invalid_request_error', 'param': 'tools', 'code': 'array_above_max_length'}}", "setting": "all tools", "rep": 0, "task": 1, "expected": "suspend_user"}

minimal and standard failed the same way, since they still send every tool. That's why the measured server stays at 122. If your agent connects several MCP servers, count the total: two ordinary servers can cross 128 together. Semantic selection sends 8 tools plus the fallback and any preserved tools, so it stays far below this limit, and tools the model unlocks with request_more_tools stop at 100 per call. OpenAI's own function calling guide suggests far fewer, under 20 at the start of a turn, as a soft rule of thumb.

Selection reads more than the latest message: before each model call it also reads the assistant reply the message answers and up to three earlier user turns. That matters in a conversation. Here are the tools offered for the same job, asked directly and then confirmed as a follow-up:

One message: ['delete_project', 'archive_project', 'rollback_deployment', 'promote_deployment', 'redeploy', 'retry_webhook_delivery', 'resolve_incident', 'post_incident_update', 'request_more_tools']
Follow-up:   ['list_incidents', 'update_incident', 'rollback_deployment', 'promote_deployment', 'list_webhook_deliveries', 'retry_webhook_delivery', 'resolve_incident', 'post_incident_update', 'request_more_tools']

After "Yes, go ahead.", rollback_deployment was still on offer, because the selection read the conversation and not just three words. Before 1.3.0 it read only the latest message, and the follow-up got a list without it. In this run the model still didn't roll back on its own: its first reply had laid out a plan with several options, so it looked up open incidents and asked which actions to take.

Selection works around a big tool list. The other strategy is to not have one. Many MCP servers are a one-to-one copy of a REST API, so they inherit dozens of look-alike CRUD endpoints. MCPcast, Promptise's OpenAPI-to-MCP generator, takes the opposite approach: it drops what agents shouldn't touch and can merge related endpoints into one well-named tool. On the Swagger Petstore it turned 19 operations into 12 tools. OpenAPI to MCP: Turn Any REST API into an MCP Server walks through it, with an evaluation of how well a real agent uses the result.

The two combine well. Curate the servers you own; use semantic selection for the ones you don't, or when the total is still over a hundred.

These are true of Promptise Foundry 1.3.2 and were checked against its source and a run:

• Selection is only as good as what it embeds. The semantic preset embeds the first 100 characters of each description, and embedding models match topics, not intent. Check your requests with ToolIndex before you switch it on.

• Recovering from a miss is up to the model. When the right tool isn't offered, the model can ask request_more_tools for it and gets it on the next call. In this experiment it did so in 12 of the 120 semantic runs, six of them on the audit question, but never on the session revoke, where it suspended the user instead.

• `request_more_tools` without arguments pages through the catalogue. Called with no query or tool_names, it unlocks the next 8 tools most relevant to the conversation that the model wasn't offered yet. In a test where the model was told to call it three times without arguments, its four model calls carried 9, 15, 23 and 31 tools. One call unlocks at most 32 tools, and a model call gets at most 100 indexed tools, so it stays under OpenAI's 128-tool limit.

• A run that loops ends in an exception. If the model is still calling tools when the reason node reaches its limit, Promptise 1.3.2 raises GraphIterationLimitError, with the tools whose results the model never saw in pending_tools, where earlier versions returned the last tool result as the answer. Catch it where you call the agent: 27 of the 300 runs here ended that way.

• This is one model, one server and 20 tasks. Three runs per task per setting is enough to see a 10-point gap, not a 2-point one. Measure with your own tools and requests before you decide.

How many tools can an LLM handle?

It depends on the model and the tools. gpt-5-mini picked the right tool in 95% of runs with 122 tools in view, so the ceiling is usually set by cost, latency and provider limits before accuracy. OpenAI rejects more than 128 tools in one request, and recommends far fewer as a rule of thumb.

How many tools should an LLM agent see?

As few as the task needs. In this experiment, 8 tools per call cut first-call input tokens by 92%, but the wrong 8 cost up to 12 points of accuracy and, once the right tool was missing, led to a heavier action than the user asked for. Aim for the smallest set that still contains the right tool, and check that with your real requests, as in the retrieval check above.

What is semantic tool selection?

Choosing which tools to offer the model by embedding similarity. The tool descriptions are embedded once, each request is embedded when it arrives, and only the closest matches go to the model. It's also called tool retrieval or dynamic tool selection. In Promptise Foundry it's build_agent(optimize_tools="semantic"), and it runs locally with sentence-transformers.

Does tool selection work with any MCP server?

Yes. Optimization happens in the agent, after it lists the servers' tools, so the servers don't need to change. It works with any tool-calling model, because the model just receives a shorter tool list.

Is it better to merge tools or to filter them?

Merge when you own the server: fewer, clearer tools help every client, not just your agent. Filter when you don't own the server, or when you connect so many that even good tools add up to more than a hundred.

• Tool Optimization: every ToolOptimizationConfig option and preset.

• Building Agents: the rest of build_agent.

• Observability: track token usage across your agents.

• MCPcast reference: generate a curated MCP server from an OpenAPI spec.

• OpenAPI to MCP: Turn Any REST API into an MCP Server: fewer, better tools in practice.

• How to Connect MCP Servers to Your AI Agent in Python: the agent and server basics this guide builds on.