MCP vs CLI for AI Agents: Token Cost, Measured

MCP vs CLI is the 2026 argument about agent tools. One side says MCP servers bloat the context with dozens of tool schemas the model never uses. The other side says shell commands are unsafe and hard to control per user. Most of the numbers on both sides come from vendors with something to sell. So we measured it. We gave one agent 25 GitHub-style tasks and ran it three ways: (a) a GitHub-style MCP server with all 48 tools loaded, (b) the same server with semantic tool selection, and (c) a single run_command tool that runs gh, git and a few Unix commands from an allowlist. All three arms used the same model (gpt-5-mini), and each ran every task three times, 225 runs in total. All three passed every run. The CLI arm used 11x fewer tokens than full MCP. Prompt caching cut the cost gap to 16%, though, and semantic selection saved 73% of tokens yet cost more than loading every tool. Everything below ran on Promptise Foundry 1.3.3 from PyPI, and every output is pasted from one run.

If your agent works alone, on a CLI the model already knows well, in an environment you control, the CLI is cheaper and at least as accurate. That is a lot of ifs. MCP's real costs and benefits sit elsewhere: in prompt caching, in typed errors, and in who is allowed to do what. Those are the sections that matter if you have more than one user.

The repository is a small fixture called acme/webshop: a real git repo with 7 commits by 3 authors and 3 release tags, plus a JSON file holding 30 issues, 6 pull requests, labels, CI checks and releases. Both arms read and write that same data. The MCP arm reaches it through a server with 48 tools. The tool names and parameters follow the style of the official GitHub MCP server (list_issues, get_pull_request_status, add_issue_comment, …), and it includes the many tools a real GitHub server offers that this job doesn't need. The CLI arm reaches it through a local stand-in for the gh CLI. It has the same subcommands and flags as gh for issues, pull requests, labels and releases, and it rejects unknown flags the way gh does. Both call the same Python functions underneath, so a "fact" is identical in both arms.

The 25 tasks are 17 reads ("Which CI check is failing on pull request #21?", "Who authored the most recent commit that touched src/payments.py?") and 8 writes ("Close #25 with a comment saying it is a duplicate of #24"). A read passes if the answer contains the right facts and the database didn't change. A write passes if the database ends up in the right state, whatever the answer says. The database is reset before every task.

"""25 GitHub-style tasks on acme/webshop. Reads are graded on the answer (and an unchanged database);
writes are graded on the database after the run."""
import json
import re

R = "in acme/webshop"
TASKS = [
    ("R01", f"How many open issues {R} are labeled bug?", {"all": ["17"]}),
    ("R02", f"Who is assigned to issue #12 {R}?", {"any": ["liwei", "li wei"]}),
    ("R03", f"What is the title of the most recently created open issue {R}?", {"all": ["currency switcher resets"]}),
    ("R04", f"Which pull request fixes issue #7 {R}? Give the PR number.", {"all": ["20"]}),
    ("R05", f"Which CI check is failing on pull request #21 {R}?", {"all": ["lint"]}),
    ("R06", f"Which files does pull request #19 {R} change?", {"all": ["src/inventory.py", "tests/test_inventory.py"]}),
    ("R07", f"What is the latest release tag {R}?", {"all": ["v2.3.1"]}),
    ("R08", f"Who authored the most recent commit that touched src/payments.py {R}?", {"all": ["maria"]}),
    ("R09", f"In which file is the function apply_discount defined {R}?", {"all": ["src/cart.py"]}),
    ("R10", f"What did the last comment on issue #5 {R} say, and who wrote it?", {"all": ["#21", "liwei"]}),
    ("R11", f"List the numbers of all open pull requests {R}.", {"nums": [19, 20, 21, 22]}),
    ("R12", f"Which open issues {R} are assigned to maria? Give the numbers.", {"nums": [16, 20, 24, 28]}),
    ("R13", f"What is the default currency set in src/payments.py {R}?", {"all": ["eur"]}),
    ("R14", f"How many commits {R} were authored by Li Wei?", {"all": ["3"]}),
    ("R15", f"Which open issues {R} are labeled both bug and payments? Give the numbers.", {"nums": [1, 5, 9, 12, 20, 23]}),
    ("R16", f"What is the base branch of pull request #22 {R}?", {"all": ["release-2.3"]}),
    ("R17", f"Which open issues {R} have no assignee? Give the numbers.", {"nums": [7, 11, 23, 27]}),
    ("W01", f"Comment 'Investigating now.' on issue #12 {R}.", {"db": lambda d: any("investigating now" in c["body"].lower() for c in issue(d, 12)["comments"])}),
    ("W02", f"Add the needs-triage label to issue #30 {R}.", {"db": lambda d: "needs-triage" in issue(d, 30)["labels"]}),
    ("W03", f"Close issue #9 {R}.", {"db": lambda d: issue(d, 9)["state"] == "closed"}),
    ("W04", f"Open a new issue {R} titled 'Refund button misaligned on mobile' with the bug label.",
     {"db": lambda d: any(i["title"].lower().startswith("refund button misaligned on mobile") and "bug" in i["labels"] for i in d["issues"])}),
    ("W05", f"Leave the comment 'Please fix the lint failure.' on pull request #21 {R}.",
     {"db": lambda d: any("lint failure" in c["body"].lower() for p in d["pulls"] if p["number"] == 21 for c in p["comments"])}),
    ("W06", f"Reopen issue #3 {R}.", {"db": lambda d: issue(d, 3)["state"] == "open"}),
    ("W07", f"Assign issue #14 {R} to maria as well, keeping its current assignee.",
     {"db": lambda d: set(issue(d, 14)["assignees"]) == {"sam", "maria"}}),
    ("W08", f"Issue #25 {R} duplicates #24. Close #25 with a comment saying it is a duplicate of #24.",
     {"db": lambda d: issue(d, 25)["state"] == "closed" and any("#24" in c["body"] for c in issue(d, 25)["comments"])}),
]


def issue(d, n):
    return next(i for i in d["issues"] if i["number"] == n)


def grade(task, answer, db_before, db_after):
    tid, _, rule = task
    text = (answer or "").lower()
    if "db" in rule:
        return bool(rule["db"](db_after))
    if db_before != db_after:  # a read task must not write
        return False
    if "all" in rule:
        return all(x.lower() in text for x in rule["all"])
    if "any" in rule:
        return any(x.lower() in text for x in rule["any"])
    if "nums" in rule:
        found = {int(n) for n in re.findall(r"(?<![\w.])#?(\d{1,3})(?!\w|\.\d)", answer or "")}
        return set(rule["nums"]) <= found and not (found & (set(range(1, 31)) - set(rule["nums"])) - {int(n) for n in re.findall(r"\d+", task[1])})
    raise ValueError(rule)


def load(path):
    return json.loads(open(path).read())

The MCP server is ordinary Promptise server code. An excerpt (the full file defines all 48 tools this way):

server = MCPServer(name="github", version="1.0.0")

Owner = Annotated[str, Field(description="Repository owner (user or organization)")]
Repo = Annotated[str, Field(description="Repository name")]
IssueNo = Annotated[int, Field(description="The number of the issue")]

# ...

@server.tool()
async def get_issue(owner: Owner, repo: Repo, issue_number: IssueNo) -> dict:
    """Get details of a specific issue in a GitHub repository, including its body and labels."""
    check(owner, repo)
    return run(lambda: {k: v for k, v in ghdb.get_issue(issue_number).items() if k != "comments"})

Docker was off on the machine we used, so the CLI arm doesn't run in a container sandbox. It runs a local subprocess through one tool that only accepts commands from an allowlist, in the repository checkout, with a minimal environment:

"""The CLI arm's only tool: run an allowlisted command (pipes allowed) in the repo checkout.
Docker was off on this machine, so this is a local subprocess, not a container sandbox."""
import os
import shlex
import subprocess
from pathlib import Path

from langchain_core.tools import tool

HERE = Path(__file__).parent
WORKDIR = Path(os.environ.get("GHREPO", HERE / "fixture" / "webshop"))
ALLOWED = {  # command -> binary. gh is the local stand-in, never the real GitHub CLI.
    "gh": str(HERE / "fakebin" / "gh"), "git": "/usr/bin/git", "grep": "/usr/bin/grep",
    "ls": "/bin/ls", "cat": "/bin/cat", "head": "/usr/bin/head", "tail": "/usr/bin/tail",
    "wc": "/usr/bin/wc", "sort": "/usr/bin/sort", "uniq": "/usr/bin/uniq", "cut": "/usr/bin/cut",
    "jq": "/opt/homebrew/bin/jq", "find": "/usr/bin/find",
}
GIT_READ_ONLY = {"log", "show", "diff", "grep", "ls-files", "blame", "status", "branch", "tag", "rev-parse", "shortlog"}
BLOCKED_TOKENS = {";", "&", "&&", "||", ">", ">>", "<", "`", "$("}
ENV = {"PATH": "/usr/bin:/bin", "GHDB": os.environ.get("GHDB", ""), "GHREPO": str(WORKDIR),
       "GIT_PAGER": "cat", "PAGER": "cat", "NO_COLOR": "1", "HOME": "/tmp"}


def parse(command: str) -> list[list[str]]:
    lex = shlex.shlex(command, posix=True, punctuation_chars="|;&<>")
    lex.whitespace_split = True
    tokens = list(lex)
    if any(t in BLOCKED_TOKENS or "$(" in t or "`" in t for t in tokens):
        raise ValueError("only single commands and pipes (|) are allowed")
    stages, cur = [], []
    for t in tokens + ["|"]:
        if t == "|":
            if not cur:
                raise ValueError("empty pipeline stage")
            stages.append(cur)
            cur = []
        else:
            cur.append(t)
    for argv in stages:
        if argv[0] not in ALLOWED:
            raise ValueError(f"command not allowed: {argv[0]}. Allowed: {', '.join(sorted(ALLOWED))}")
        if argv[0] == "git" and (len(argv) < 2 or argv[1] not in GIT_READ_ONLY):
            raise ValueError(f"git is read-only here; allowed: {', '.join(sorted(GIT_READ_ONLY))}")
        if argv[0] == "find" and any(a in ("-exec", "-execdir", "-delete", "-ok") for a in argv):
            raise ValueError("find -exec/-delete are not allowed")
        argv[0] = ALLOWED[argv[0]]
    return stages


@tool
def run_command(command: str) -> str:
    """Run a shell command in the acme/webshop checkout and return its output.
    gh (the GitHub CLI, already authenticated for acme/webshop), read-only git, grep, ls, cat,
    head, tail, wc, sort, uniq, cut, jq and find are available. Pipes (|) work; ;, &&, redirects
    and subshells do not."""
    try:
        stages = parse(command)
    except ValueError as e:
        return f"error: {e}"
    procs, prev = [], None
    for argv in stages:
        p = subprocess.Popen(argv, cwd=WORKDIR, env=ENV, stdin=prev, stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE, text=True)
        if prev:
            prev.close()
        prev = p.stdout
        procs.append(p)
    try:
        out, last_err = procs[-1].communicate(timeout=15)
    except subprocess.TimeoutExpired:
        for p in procs:
            p.kill()
        return "error: command timed out after 15s"
    err = "".join(p.stderr.read() for p in procs[:-1] if p.wait(timeout=5) is not None) + last_err
    code = procs[-1].returncode
    result = out + (("\n" if out else "") + err if err else "")
    if code:
        result += f"\n(exit code {code})"
    return result[:6000] or "(no output)"

What this allows: gh, read-only git subcommands, and grep, ls, cat, head, tail, wc, sort, uniq, cut, jq and find (without -exec or -delete), joined by pipes. What it blocks: ;, &&, ||, redirects, subshells and every other binary. That is an honest stand-in for "give the agent a shell", but it is not isolation. A real deployment should run the same tool inside a container or VM. Our code execution sandbox guide shows how. Containment changes the security picture but not the token numbers, which is what this experiment measures.

All three arms are the same Promptise agent with the same instructions. Only the tools differ. Semantic selection is a single setting: optimize_tools="semantic" embeds every tool description locally with a small sentence-transformer model. Before each model call it offers the 8 tools that best match the conversation, plus a request_more_tools fallback (see tool optimization).

"""Run the 25 tasks through one arm. Usage: python bench.py <mcp_full|mcp_semantic|cli> <trial> [task ids...]"""
import asyncio
import json
import os
import shutil
import sys
import time
from pathlib import Path

from promptise import build_agent
from promptise.config import StdioServerSpec

from tasks import TASKS, grade, load

HERE = Path(__file__).parent
STATE = Path(os.environ["BENCH_DIR"])
MODEL = "openai:gpt-5-mini"
INSTRUCTIONS = ("You work on the GitHub repository acme/webshop for Acme's engineering team. "
                "Use your tools to look things up and to make the changes asked for; never guess. "
                "Answer in one or two sentences.")
GITHUB = StdioServerSpec(
    command="/tmp/gt10f_2/bin/python", args=[str(HERE / "github_server.py")],
    env={"GHDB": os.environ["GHDB"], "GHREPO": os.environ["GHREPO"], "PATH": "/usr/bin:/bin"},
)


async def make(arm):
    if arm == "mcp_full":
        return await build_agent(model=MODEL, servers={"github": GITHUB}, instructions=INSTRUCTIONS)
    if arm == "mcp_semantic":
        return await build_agent(model=MODEL, servers={"github": GITHUB}, instructions=INSTRUCTIONS,
                                 optimize_tools="semantic")
    if arm == "cli":
        from cli_tool import run_command
        return await build_agent(model=MODEL, servers={}, extra_tools=[run_command], instructions=INSTRUCTIONS)
    raise ValueError(arm)


def proxy_rows(run_id):
    with open(STATE / "proxy.jsonl") as f:
        return [r for r in map(json.loads, f) if r["run"] == run_id]


HELP_MARKERS = ("unknown flag", "unknown command", "Usage:", "invalid choice", "error: ")


async def main(arm, trial, only):
    agent = await make(arm)
    out = open(STATE / f"results_{arm}.jsonl", "a")
    trace = open(STATE / f"trace_{arm}.jsonl", "a")
    try:
        for task in TASKS:
            tid, prompt, _ = task
            if only and tid not in only:
                continue
            run_id = f"{arm}-t{trial}-{tid}"
            shutil.copy(HERE / "fixture" / "db.json", STATE / "db.json")
            before = load(STATE / "db.json")
            (STATE / "current_run.txt").write_text(run_id)
            t0 = time.perf_counter()
            status, messages, answer = "ok", [], ""
            try:
                result = await asyncio.wait_for(
                    agent.ainvoke({"messages": [{"role": "user", "content": prompt}]}), timeout=300)
                messages = result["messages"]
                answer = messages[-1].content if isinstance(messages[-1].content, str) else str(messages[-1].content)
            except Exception as e:
                status, answer = "error", f"{type(e).__name__}: {e}"[:300]
            secs = time.perf_counter() - t0
            await asyncio.sleep(0.3)
            after = load(STATE / "db.json")
            passed = status == "ok" and grade(task, answer, before, after)
            calls, results = [], []
            for m in messages:
                for tc in getattr(m, "tool_calls", None) or []:
                    calls.append({"name": tc["name"], "args": tc["args"]})
                if m.__class__.__name__ == "ToolMessage":
                    results.append(str(m.content)[:600])
            help_cmds = sum(1 for c in calls if c["name"] == "run_command"
                            and ("--help" in c["args"].get("command", "") or " help" in c["args"].get("command", "")))
            errors = sum(1 for r in results if any(k in r for k in HELP_MARKERS) or '"error"' in r or "Error" in r[:40])
            usage = proxy_rows(run_id)
            rec = {"arm": arm, "trial": trial, "task": tid, "passed": passed, "status": status, "secs": round(secs, 2),
                   "model_calls": len(usage), "tool_calls": len(calls), "help_calls": help_cmds, "tool_errors": errors,
                   "more_tools": sum(1 for c in calls if c["name"] == "request_more_tools"),
                   "input": sum(u["input"] or 0 for u in usage), "cached": sum(u["cached"] or 0 for u in usage),
                   "output": sum(u["output"] or 0 for u in usage),
                   "tools_offered": [u["n_tools"] for u in usage], "answer": answer[:400]}
            out.write(json.dumps(rec) + "\n")
            out.flush()
            trace.write(json.dumps({"run": run_id, "calls": calls, "results": results}) + "\n")
            trace.flush()
            print(f"{tid} {'PASS' if passed else 'FAIL'} {status} {secs:5.1f}s calls={len(usage)} tools={len(calls)} "
                  f"err={errors} in={rec['input']} offered={rec['tools_offered']}", flush=True)
    finally:
        await agent.shutdown()


if __name__ == "__main__":
    asyncio.run(main(sys.argv[1], int(sys.argv[2]), set(sys.argv[3:])))

A small local proxy between the agent and OpenAI records each request's token usage, including how many input tokens came from OpenAI's prompt cache. The proxy reads the run id from a file, which is why every run is sequential. We ran bench.py <arm> <trial> for trials 1 to 3 and each arm, then aggregated:

arm             passed  reads writes tokens/task in/call calls p50 s help errors $/100 tasks
mcp_full        75/75   51/51  24/24       16091    6981  2.28   3.9    0      0       0.086
mcp_semantic    75/75   51/51  24/24        4381    1558  2.65   4.1    0      0       0.101
cli             75/75   51/51  24/24        1483     428  2.97   3.4    3     31       0.074

Here is one write task, run once in each arm (trial 1, pasted from show_task.py):

[mcp_full] PASS  model calls 4, tools offered per call [48, 48, 48, 48], input tokens 28030 (27136 cached), output 328
  get_issue: {"owner": "acme", "repo": "webshop", "issue_number": 25}
  add_issue_comment: {"owner": "acme", "repo": "webshop", "issue_number": 25, "body": "Closing this issue as a duplicate of #24."}
  update_issue: {"owner": "acme", "repo": "webshop", "issue_number": 25, "state": "closed"}
  answer: Done — I closed issue #25 and added a comment stating it's a duplicate of #24.

[mcp_semantic] PASS  model calls 5, tools offered per call [9, 9, 9, 9, 9], input tokens 8062 (7296 cached), output 222
  get_issue: {"owner": "acme", "repo": "webshop", "issue_number": 25}
  get_issue: {"owner": "acme", "repo": "webshop", "issue_number": 24}
  add_issue_comment: {"owner": "acme", "repo": "webshop", "issue_number": 25, "body": "Closing as duplicate of #24."}
  update_issue: {"owner": "acme", "repo": "webshop", "issue_number": 25, "state": "closed"}
  answer: Done — I added a comment "Closing as duplicate of #24." on issue #25 and closed it.

[cli] PASS  model calls 4, tools offered per call [1, 1, 1, 1], input tokens 1688 (0 cached), output 275
  run_command: gh issue view 25 --repo acme/webshop --json number,title,state --jq '. | {number: .number, title: .title, state: .state}'
  run_command: gh issue comment 25 --repo acme/webshop --body "Closing as duplicate of #24."
  run_command: gh issue close 25 --repo acme/webshop
  answer: Done — I added the comment "Closing as duplicate of #24." to issue #25 and closed it.

The work is the same: read the issue, comment, close. The difference is what rides along with every call. The MCP arm resends 48 tool definitions each time. The CLI arm sends one, and the model brings its own knowledge of gh for free. We measured the first request of each arm exactly, by sending it to OpenAI with and without the pieces:

One row in that table is our own framework's doing. Promptise's ReAct node adds an "Available tools" summary of every tool's parameters to the system prompt, on top of the schemas the API already receives. With 48 tools that is 1,882 tokens, 27% of the request. It is always on in 1.3.3, and it makes the MCP arms look worse than plain MCP needs to. Without it the full-MCP request would be about 4,960 tokens. The CLI arm would still win on tokens by a wide margin.

Tokens are not dollars. OpenAI bills cached input tokens at a tenth of the normal rate ($0.025 instead of $0.25 per million for gpt-5-mini), and caching works on an identical prompt prefix. The full-MCP arm sends the same 48 tool definitions on every call, so 97% of its input was cached. At list prices without the cache it would have cost $0.433 per 100 tasks, about 6x the CLI arm. With the cache it cost $0.086, 16% more than the CLI arm.

Semantic selection shows the other side of this. It cut tokens per task by 73%, from 16,091 to 4,381. But the tool set changes from call to call as the conversation moves, which breaks the identical prefix, so only 56% of its input was cached. It ended up the most expensive arm at $0.101 per 100 tasks. It used request_more_tools 6 times across 75 runs, which added model calls too. Four of those were for code questions, where the 8 tools it was offered didn't include code search or file contents. The other two were for "Comment 'Investigating now.' on issue #12", where the comment tool itself wasn't among the 8.

The CLI arm's prompts were short (428 input tokens per call) and none of its input was served from cache. It won on cost anyway, because there was so little to send.

What this means in practice:

• For single-tenant cost, a stable tool list can beat a smaller one. If the same agent and tools serve many requests, caching hides most of the schema cost. Measure cost in dollars, not tokens, before you optimize tool lists.

• Semantic selection pays off on context, not on price here. It keeps the window small. That matters for long conversations, smaller-context models and tool-choice accuracy with hundreds of tools (our tool selection guide). On 48 tools with a cache, it wasn't cheaper.

• Latency followed calls, not tokens. The CLI arm was fastest at the median (3.4 s), but its slowest task took 10 model calls.

The CLI arm made 30% more model calls than full MCP (2.97 against 2.28 per task), and 31 of its tool results were errors. In this run they fell into three groups:

• 15 came from our sandbox rules. The model wrote grep -R ... || true (9 times), reached for sed (5) and tried git rev-list (1). All three are reasonable, and all are blocked by our allowlist. Every restriction you add to a shell tool costs some retries.

• 9 came from gaps in our `gh` stand-in. gh api (3), gh release list --limit (3), gh pr list --search, gh run list and gh --version don't exist in our stub but do in the real CLI. Against the real gh most of these would have worked.

• 7 were model mistakes. These were flags gh doesn't have, such as gh issue edit --state and gh issue close --confirm, plus two jq syntax errors. The model recovered from each, at the cost of a call.

The worst case is worth reading in full: a one-line question that took the CLI arm 10 model calls, while semantic MCP needed 2.

[mcp_full] PASS  model calls 3, tools offered per call [48, 48, 48], input tokens 20808 (20224 cached), output 145
  get_pull_request: {"owner": "acme", "repo": "webshop", "pullNumber": 21}
  get_pull_request_status: {"owner": "acme", "repo": "webshop", "pullNumber": 21}
  answer: The failing CI check on pull request #21 is "lint".

[mcp_semantic] PASS  model calls 2, tools offered per call [9, 9], input tokens 2169 (0 cached), output 52
  get_pull_request_status: {"owner": "acme", "repo": "webshop", "pullNumber": 21}
  answer: The failing CI check on pull request #21 is: lint.

[cli] PASS  model calls 10, tools offered per call [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], input tokens 8083 (0 cached), output 1034
  run_command: gh pr view 21 --repo acme/webshop --json number,title,checks,status,commit,url --jq '. | {number: .number, title: .title, url: .url, status: .status, checks: .commits[0].commit.status.contexts}'
  run_command: gh pr view 21 --repo acme/webshop --json headRefOid,commits --jq '{head: .headRefOid, sha: .commits[-1].commit.oid}'
  run_command: gh pr view 21 --repo acme/webshop --json number,title,url,headRefName,headRepositoryOwner,headRepositoryId,mergeStateStatus,mergeable --jq '{number: .number, title: .title, url: .url, headRefName: .headRefName, headRepoOwner: .headRepositoryOwner.login, headRepoId: .headRepositoryId, mergeStateStatus: .mergeStateStatus, mergeable: .mergeable}'
  run_command: gh api repos/acme/webshop/pulls/21/checks --paginate -q '.[].check_runs[] | {name: .name, conclusion: .conclusion, status: .status, details_url: .details_url}'
  run_command: gh pr view 21 --repo acme/webshop --json commits,headRefName --jq '.commits[] | {sha: .commit.oid, message: .commit.message}'
  run_command: gh pr checks 21 --repo acme/webshop --app  --json name,status,conclusion --jq '.[] | {name: .name, status: .status, conclusion: .conclusion}'
  run_command: gh --version
  run_command: gh pr checks 21 --repo acme/webshop --json name,status,conclusion --jq '.[] | {name: .name, status: .status, conclusion: .conclusion}'
  run_command: gh pr checks 21 --repo acme/webshop
  answer: The failing CI check on pull request #21 is "lint" (the "tests" check is passing).

The model guessed at --json field names, tried gh api, checked the version, and only then ran plain gh pr checks. The MCP arms had a tool literally named get_pull_request_status. Our stand-in made this worse in one way: it returns null for an unknown --json field, while the real gh errors and lists the valid fields. On the other hand, "help loops" (the model reading --help to learn a command) were rare: 3 calls in 75 runs. gpt-5-mini already knows gh well.

We saw how much this depends on the CLI matching what the model knows. Our first complete run used an earlier gh stand-in that didn't accept --repo/-R, gh help or --json on a few commands. The model kept reaching for exactly those flags. In that run the CLI arm still passed 75/75, but it produced 117 error results and 14 help calls, used 2,182 tokens per task, and cost $0.104 per 100 tasks, against $0.092 for full MCP. So the CLI's lead in the table above holds only for a CLI the model knows well, with nothing missing. For your own internal tools there is no such CLI in the model's training data, and the comparison may well reverse.

The experiment had one user. Real agents usually have many, and that's where the two approaches really differ:

• Credentials. The CLI arm acts with whatever gh is logged in as on the box, and every user of the agent shares that identity. To act as the user, you need per-user credentials in each subprocess environment, and you must keep the model from reading them (env, cat ~/.config/gh/hosts.yml). With MCP over HTTP the token travels in the transport, and the tool code never shows it to the model. In Promptise, forward_caller_token sends each caller's own bearer token to the server, with one MCP session per caller (server specs).

• Authorization. An allowlist works at the level of commands. It can block rm. It can't easily say "Maria can close issues, Sam can only comment", because both are gh issue .... An MCP server sees typed calls (update_issue(state="closed")) and can check roles per tool with guards, the way our MCP security checklist does.

• Audit. A shell log tells you which command line ran. An MCP audit log records which user called which tool with which arguments. That is the record a security review asks for (identity and authentication for agents).

• Errors. MCP tool failures came back as structured errors ({"error": "issue #99 not found"}); the CLI's came back as stderr text. In this run neither MCP arm hit a single error, so this one we can't put a number on.

None of this was measured here, so treat it as a design trade-off, not a result. But it's why "CLI is cheaper" and "MCP is the right interface for a multi-user product" can both be true.

• A CLI the model knows, one trusted user, a sandbox you control (a coding agent on a developer's machine, CI automation): use the CLI. It was cheapest and fastest here, and accuracy was the same.

• Many users, per-user permissions, an audit trail, or your own internal APIs: use MCP. Keep the tool list stable so prompt caching works, and look at the per-call cost before cutting tools.

• Hundreds of tools or a small context window: add semantic selection, accepting that it lowers your cache hit rate. Decide with dollars and accuracy, not tokens alone.

• Both: they combine fine. An agent can hold an MCP server for the shared, permissioned systems and one allowlisted shell tool for local work. See MCP tools vs resources vs prompts for what else MCP gives you besides tools.

• Not the real GitHub. Both arms hit a local fixture through a gh stand-in and a GitHub-style MCP server we wrote. The stand-in covers issues, PRs, labels and releases but not gh api, gh run and some flags. It cost the CLI arm 9 errors in the final run and many more in the earlier one. The MCP server mimics the official GitHub MCP server's tool style but is not that server, and its schemas are not the same size.

• The CLI arm was not in a container. Docker was off, so it ran as a local subprocess restricted by an allowlist. The token and accuracy numbers don't depend on that. The security of the setup does.

• One model, one framework. gpt-5-mini with default settings, on Promptise. Promptise's always-on tool summary adds 1,882 tokens per call to the full-MCP arm. In a framework without it, MCP's token count would be lower. Another model may know gh less well, or use tools differently.

• Prompt caching is OpenAI's, at today's prices. Other providers cache differently or charge for cache writes. With no caching, full MCP cost about 6x the CLI arm in this run.

• Three trials, 25 small tasks. All three arms passed every run, so this test cannot tell them apart on accuracy. Harder tasks might.

• We reran the experiment twice. Our first attempt had a grader bug: it missed numbers followed by a full stop and failed correct answers. Its fixture also lacked git tags for the releases. We fixed both and reran everything. The second complete run is the one quoted in the CLI-errors section. Then we completed the gh stand-in and ran everything a third time. That third complete run is the one in every table.

Is MCP or CLI better for AI agents?

It depends on who uses the agent. In our test, with one user and a CLI the model knows (gh), the CLI used 11x fewer tokens, cost 14% less and was just as accurate as an MCP server with 48 tools. For multi-user agents, MCP gives per-user credentials, per-tool authorization and an audit trail, which an allowlisted shell can't easily match.

Does MCP use more tokens than CLI tools?

Yes, by a lot: 16,091 against 1,483 tokens per task in our run, because every call carries all the tool schemas. In money the gap is much smaller, because the schemas are an identical prefix that OpenAI's prompt cache serves at a tenth of the price. 97% of the MCP arm's input was cached, and it cost only 16% more than the CLI arm.

Does semantic tool selection reduce MCP token cost?

It reduced tokens by 73% (4,381 per task), but in our run it raised the bill: $0.101 per 100 tasks against $0.086 for loading every tool. Choosing different tools on each call breaks the prompt cache. Use it to keep the context small or to help the model choose among hundreds of tools, and check the cost in dollars.

Are CLI tools safe to give an AI agent?

Only inside limits you enforce. We allowed a fixed list of binaries, made git read-only, and blocked ;, &&, redirects and subshells. That stopped every out-of-policy command in our run, but it is not isolation. Run the tool in a container or VM, and keep credentials the model shouldn't read out of its environment.

What is a help loop in AI agents?

It's when an agent keeps running --help or guessing flags to learn a CLI instead of doing the task. With gh and gpt-5-mini it was rare: 3 help calls in 75 runs. Guessing wrong flags was the bigger cost, especially when our gh stand-in lacked flags the model expected. Expect more of both with CLIs the model hasn't seen in training.

• Too Many MCP Tools? — how semantic tool selection works and when it helps.

• Python AI Agent Frameworks Compared — the same measuring approach applied to five frameworks.

• AI Agent Code Execution Sandbox — put the CLI arm in a real container.

• Connect MCP Servers to an AI Agent — the client side, with auth.

• Docs: Tool optimization, Server specs and per-user tokens and MCP server authentication.