Ollama AI Agent with MCP: Run Local Models in Promptise

Install Ollama, pull a model that supports tool calling (for example ollama pull qwen3:8b), and pass "ollama:qwen3:8b" or Model("qwen3:8b", provider="ollama") to build_agent(). No API key is needed. Promptise talks to Ollama's OpenAI-compatible endpoint at http://localhost:11434/v1 using langchain-openai, which is part of the core install. Add your MCP servers in servers=, and the agent sends their tools to the local model as tool definitions and runs the calls it makes.

• The model must support tools. Only models tagged "tools" in the Ollama library can drive MCP tools. Others reply in prose and never call anything.

• Context length is the most common problem. On machines with less than 24 GiB of VRAM, Ollama defaults to a 4k-token context, which an agent's system prompt, tool definitions and results fill quickly.

• Nothing extra to install. The default route needs only pip install promptise. The native langchain-ollama route is optional.

• `promptise models check` checks that Ollama is running, not just that the configuration is complete.

• Ollama has no authentication. Keep port 11434 off the public internet.

You need Python 3.10 or later, Promptise Foundry 1.4.2 or later, and Ollama from ollama.com/download. The desktop app starts the server for you. On a Linux server, run ollama serve. Then pull a model and install Promptise:

ollama pull qwen3:8b
pip install "promptise>=1.4.2"

There is nothing to put in .env for a local Ollama. The only variable Promptise reads is optional:

# Ollama (local models) — model string example: ollama:llama3.1
export OLLAMA_HOST=http://localhost:11434  # optional
#   ↳ only if Ollama is not on the default http://localhost:11434

promptise models check goes one step further for local models: it connects to the server. This is the real output on a machine where Ollama isn't running:

$ promptise models check ollama:qwen3:8b
ollama:qwen3:8b → ollama:qwen3:8b  (Ollama (local models))
  model part: qwen3:8b — a model you have pulled (`ollama pull llama3.1`)
  route: OpenAI-compatible endpoint http://localhost:11434/v1 (core, nothing to install)
  OLLAMA_HOST: unset (optional) — only if Ollama is not on the default http://localhost:11434
  No API key. The model must support tool calling to drive MCP tools.
Not reachable. The configuration is complete, but nothing is listening at http://localhost:11434 — is Ollama running? (ollama serve) If it runs on another host or port, set OLLAMA_HOST.

When Ollama is up, the check reports the route and stops before calling the model. Add --ping to make one real call.

Which model to pull

Pick from the models tagged "tools" in the Ollama library. Some reasonable starting points at the time of writing:

Small models call tools less reliably than hosted frontier models. They pick the wrong tool more often and invent arguments more often. Test your own tasks before you commit to one. The best LLM for AI agents guide shows how to compare models, Ollama included, on your own tool calls.

The agent answers questions about a shop's products. The model is set once at the top of the file:

import asyncio
import sys

from promptise import Model, build_agent
from promptise.config import StdioServerSpec

# A model you pulled with `ollama pull qwen3:8b`. No API key.
LOCAL = Model("qwen3:8b", provider="ollama", timeout=120)


async def make_agent(model=LOCAL):
    return await build_agent(
        model=model,
        servers={"shop": StdioServerSpec(command=sys.executable, args=["shop_server.py"])},
        instructions="You answer questions about the shop's products. "
                     "Use the tools for stock and prices. Never guess them.",
        max_agent_iterations=10,
    )


async def main():
    agent = await make_agent()
    try:
        result = await agent.ainvoke(
            {"messages": [{"role": "user", "content": "Is KB-101 in stock, and what does it cost?"}]}
        )
        calls = [c["name"] for m in result["messages"] for c in getattr(m, "tool_calls", None) or []]
        print("tool calls:", ", ".join(calls) or "none")
        print(result["messages"][-1].text)
    finally:
        await agent.shutdown()


if __name__ == "__main__":
    asyncio.run(main())

What each part does:

• `Model("qwen3:8b", provider="ollama")` names a model you have pulled. The tag after the colon is part of the model name, so "ollama:qwen3:8b" works as a string too.

• `timeout=120` gives each model call two minutes. A local model on a laptop can take far longer per call than a hosted API, especially on the first call while the model loads into memory.

• `max_agent_iterations=10` stops a run where the model keeps calling tools without finishing, which happens more often with small models.

• `result["messages"][-1].text` reads the answer text. Thinking models such as Qwen 3 can return reasoning alongside the answer, and .text gives you only the text.

Run it with python ollama_agent.py. When Ollama is running and the model is pulled, the script prints the tools the model called and then its answer. If Ollama isn't running, the run fails with OpenAIConnectionError: Connection error., because the default route uses an OpenAI-compatible client.

The tools come from a small MCP server that the agent starts over stdio:

"""A small MCP tool server: product stock and prices for a shop."""
from promptise.mcp.server import MCPServer

server = MCPServer("shop")

STOCK = {"KB-101": 14, "MS-220": 0, "HD-350": 3}
PRICES = {"KB-101": 89.00, "MS-220": 39.50, "HD-350": 129.00}


@server.tool()
async def check_stock(sku: str) -> dict:
    """How many units of a product (by SKU) are in stock."""
    if sku not in STOCK:
        return {"sku": sku, "error": "unknown SKU"}
    return {"sku": sku, "in_stock": STOCK[sku]}


@server.tool()
async def get_price(sku: str) -> dict:
    """The current price of a product (by SKU), in USD."""
    if sku not in PRICES:
        return {"sku": sku, "error": "unknown SKU"}
    return {"sku": sku, "price_usd": PRICES[sku]}


if __name__ == "__main__":
    server.run()  # stdio by default

Everything runs on your hardware:

your code ──▶ Promptise agent ──HTTP──▶ Ollama (localhost:11434/v1) ──▶ local model
                  │     ▲                     │
                  │     └── tool call ◀───────┘
                  ▼
           MCP client ──stdio──▶ shop_server.py ──▶ tool result ──▶ next model call

Promptise connects to each server in servers=, lists its tools and turns them into tool definitions. Every call to Ollama includes those definitions. When the model returns a tool call, Promptise runs the MCP tool and calls the model again with the result. Promptise binds tools with automatic tool choice and never sends a forced tool_choice, which matters because Ollama's OpenAI-compatible endpoint doesn't support that field.

Every tool definition costs context. A server with 40 tools can use thousands of tokens before the user's question arrives. With a local model, keep the tool list short: connect only the servers the agent needs, or let Promptise select tools per step as described in too many MCP tools.

Set the context length

Ollama sets the default context length from your GPU memory: 4k tokens below 24 GiB of VRAM, 32k from 24 to 48 GiB, and 256k at 48 GiB or more, according to the context length docs. When an agent's conversation is longer than the context, Ollama drops the oldest tokens, and the model can lose the system prompt or the tool definitions. The symptom is an agent that stops calling tools after a few turns.

The OpenAI-compatible endpoint has no way to set the context per request, so set it on the server:

OLLAMA_CONTEXT_LENGTH=32768 ollama serve

Ollama's docs recommend at least 64,000 tokens for agents and coding tools if you have the memory. Run ollama ps to see the context a loaded model got and whether it spilled onto the CPU.

You can also set it per agent with the native integration. Install langchain-ollama and pass native=True. Promptise then builds LangChain's ChatOllama, which accepts num_ctx:

LOCAL = Model("qwen3:8b", provider="ollama", native=True, extra={"num_ctx": 32768})

Run Ollama on another machine

A GPU server on your network works the same way. Set OLLAMA_HOST=http://gpu-box:11434, or pass endpoint="http://gpu-box:11434" to Model(...). Promptise accepts the address with or without http:// and with or without a trailing /v1.

Ollama has no API key or user accounts. Anyone who can reach port 11434 can use the models. Keep it on a private network, or put it behind a proxy that checks a token. A local Ollama refuses api_key= in Promptise. For a protected proxy, send the header with extra={"default_headers": {"Authorization": "Bearer ..."}}.

Speed, cost and limits

A local model has no per-token price and no rate limit other than your hardware. The cost is the machine: memory decides which models fit, and the GPU decides how fast they run. Run one request at a time on a laptop, and measure before you put several agents on one GPU. Ollama's FAQ covers concurrency settings such as OLLAMA_NUM_PARALLEL.

Mixing local and hosted models

A common setup sends routine steps to a local model and falls back to a hosted one when the local server is down. FallbackChain tries models in order:

from promptise import FallbackChain

model = FallbackChain([LOCAL, "openai:gpt-5-mini"], timeout_per_model=120)
agent = await make_agent(model=model)

This sends data to the hosted provider whenever the fallback runs, so don't use it where data must stay on your network.

Can a local Ollama model call MCP tools?

Yes, if the model supports tool calling. Promptise is the MCP client: it reads the tools from your MCP servers and sends them to Ollama as tool definitions. The model returns a tool call, Promptise runs it and sends back the result. Pick a model with the "tools" tag in the Ollama library, because models without it ignore the tools.

Does an Ollama agent need an API key or internet access?

No. A local Ollama needs no key, and once the model is pulled, the agent, Ollama and a stdio MCP server all run on your machine. Internet access is only needed to download Ollama and the model, and for any MCP tools that call outside services themselves.

Which Ollama model is best for an AI agent?

It depends on your hardware and your tasks. An 8B model with tool support, such as qwen3:8b, is a reasonable start on a laptop. Larger models plan better across several tools but need much more memory. Run the same questions through two or three candidates and compare which tools they call and whether the answers are right.

Why does my Ollama agent stop using tools after a few turns?

Usually the context is too short. Below 24 GiB of VRAM, Ollama defaults to a 4k-token context, and the system prompt, tool definitions and tool results soon exceed it. Ollama then drops the oldest tokens, which can include the tools. Set OLLAMA_CONTEXT_LENGTH when starting the server, or pass num_ctx with the native integration.

Can I use the same code with a hosted model later?

Yes. Only the Model(...) line changes. Change it to "openai:gpt-5-mini", a Claude model or any other provider and the agent, the MCP servers and the instructions stay the same. Rerun your tests after switching, because different models make different tool choices.

• Compare local and hosted models on your own tasks: best LLM for AI agents.

• Build the full agent step by step: how to build an AI agent.

• Keep retrieval local too: agentic RAG runs embeddings on your own machine.

• Package the agent: deploy an AI agent with Docker.

• Connect more servers: connect MCP servers to an AI agent.

• Every provider setting: Models and providers reference.