Should AI Agents Use Your Website? Allow, Verify or Block

In most cases, yes, on your terms. Let identified agents read public information and do low-risk tasks, give them a proper door such as an API for the actions you want to support, and require the customer's explicit permission for anything that touches an account or money. Block or slow down traffic that won't identify itself. Blocking every agent also turns away the customers who sent them.

Key takeaways:

• Not all AI traffic is the same. Search crawlers, training crawlers and agents acting for a customer need different answers.

• Identity comes first. New standards let agents sign their requests, so your site can tell a known agent from an anonymous bot.

• Decide per action, not per bot. Reading a price list and placing an order carry very different risks.

• A good door beats scraping. An API or MCP server is cheaper for you and more reliable for agents than having them click through pages.

• The protocols are young. Several are drafts or betas, so plan for change.

Much of your traffic is already not human. Cloudflare, which carries a large share of web traffic, reported in July 2026 that "more than 50% of traffic on the Internet is now non-human."

Cloudflare's AI traffic categories (July 1, 2026) are a useful way to sort it:

This guide is about the third kind: an agent working for one of your customers or prospects. It might be a browser agent that clicks through your pages, a chat assistant that fetches a page to answer a question, or a personal agent that books, orders or changes things.

Note that Cloudflare also changed a default: from September 15, 2026, new domains on its network have training and agent traffic blocked by default on pages that show ads. If your site sits behind Cloudflare or a similar service, check what your current settings allow before you decide anything.

Until recently it couldn't, reliably. Bots can claim any name, and blocking by network address fails against traffic routed through home internet connections. Three developments change that:

• Signed requests. An agent can add a cryptographic signature to each request, which proves which company operates it. The approach is being standardized at the IETF, the body that writes internet standards, in the Web Bot Auth working group as "HTTP Message Signatures for automated traffic", a working-group draft dated September 1, 2026. Cloudflare lists agents that sign this way as signed agents.

• The customer's permission. The Personal Agent Protocol, announced by Sierra and Meta on October 6, 2026, lets a customer decide what access their agent gets, including read-only or write, and lets the company set what agents may do. It is built on OAuth, the same standard behind "Sign in with..." buttons. It is a draft heading toward version 1.0.

• Payment-specific checks. Card networks and payment companies have their own protocols so merchants can trust an agent that pays, such as Visa's Trusted Agent Protocol.

Put simply: identity tells you who runs the agent, permission tells you what the customer allowed, and payment protocols tell you the purchase is authorized.

You don't need all of these. Your choice depends on what you sell and which payment providers and platforms you already use; ask them which protocols they support.

The useful question is not "agents: yes or no?" but "which actions, by which agents, with what permission?"

Whether your terms of use allow agents, and what they should say, is a question for your legal team.

An agent that clicks through your website is slow, breaks when you change a page, and costs you server capacity. If you want agents to do something, a direct door is better for both sides. There are three:

• The website, for reading. Clear pages, structured data and an llms.txt summary help agents read accurately.

• An API described with an OpenAPI file, a standard format that tells software what each operation does. Developers and their agents can call it directly.

• An MCP server, a small layer on top of your API that AI assistants such as Claude and ChatGPT can connect to. The Personal Agent Protocol names APIs, via MCP and OpenAPI, as one of the ways agents can reach a company.

Your product and engineering teams can find the technical side in should your SaaS ship an MCP server, an OpenAPI spec, or both?. Tools exist to generate an MCP server from an existing API; Promptise Foundry's MCPcast is one, and the guide on turning a REST API into an MCP server covers the approach.

1. Measure. Ask your web team how much traffic is automated, which agents identify themselves, and which pages they hit. Your CDN or bot-management provider usually has this report.

2. Check your defaults. Find out what your CDN and security settings already block or allow for AI traffic.

3. List the actions. Write down what customers do on your site: browse, search, book, buy, change, cancel.

4. Set a rule for each action. Use the table above as a starting point. Involve customer service, security and the owners of checkout and accounts.

5. Pick a door for the actions you want agents to do. Website for reading; API or MCP for anything frequent or account-related.

6. Require permission for accounts and money. Prefer standard sign-in (OAuth) and your payment provider's agent support over building your own.

7. Set limits and logging. Rate limits per agent, and records of which agent did what for which customer.

8. Publish your stance. A short developer page saying how agents should identify themselves and what they may do.

9. Review quarterly. The protocols and the agents change fast.

• How much of our traffic is automated today, and how much of it identifies itself?

• What does our CDN or bot protection block by default for AI agents?

• Which customer tasks would we like agents to complete, and which would we not?

• Do we have an API that covers those tasks? Is it documented with OpenAPI?

• How would we know which agent acted for which customer, if something went wrong?

• Which agent protocols do our payment provider and e-commerce platform support?

• Who owns this decision, and who reviews it each quarter?

What is agentic commerce?

Agentic commerce is shopping where an AI agent does some of the work for a person: searching, comparing, filling in forms and sometimes paying. Businesses take part by making their products readable and their actions available to agents, with the customer's permission for anything that involves an account or money.

What is the Personal Agent Protocol?

It is an open standard announced by Sierra and Meta on October 6, 2026, with partners including Shopify, Stripe and Walmart. It defines how a customer's personal agent identifies itself to a business and signs in with the customer's permission, built on OAuth. As of October 2026 it is a draft working toward version 1.0.

Should I block AI agents on my website?

Block or limit agents that won't identify themselves, especially on account and checkout pages. Allow identified agents to read public information, and let them act on accounts only with the customer's permission. A blanket block also turns away customers who use agents.

How do I know if AI agents are visiting my site?

Your CDN, bot-management or analytics provider can usually separate automated traffic and show which agents identify themselves. Agents that sign their requests with Web Bot Auth can be verified; others can only be guessed from their behavior.

Do I need an MCP server for AI agents?

Not to be readable. For agents that should take actions, such as booking or checking orders, a documented API comes first, and an MCP server on top of it makes those actions easy for AI assistants to use.

• Understand what agents are: AI agents explained for business leaders and what is an AI agent.

• Share the technical side with your product team: MCP server, OpenAPI spec, or both and what is MCP.

• Secure the door: MCP security best practices and AI agent identity and authentication.

• Decide who owns it: who owns an AI agent?.