Contents · 10 sections↓
- 01What should you ask AI vendors about keys and supply chain?
- 02Why the usual AI vendor questionnaire misses this
- 03Which vendors need this questionnaire?
- 04The questions, by area
- 05How to score the answers
- 06How to run the review, step by step
- 07The short version to copy
- 08Common mistakes
- 09FAQ
- 10Next steps
An AI vendor security questionnaire usually asks about encryption, certifications and whether your data trains the vendor's models. It rarely asks about the thing attackers went after in 2026: the API keys a vendor holds for you, and the release process that ships its software. This guide gives leaders and security teams a ready list of questions about keys, logs, releases and dependencies, explains why each one matters, and shows what a good answer sounds like.
What should you ask AI vendors about keys and supply chain?
Ask four things: how the vendor stores and limits the AI provider keys you give it, whether those keys can end up in its logs, who can publish a new release of its software, and how it chooses and checks the code it depends on. Then ask how fast it would tell you if any of that failed, and what it would tell you. A vendor that answers these with specifics and evidence is easier to trust than one that points to a certificate.
Key takeaways:
Any tool that holds your AI keys can spend your money and see your data. Gateways, coding tools, agent platforms and monitoring tools all qualify.
Logs are a common leak path. A token inside a request or response can be saved in plain text.
Releases are a target. If an attacker can publish a release, every customer who updates installs it.
Ask for evidence, not adjectives. "We take security seriously" is not an answer; "two people must approve each release, including admins" is.
Plan the bad day now. Know which keys you would rotate, and in what order.
[02]
Why the usual AI vendor questionnaire misses this
Most published AI vendor questionnaires focus on data: training, storage, subprocessors, prompt injection. Those questions matter. But in the top-ranking templates we read in October 2026, few asked how the vendor protects the provider keys it holds for customers, and none asked whether secrets can land in its logs or how its own release pipeline is protected.
The LiteLLM incident of October 21, 2026 shows why these gaps matter. LiteLLM is a popular open-source gateway that sits between applications and AI providers and holds their keys. According to LiteLLM's own incident post, version 1.107.0 was published to PyPI, the Python package index, with malware that steals credentials. It was live from 11:39 to 13:23 UTC, and the matching Docker images were affected for longer. The post describes these steps:
developers' GitHub tokens ended up in the logs of an internal LiteLLM deployment, because a coding tool can place such tokens in response bodies and the deployment logged response bodies;
that internal deployment was reachable from the internet, and the post says the attacker reportedly took the tokens from its logs (how the attacker got in was still being confirmed);
with credentials from several developer accounts, including an organization owner, the attacker overrode branch protections, passed the release approval step and published the bad version.
LiteLLM told affected users to rotate every provider key stored in the gateway, plus other secrets on the host. See which gateways teams compared in LiteLLM alternatives.
No questionnaire stops an attack on its own. But the questions below ask about the weak points the post describes: secrets in logs, release rules that admins can bypass, and how fast customers learn what to rotate.
[03]
Which vendors need this questionnaire?
Use the full list for any vendor or tool that holds, sees or can use your AI provider keys or tokens. Typical examples:
AI gateways and routers, hosted or self-hosted, that forward requests to several model providers.
Agent platforms that call models and your business systems on your behalf.
Coding assistants and agents with a hosted console where developers save provider keys.
Monitoring, evaluation and logging tools that receive copies of prompts and responses.
Open-source packages that handle keys. There is no vendor to ask, so your own engineers answer the same questions.
Tools that never see your keys or AI traffic need only a general AI questionnaire.
[04]
The questions, by area
Send all 24 questions to vendors with deep access, and the first two or three per area to the rest.
1. Your keys in their hands
Question | Why it matters | A good answer sounds like |
|---|---|---|
Where are our provider keys stored, and who at your company can read them? | Stored keys are what attackers want. | "Encrypted in a secrets vault; no engineer can view them in plain text." |
Can we give you a key limited to one project, with its own spend cap? | A limited key caps the damage. Providers such as Anthropic offer workspace-scoped keys and spend limits. | "Yes, and our setup guide shows how." |
Do you need our keys at all, or can you work with short-lived tokens? | Short-lived tokens expire on their own. | "We support short-lived tokens or your own key vault." |
How do we rotate or revoke a key, and how fast does it take effect? | You will need it during an incident. | "Self-service in the console, effective within minutes." |
What happens to our keys when we leave? | Old keys in a former vendor's database are still keys. | "Deleted within a stated number of days, confirmed in writing." |
2. Logs and traces
Question | Why it matters | A good answer sounds like |
|---|---|---|
Do you log full request and response bodies? Can we turn that off? | Bodies can contain tokens and customer data, as in the LiteLLM case. | "Off by default, or on with redaction. You control it per project." |
Do you remove secrets from logs before they are written? | Redaction has to happen before storage, not after. | "Yes, known key formats are masked before logging, and we test it." |
Who can read logs, and how long are they kept? | Logs are often readable by more people than production systems. | "A small named group, access recorded, kept a stated number of days." |
Are any internal tools or test systems that hold logs reachable from the internet? | The LiteLLM post describes an internal deployment exposed to the internet. | "No. Internal tools sit behind single sign-on." |
3. Releases and build pipeline
Question | Why it matters | A good answer sounds like |
|---|---|---|
Who can publish a new release, and how many people must approve it? | One stolen account should not ship code to every customer. | "Two people, and admins cannot skip the rule." |
Can an administrator turn off branch protection or approval rules? Is that alerted? | In the LiteLLM case an owner token was used to disable protections. | "Only with a second approval, and every change alerts the security team." |
Do you publish with short-lived credentials instead of stored tokens? | Stored tokens can be stolen. PyPI's trusted publishing uses short-lived ones instead. | "Yes, trusted publishing; no long-lived publish tokens exist." |
Are releases signed, and can we check that a download came from your official build? | Customers can reject a release built elsewhere. | "Yes, signed, and we document how to verify." |
Can a published version or image tag be changed after release? | If tags can be moved, "latest" can silently point at new code. | "No, tags are immutable." |
4. Dependencies
Question | Why it matters | A good answer sounds like |
|---|---|---|
Do you pin exact versions of what you depend on, with checksums? | Unpinned installs pick up whatever was published last, including a bad release. | "Yes, every dependency is pinned and hash-checked." |
Do you wait before adopting brand-new versions of dependencies? | Bad releases are often caught and pulled quickly; the LiteLLM one was live on PyPI for under two hours. | "We wait a set number of days unless it is a security fix." |
Can you give us a list of the components in your product? | You need it to answer "are we affected?" next time. | "Yes, a machine-readable list for each release." |
How fast did you respond to the last compromised package that affected you? | A real example tells you more than a policy. | A date, what they did and how long it took. |
5. Their own AI tools
Question | Why it matters | A good answer sounds like |
|---|---|---|
Do your engineers use AI coding agents with access to production or release credentials? | Agents can read and repeat tokens, as the LiteLLM post describes. | "Agents get limited credentials, never release rights." |
Where does the output of those tools go, and is it logged? | Tool output that contains tokens can be stored in plain text. | "It is not logged, or it is logged with redaction." |
6. Detection, notification and exit
Question | Why it matters | A good answer sounds like |
|---|---|---|
How would you detect a stolen key or a bad release? | You want detection, not luck. | Named alerts, such as new admin tokens, protection changes or unusual key use. |
How quickly will you tell us, and what will you tell us? | The first hours decide what a leak costs. | A time in hours, plus affected versions, time windows and which keys to rotate. |
Have you had a security incident in the last two years? Is there a public write-up? | A clear write-up is a good sign, not a bad one. | Yes or no, with a link if yes. |
Can we see usage of our keys through you, so we can spot misuse? | Strange spend is often the first sign of a stolen key. | "Yes, per key and per day, with alerts." |
[05]
How to score the answers
Score each answer 0, 1 or 2:
0: no answer, a vague answer or a red flag.
1: the control exists but is manual, partial or unproven.
2: the control exists and the vendor shows evidence you can check.
Some questions are must-pass. A 0 on any of these should stop the purchase or send it to your security lead:
Keys stored in plain text or readable by staff.
Request bodies logged with no redaction and no way to turn it off.
One person, or any admin, can publish a release alone.
No commitment to tell you about an incident.
Match the depth to the access: a gateway holding every provider key you own deserves the full list and a call with its security team; a tool with one test key needs the must-pass questions.
[06]
How to run the review, step by step
List every tool that holds AI keys. Ask engineering, IT and finance; bills often reveal tools nobody listed.
Rank them by access. How many keys, which providers, what spend limits, what data passes through.
Send the questions and ask for evidence. For example, a screenshot of the release approval rule or the last incident write-up.
Start with a limited key. During the pilot, give the vendor a key scoped to one project with a low spend cap, and watch its usage.
Write your rotation plan. For each tool: which keys, who rotates them, in what order, and how you confirm the old ones stopped working. Engineers can follow what to do with a leaked LLM API key.
Set re-review triggers. Re-ask after a vendor incident, a change of ownership, or once a year.
Contract terms such as notification deadlines belong to your legal and procurement teams; ask them how to handle those.
[07]
The short version to copy
Paste these ten questions into an email or a procurement form:
Where are our AI provider keys stored, and can any person at your company read them in plain text?
Can we give you a key limited to one project with its own spend cap?
How do we rotate or revoke a key, and how fast does it take effect?
Do you log full request and response bodies, and are secrets removed before logs are written?
Who can read your logs, and how long are they kept?
How many people must approve a release, and can an administrator bypass that rule?
Do you publish releases with short-lived credentials, and are releases signed?
Do you pin and hash-check your dependencies, and do you wait before adopting new versions?
Do your engineers' AI tools have access to production or release credentials?
If you are breached, how many hours until you tell us, and will you tell us which keys to rotate?
[08]
Common mistakes
Mistake | Why it fails | Fix |
|---|---|---|
Accepting a certificate as the whole answer | An audit covers what was in scope; key handling and releases may not be | Ask the specific questions anyway |
Giving every tool an unrestricted key | One leak exposes all your spend and data | One limited key per tool, with a spend cap |
Reviewing only paid vendors | Open-source packages hold keys too | Your engineers answer the same questions for them |
Asking once and filing the answer | Vendors change pipelines, owners and tools | Re-review on triggers and yearly |
No rotation plan | Rotation during an incident is slow and error-prone | Write it per tool before go-live |
[09]
FAQ
What is an AI vendor security questionnaire?
It is a list of questions a company sends to an AI vendor before buying, to learn how the vendor protects data, keys and its own software. Good ones ask for evidence, not just yes or no.
Why ask AI vendors about API keys specifically?
Many AI tools hold the keys to your model providers. Anyone with those keys can run up charges on your accounts and, depending on the provider, reach your stored files or data. A vendor's key storage, logging and release process decide how easily those keys could leak.
What is a software supply chain attack, in plain words?
It is an attack on the way software is built and delivered. The attacker gets the vendor's release process to ship harmful code, so customers install it through a normal update. The October 2026 LiteLLM release is one example, described in the vendor's own incident post.
How many questions should an AI vendor questionnaire have?
Enough to cover the vendor's real access. For a tool that holds many provider keys, use all 24 questions here plus your general data questions. For a low-access tool, the ten-question short version is usually enough.
What should we do if a vendor that holds our keys reports an incident?
Rotate the keys that vendor held, highest spend and data access first, and confirm the old keys stopped working. Check provider usage for the incident window, then re-run this questionnaire before restoring access.
Do these questions apply to open-source tools?
Yes, but your own team answers them. Ask your engineers how they pin and check the package, whether they wait before upgrading, what it logs and which keys it holds. Dependency cooldowns and pinning shows one way to do this for Python.
[10]
Next steps
See how teams compared gateways after the incident: LiteLLM alternatives.
Give your engineers the response steps: leaked LLM API key: what to do and stop secrets leaking into LLM logs.
Understand the wider risk: LLM supply chain security.
Evaluate agent platforms more broadly: AI agent platform evaluation checklist and build vs buy AI agents.
Set up the operating rules around it: AI agent governance in practice.