PromptisePromptise
Why PromptiseUse casesRoadmapDocs
GitHubGet started
Promptise - AI Framework LogoPromptise

The foundation layer for agentic intelligence. Build, secure and operate autonomous AI systems with Promptise Foundry.

pip install promptise

[01] Foundry

  • MCPcast
  • The Promptise Agent
  • Reasoning Engine
  • MCP
  • Agent Runtime
  • Prompt Engineering
  • Execution Engine
  • Agent Identity

[02] Resources

  • Documentation
  • GitHub
  • Guides
  • Learning Paths
  • Questions

[03] Company

  • About
  • Terms of Service
  • Privacy Policy
  • Cookie Policy
  • Subprocessors

© 2026 Promptise by Manser Ventures. All rights reserved.

Open source · Python

← Learning paths> Mental Models: How to Think with LLMs
Lesson 04 / 19
21%

Prompts as Code: The LLM as a Probabilistic Compiler

Treat prompts as code: write specs, enforce output shape with structured outputs, surface assumptions and test every change. Real date-extraction example.

Intermediate5 minLLM Mental ModelsPrompts as CodeStructured OutputsEvals and Testing
All 19 lessons↓
  1. ✓Are LLMs Stochastic Parrots? What the Term Gets Right
  2. ✓Can LLMs Simulate Users? Role-Play, Personas and Limits
  3. ✓Why LLMs Get Niche Facts Wrong: Popularity and Training Data
  4. 04Prompts as Code: The LLM as a Probabilistic Compiler
  5. 05Are LLMs Black Boxes? What You Can and Can't Inspect
  6. 06Why LLM Output Is Untrusted Input: Design for Injection
  7. 07Why LLMs Fail Modified Puzzles: Pattern-Matching Bias
  8. 08LLMs as Lossy Compression: Why They Can't Quote Exactly
  9. 09How LLMs Generate Text: Next-Token Probabilities Explained
  10. 10How to Brief an LLM: Context Gaps and Clarifying Questions
  11. 11How LLM Tool Calling Works: The Model Never Runs Tools
  12. 12Why Can't LLMs Count Letters? Tokens and Probability
  13. 13Does AI Boost Productivity? The LLM as Cognitive Multiplier
  14. 14How Context Shapes LLM Output: The Model Continues It
  15. 15Prompts as Interfaces: Design Prompts Like API Contracts
  16. 16LLM as a Judge vs a Jury: Making AI Evaluations Reliable
  17. 17AI Code Hallucinations: Why Plausible Output Fools You
  18. 18Why LLMs Give Similar Answers (and How to Get Variety)
  19. 19Why LLMs Mirror You: Sycophancy, Bias and Leading Prompts

Lessons in this path

  1. ✓Are LLMs Stochastic Parrots? What the Term Gets Right
  2. ✓Can LLMs Simulate Users? Role-Play, Personas and Limits
  3. ✓Why LLMs Get Niche Facts Wrong: Popularity and Training Data
  4. 04Prompts as Code: The LLM as a Probabilistic Compiler
  5. 05Are LLMs Black Boxes? What You Can and Can't Inspect
  6. 06Why LLM Output Is Untrusted Input: Design for Injection
  7. 07Why LLMs Fail Modified Puzzles: Pattern-Matching Bias
  8. 08LLMs as Lossy Compression: Why They Can't Quote Exactly
  9. 09How LLMs Generate Text: Next-Token Probabilities Explained
  10. 10How to Brief an LLM: Context Gaps and Clarifying Questions
  11. 11How LLM Tool Calling Works: The Model Never Runs Tools
  12. 12Why Can't LLMs Count Letters? Tokens and Probability
  13. 13Does AI Boost Productivity? The LLM as Cognitive Multiplier
  14. 14How Context Shapes LLM Output: The Model Continues It
  15. 15Prompts as Interfaces: Design Prompts Like API Contracts
  16. 16LLM as a Judge vs a Jury: Making AI Evaluations Reliable
  17. 17AI Code Hallucinations: Why Plausible Output Fools You
  18. 18Why LLMs Give Similar Answers (and How to Get Variety)
  19. 19Why LLMs Mirror You: Sycophancy, Bias and Leading Prompts
← Path overview

A prompt that feeds another program is code. It has inputs, an expected output type and edge cases, and like code it can be underspecified. The difference is the "compiler": an LLM fills the gaps with the most plausible guess, and guesses vary from run to run. This essay, part of the Mental Models learning path, explains how to write prompts as specifications, enforce their output shape, and test them like code, with a real example where a vague extraction prompt returns prose and a specified one returns data plus the assumptions it made.

[01]

How do you treat prompts as code?

Write the prompt as a specification: the input, the output type, and a rule for every ambiguity you can foresee. Enforce the output shape with a schema the API guarantees, and validate the meaning in code, because a schema can only promise the shape. Keep prompts in version control with a set of test cases, and rerun the cases on every change to the prompt or the model. The model works like a compiler that fills missing parts of the spec with the most likely guess.

Key takeaways:

  • Every unstated decision becomes a sample. Ambiguity in the prompt shows up as variance in the output.

  • Schemas fix shape, not meaning. Constrained decoding guarantees valid JSON; it can't guarantee the right date.

  • Make the model report its assumptions, so the ambiguous cases surface instead of being silently decided.

  • Test prompts like functions: fixed inputs, expected outputs, run on every change.

[02]

Why the compiler analogy works

A compiler turns a specification into behavior and rejects what it can't resolve. An LLM never rejects; it predicts the most likely continuation of whatever it was given. If the prompt doesn't say which date format, which time zone, or what "next Thursday" means, the model picks a plausible answer, and sampling means it may pick differently next time. That is the probabilistic part.

Two developments make the analogy practical. Structured outputs use constrained decoding: at each step, tokens that would break the supplied JSON Schema are excluded, so the response is guaranteed to parse and match the schema (OpenAI structured outputs guide; other providers offer similar modes). And prompt compilation treats the prompt itself as a build artifact. DSPy (Khattab et al., ICLR 2024) has you declare modules with input and output signatures plus a metric, then "compiles" the pipeline by searching for prompts and few-shot examples that maximize the metric on your data. You don't need DSPy to use the idea: a spec, a metric and test cases are what turn a prompt into maintainable code.

[03]

Example: extracting a meeting time

The email:

Text
Hi both, can we move the vendor review to next Thursday at 3? Berlin time, since Ana is joining from there. If that doesn't work, Friday morning is fine too. Today is Monday 12 October 2026. - Tom

An underspecified prompt:

Text
Extract the meeting time from this email.

Three samples:

Output
Next Thursday at 3 PM Berlin time (i.e., 15:00 CET/CEST).
Next Thursday at 3 PM Berlin time (i.e., 15:00 CET/CEST).
Next Thursday at 3 PM Berlin time (or Friday morning as backup).

Readable, but no program can use it. The date isn't resolved, "CET/CEST" dodges the offset, and the fallback appears in one run only.

A specified prompt, with a strict JSON Schema passed as the response format (fields: proposed_start as ISO 8601 with offset, timezone as an IANA name, fallback as string or null, assumptions as a list):

Text
Extract the proposed meeting slot from the email. Resolve relative dates against the date stated in the email. If the hour is ambiguous between AM and PM, pick the business-hours reading and record it in assumptions.
Output
{"proposed_start":"2026-10-15T15:00:00+02:00","timezone":"Europe/Berlin","fallback":"Friday morning (alternative)","assumptions":["'next Thursday' relative to 'Today is Monday 12 October 2026' means Thursday 15 October 2026","'3' interpreted as 3:00 PM (business-hours reading) in Berlin timezone","'Friday morning' left as an imprecise fallback, not converted to a specific datetime"]}

Every sample parsed and agreed on 2026-10-15T15:00:00+02:00. The assumptions field earned its place: it exposed that "next Thursday" was read as 15 October, when some readers would mean the 22nd. That is a spec bug only a human can resolve, and now it is visible. The free-text fallback field still varied between runs ("Friday morning (alternative)" and "Friday morning (2026-10-16)"), which is the cue to give it a type too. Results vary by model.

The same email through two prompts: the underspecified one returns prose that varies, with the date unresolved, the offset dodged and the fallback in one run only, while the specified one with a strict JSON Schema returns 2026-10-15T15:00:00+02:00 every time and its assumptions show that next Thursday was read as 15 October.
FigureThe same email through an underspecified prompt and a specified one with a schema
[04]

What it changes in practice

  • Write the spec before the prose. List the inputs, the output schema and the ambiguity rules: relative dates, missing values, conflicting instructions. Each rule you write is one less thing the model decides at random.

  • Enforce shape with the API, check meaning in code. Use structured outputs where available. Then validate what the schema can't express: dates in the future, totals that add up, IDs that exist. Fail loudly, like a type error.

  • Version and test prompts. Keep a few dozen real inputs with expected outputs, including edge cases, and run them on every prompt edit and model upgrade. If you use Promptise Foundry, its prompt testing utilities provide a PromptTestCase with schema and content assertions; any test runner works for the same job.

[05]

Frequently asked questions

What is a probabilistic compiler?

It is a mental model for LLMs: like a compiler, the model turns a specification (your prompt) into output, but it fills gaps with the most likely guess instead of raising an error, and sampling means the guess can change between runs. The fix is the same as for code: tighter specs and tests.

Do structured outputs guarantee correct answers?

No. They guarantee the response matches your JSON Schema: required fields present, types right, enums respected. A well-formed object can still hold a wrong date or a made-up value. Validate content separately.

What is DSPy?

DSPy is a framework from Stanford in which you declare LLM pipeline steps as modules with typed signatures and a metric, and an optimizer "compiles" them into effective prompts and examples. It applies software-engineering ideas, such as modules and optimization against tests, to prompting.

How many test cases does a prompt need?

Enough to cover the cases that matter: typical inputs, each ambiguity rule, and every bug you've fixed. For many production prompts that is 20 to 100 cases. Add each failure you find in production as a new case.

[06]

Next steps

  • Structured outputs: reliable JSON: schemas, validation and Pydantic.

  • How to test LLM prompts in Python: build a regression suite.

  • How to write clear prompts: asks, constraints and format.

  • Prompts as interfaces, not commands: the companion essay on prompt design.

  • Mental Models: How to Think with LLMs: the full learning path.

← PreviousWhy LLMs Get Niche Facts Wrong: Popularity and Training DataNext lesson →Are LLMs Black Boxes? What You Can and Can't Inspect