Contents · 8 sections↓
PAL and Program-of-Thought (PoT) are prompting techniques where the model writes a short program that solves the problem, and a Python interpreter, not the model, computes the answer. They fix a specific weakness: models often set up a calculation correctly and then get the arithmetic wrong. This guide explains what the two papers proposed, how they differ, and when code-based reasoning beats chain-of-thought. It's for developers who already know basic prompting. A real word problem shows the failure and the fix.
What are PAL and Program-of-Thought prompting?
Both ask the model to express its reasoning as code and hand the computation to an interpreter. PAL: Program-aided Language Models (Gao et al., 2022, ICML 2023) has the model turn a problem into Python steps and "offloads the solution step to a runtime such as a Python interpreter". Program of Thoughts Prompting (Chen et al., 2022, TMLR 2023) makes the same move to "disentangle computation from reasoning" for numerical tasks, including financial question answering. The model decides what to compute. The interpreter computes it.
Key takeaways:
Use it when the hard part is computation: money, dates, counting, units, many numbers.
The model still has to understand the problem. Code fixes arithmetic slips, not a misread question.
You must run the code somewhere. That means a sandbox or a hosted code tool, never exec() in your app process.
Reasoning models narrow the gap on simple math, but code is still exact, cheap and checkable.
[02]
How PAL and PoT work
The flow has three steps: the model writes a program, an interpreter runs it, and your code reads the answer from its output.
The prompt asks for code instead of an answer. Each quantity from the problem becomes a named variable, each reasoning step becomes a line, and the result goes into a known variable or a printed line. Your code runs the program and reads the result.
Chain-of-thought keeps everything in text: the model writes "2112 − 137 = 1975" and has to get each operation right while it writes. With PAL or PoT, the model only writes undamaged_boxes = total_boxes - damaged_boxes. A wrong digit can't happen in that line, because the model never computes the digit.
How the two papers differ
The ideas are close enough that people use the names interchangeably. The differences are in emphasis and evaluation:
PAL (Gao et al.) | PoT (Chen et al.) | |
|---|---|---|
Core idea | Model writes Python as its reasoning steps; interpreter solves | Model expresses reasoning as a program; computation is "relegated to an external computer" |
Prompting | Few-shot examples of problems solved as code, with comments | Few-shot and zero-shot; meaningful variable names; library calls such as SymPy for equations |
Evaluated on | 13 math, symbolic and algorithmic tasks, including GSM8K and BIG-Bench Hard | 5 math word-problem sets and 3 financial QA sets (FinQA, ConvFinQA, TATQA) |
Reported result | PAL with Codex beat PaLM-540B with chain-of-thought on GSM8K by 15 points (top-1) | About 12% average gain over chain-of-thought across the evaluated datasets |
Add-ons | | Combined with self-consistency (sample several programs, vote) for further gains |
Those numbers come from 2022 models (mainly Codex) and benchmark setups. Treat them as evidence the idea works, not as the gain you'll see with today's models.
[03]
Worked example: a warehouse problem
The problem:
A warehouse has 17 pallets of 48 boxes, 23 pallets of 36 boxes and 9 pallets of 52 boxes.
6.5% of all boxes are damaged (round down to whole boxes). The undamaged boxes are packed
into containers of 140. How many containers are completely full, and how many boxes are left over?The correct answer is 14 full containers and 15 boxes left over (1,975 undamaged boxes; 14 × 140 = 1,960).
Asking directly
With the model's reasoning set to minimal and the question asked as is:
Total boxes = 17×48 + 23×36 + 9×52 = 816 + 828 + 468 = 2112. Damaged = floor(0.065×2112) = floor(137.28) = 137. Undamaged = 2112 − 137 = 1975. Full 140-box containers = 1975 ÷ 140 = 14 remainder 35.
Answer: 14 full containers, 35 boxes left over.The setup is right. Every step until the last one is right. Then the remainder is wrong: 1975 − 1960 is 15, not 35. That is the failure PAL was designed for: correct decomposition, wrong arithmetic.
Asking for a program
Solve the problem by writing a Python program. Don't compute anything yourself.
- Give each quantity from the problem a variable with a meaningful name.
- Add a short comment for each step of the reasoning.
- Store the final result in a variable named `answer`.
Return only the code, no explanation.
Problem: <the warehouse problem>The model returned (trimmed):
# number of pallets of each type and boxes per pallet
pallets_type1 = 17
boxes_per_pallet_type1 = 48
…
total_boxes = total_boxes_type1 + total_boxes_type2 + total_boxes_type3
# percentage damaged and compute damaged boxes (rounded down)
damage_rate = 6.5 / 100
damaged_boxes = int(total_boxes * damage_rate) # round down to whole boxes
# compute undamaged boxes
undamaged_boxes = total_boxes - damaged_boxes
# container capacity and how many full containers and leftover boxes
container_capacity = 140
full_containers = undamaged_boxes // container_capacity
leftover_boxes = undamaged_boxes % container_capacity
answer = (full_containers, leftover_boxes)Running it prints (14, 15), the correct answer. The program also documents the reasoning: you can read damaged_boxes = int(total_boxes * damage_rate) and check that "round down" was handled.
Same model, same setting, different job. Results vary by model; with more reasoning effort the direct answer is more often right, at the cost of more output tokens.
[04]
When to use code-based reasoning (and when not to)
Use PAL or PoT for:
Arithmetic over more than a few numbers: invoices, totals, percentages, unit conversions.
Dates and times: days between dates, business days, time zones.
Counting, sorting and filtering a list you put in the prompt.
Iterative processes: interest month by month, inventory over time, simulations.
Table questions where the answer is a computation over cells (the PoT financial-QA case).
Don't use it for:
Questions that are mostly reading or judgment ("is this clause risky?"). There's nothing to compute.
A single simple operation. The round trip to an interpreter costs more than it saves.
Cases where you can't run code safely. Generated code is untrusted input.
If the model needs computation only now and then, a calculator or code-execution tool that the model can choose to call is often simpler than forcing every answer into a program.
[05]
PAL and PoT vs chain-of-thought vs tools
Approach | Who computes | Good for | Weak spot |
|---|---|---|---|
Chain-of-thought | The model, in text | Multi-step logic, explanations | Arithmetic slips, long calculations |
PAL / PoT | An interpreter running the model's code | Exact math, dates, iteration | Wrong setup gives a confidently wrong result; needs a sandbox |
Calculator or code tool | An interpreter, when the model decides | Agents and chats that compute occasionally | The model may skip the tool and compute in its head |
Code-based reasoning and chain-of-thought combine well. Ask for a short plan in comments, then the code. The comments are the reasoning; the code is the computation.
[06]
Common mistakes
Mistake | Why it fails | Fix |
|---|---|---|
Letting the model also state the answer in text | It can disagree with its own code | Read the answer only from the program's output |
No output contract | Your code can't find the result | Require one variable (answer) or one printed JSON line |
Floats for money | 0.1 + 0.2 errors creep into cents | Ask for decimal.Decimal and round once at the end |
Running generated code with exec() in your service | The code can read files, secrets and the network | Run it in an isolated sandbox with no network and a timeout |
Trusting the result because it came from code | A wrong variable or misread condition still runs | Check the setup lines, or generate twice and compare |
[07]
Frequently asked questions
What is the difference between PAL and Program-of-Thought?
Little in practice. Both have the model write a program and an interpreter run it. PAL (Gao et al.) was evaluated on math, symbolic and algorithmic reasoning with few-shot code examples. PoT (Chen et al.) emphasized separating computation from reasoning for numerical and financial QA, tested zero-shot too, and added self-consistency voting.
Do I still need PAL with reasoning models?
Less often for simple math, since reasoning models check their work internally. Code still wins when there are many numbers, when you need exact cents or dates, or when you want an answer you can audit line by line. It's also usually cheaper than long hidden reasoning.
Is PAL the same as giving the model a code interpreter tool?
Close. PAL makes code the required output for every answer. A code interpreter tool lets the model decide when to write and run code, inside a chat or agent loop. Use PAL for a fixed calculation pipeline and a tool when computation is occasional.
Is it safe to run code the model writes?
Only in a sandbox. Treat generated code like code from an anonymous user: no network, no secrets, read-only filesystem, CPU, memory and time limits. The companion guide shows a working setup.
Can PAL use libraries like pandas or SymPy?
Yes. PoT's paper used SymPy for equation solving. Allow only libraries installed in your sandbox image, and tell the model which ones are available, or it will import things that aren't there.
[08]
Next steps
Run model-written code safely, with a Docker sandbox and a rerun check: Program-of-Thought in Python: run model code safely.
The text-only alternative: chain-of-thought prompting.
Sample several answers and vote: self-consistency and least-to-most.
Agents that act by writing code: code-action agents in Python.
Sandboxing for agents: AI agent code execution sandbox.