Skip to content

Commit 80b7160

Browse files
committed
docs: add Braintrust Python integration page
1 parent cdd3de6 commit 80b7160

1 file changed

Lines changed: 261 additions & 0 deletions

File tree

Lines changed: 261 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,261 @@
1+
---
2+
id: braintrust
3+
title: Braintrust Integration
4+
sidebar_label: Braintrust Integration
5+
toc_max_heading_level: 2
6+
keywords:
7+
- ai
8+
- agents
9+
- braintrust
10+
- observability
11+
- tracing
12+
- prompts
13+
tags:
14+
- Braintrust
15+
- Python SDK
16+
- Temporal SDKs
17+
description: Add LLM observability and prompt management to Python Workflows using the Temporal Python SDK and Braintrust.
18+
---
19+
20+
Temporal's integration with [Braintrust](https://braintrust.dev) gives you full observability into your AI agent Workflows—tracing every LLM call, managing prompts without code deploys, and tracking costs across models.
21+
22+
When building AI agents with Temporal, you get durable execution: automatic retries, state persistence, and the ability to recover from failures mid-workflow.
23+
Braintrust adds the observability layer: see exactly what your agents are doing, iterate on prompts in a UI, and measure whether changes improve outcomes.
24+
25+
The integration connects these capabilities with minimal code changes.
26+
Every Workflow and Activity becomes a span in Braintrust, and every LLM call is traced with inputs, outputs, tokens, and latency.
27+
28+
All code snippets in this guide are taken from the [deep research sample](https://docs.temporal.io/ai-cookbook).
29+
Refer to the sample for the complete code and run it locally.
30+
31+
## Prerequisites
32+
33+
- This guide assumes you are already familiar with Braintrust. If you aren't, refer to the [Braintrust documentation](https://www.braintrust.dev/docs) for more details.
34+
- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
35+
- Ensure you have set up your local development environment by following the [Set up your local development environment](/develop/python/core-application) guide.
36+
When you're done, leave the Temporal Development Server running if you want to test your code locally.
37+
38+
## Configure Workers to use Braintrust
39+
40+
Workers execute the code that defines your Workflows and Activities.
41+
To trace Workflow and Activity execution in Braintrust, add the `BraintrustPlugin` to your Worker.
42+
43+
Follow the steps below to configure your Worker.
44+
45+
1. Install the Braintrust SDK with Temporal support.
46+
47+
```bash
48+
uv pip install "braintrust[temporal]"
49+
```
50+
51+
2. Initialize the Braintrust logger before creating your Worker.
52+
The logger must be initialized first so that spans are properly connected.
53+
54+
```python
55+
import os
56+
from braintrust import init_logger
57+
58+
# Initialize BEFORE creating the Temporal client or worker
59+
init_logger(project=os.environ.get("BRAINTRUST_PROJECT", "my-project"))
60+
```
61+
62+
3. Add the `BraintrustPlugin` to your Worker.
63+
64+
```python
65+
from braintrust.contrib.temporal import BraintrustPlugin
66+
from temporalio.worker import Worker
67+
68+
worker = Worker(
69+
client,
70+
task_queue="my-task-queue",
71+
workflows=[MyWorkflow],
72+
activities=[my_activity],
73+
plugins=[BraintrustPlugin()], # Add this line
74+
)
75+
```
76+
77+
4. Add the plugin to your Temporal Client as well.
78+
This enables span context propagation, linking client code to the Workflows it starts.
79+
80+
```python
81+
from temporalio.client import Client
82+
from braintrust.contrib.temporal import BraintrustPlugin
83+
84+
client = await Client.connect(
85+
"localhost:7233",
86+
plugins=[BraintrustPlugin()],
87+
)
88+
```
89+
90+
5. Run the Worker.
91+
Ensure the Worker process has access to your Braintrust API key via the `BRAINTRUST_API_KEY` environment variable.
92+
93+
```bash
94+
export BRAINTRUST_API_KEY="your-api-key"
95+
python worker.py
96+
```
97+
98+
:::tip
99+
100+
You only need to provide API credentials to the Worker process.
101+
The client application that starts Workflow Executions doesn't need the Braintrust API key.
102+
103+
:::
104+
105+
## Trace LLM calls with wrap_openai
106+
107+
The simplest way to trace LLM calls is to wrap your OpenAI client.
108+
Every call through the wrapped client automatically creates a span in Braintrust with inputs, outputs, token counts, and latency.
109+
110+
```python
111+
from braintrust import wrap_openai
112+
from openai import AsyncOpenAI
113+
114+
# Wrap the client - all calls are now traced
115+
# max_retries=0 because Temporal handles retries
116+
client = wrap_openai(AsyncOpenAI(max_retries=0))
117+
```
118+
119+
Use this client in your Activities:
120+
121+
```python
122+
from temporalio import activity
123+
124+
@activity.defn
125+
async def invoke_model(prompt: str) -> str:
126+
client = wrap_openai(AsyncOpenAI(max_retries=0))
127+
128+
response = await client.chat.completions.create(
129+
model="gpt-4o",
130+
messages=[
131+
{"role": "system", "content": "You are a helpful assistant."},
132+
{"role": "user", "content": prompt},
133+
],
134+
)
135+
136+
return response.choices[0].message.content
137+
```
138+
139+
After running a Workflow, you'll see a trace hierarchy in Braintrust:
140+
141+
```
142+
my-workflow-request (client span)
143+
└── temporal.workflow.MyWorkflow
144+
└── temporal.activity.invoke_model
145+
└── Chat Completion (gpt-4o)
146+
```
147+
148+
## Add custom spans for application context
149+
150+
Add your own spans to capture business-level context like user queries, workflow inputs, and final outputs.
151+
152+
```python
153+
from braintrust import start_span
154+
155+
async def run_research(query: str):
156+
with start_span(name="research-request", type="task") as span:
157+
span.log(input={"query": query})
158+
159+
result = await client.execute_workflow(
160+
ResearchWorkflow.run,
161+
query,
162+
id=f"research-{uuid.uuid4()}",
163+
task_queue="research-task-queue",
164+
)
165+
166+
span.log(output={"result": result})
167+
return result
168+
```
169+
170+
## Manage prompts with load_prompt
171+
172+
Braintrust lets you manage prompts in a UI and deploy changes without code deploys.
173+
The workflow is:
174+
175+
1. **Develop** prompts in code, see results in Braintrust traces
176+
2. **Create** a prompt in the Braintrust UI from your best version
177+
3. **Evaluate** different versions using Braintrust's eval tools
178+
4. **Deploy** by pointing your code at the Braintrust prompt
179+
5. **Iterate** in the UI—changes go live without code deploys
180+
181+
To load a prompt from Braintrust in your Activity:
182+
183+
```python
184+
import braintrust
185+
from temporalio import activity
186+
187+
@activity.defn
188+
async def invoke_model(prompt_slug: str, user_input: str) -> str:
189+
# Load prompt from Braintrust
190+
prompt = braintrust.load_prompt(
191+
project=os.environ.get("BRAINTRUST_PROJECT", "my-project"),
192+
slug=prompt_slug,
193+
)
194+
195+
# Build returns the full prompt configuration
196+
built = prompt.build()
197+
198+
# Extract system message
199+
system_content = None
200+
for msg in built.get("messages", []):
201+
if msg.get("role") == "system":
202+
system_content = msg["content"]
203+
break
204+
205+
client = wrap_openai(AsyncOpenAI(max_retries=0))
206+
207+
response = await client.chat.completions.create(
208+
model="gpt-4o",
209+
messages=[
210+
{"role": "system", "content": system_content},
211+
{"role": "user", "content": user_input},
212+
],
213+
)
214+
215+
return response.choices[0].message.content
216+
```
217+
218+
:::tip
219+
220+
Provide a fallback prompt in your code for resilience.
221+
If Braintrust is unavailable, your Workflow continues with the hardcoded prompt.
222+
223+
```python
224+
DEFAULT_SYSTEM_PROMPT = "You are a helpful assistant."
225+
226+
try:
227+
prompt = braintrust.load_prompt(project="my-project", slug="my-prompt")
228+
system_content = extract_system_message(prompt.build())
229+
except Exception as e:
230+
activity.logger.warning(f"Failed to load prompt: {e}. Using fallback.")
231+
system_content = DEFAULT_SYSTEM_PROMPT
232+
```
233+
234+
:::
235+
236+
## Example: Deep Research Agent
237+
238+
The [deep research sample](https://docs.temporal.io/ai-cookbook) demonstrates a complete AI agent that:
239+
240+
- Plans research strategies
241+
- Generates search queries
242+
- Executes web searches in parallel
243+
- Synthesizes findings into comprehensive reports
244+
245+
The sample shows all integration patterns: wrapped OpenAI client, BraintrustPlugin on Worker and Client, custom spans, and prompt management with `load_prompt()`.
246+
247+
To run the sample:
248+
249+
```bash
250+
# Terminal 1: Start Temporal
251+
temporal server start-dev
252+
253+
# Terminal 2: Start the worker
254+
export BRAINTRUST_API_KEY="your-api-key"
255+
export OPENAI_API_KEY="your-api-key"
256+
export BRAINTRUST_PROJECT="deep-research"
257+
uv run python -m worker
258+
259+
# Terminal 3: Run a research query
260+
uv run python -m start_workflow "What are the latest advances in quantum computing?"
261+
```

0 commit comments

Comments
 (0)