Set up alerts

Alerts notify you when events occur in Braintrust. Send notifications to webhooks or Slack channels to catch errors, track quality issues, monitor usage patterns, or track prompt deployment changes.

Enable the Slack integration before creating an alert to send to a Slack channel.

Alert types

Braintrust supports two types of alerts:

Log alerts: Trigger when conditions are met on production logs
Environment alerts: Trigger when prompt or dataset environments are assigned or removed

Create a log alert

Go to Settings > Alerts.
Click Alert, or Create alert if you don’t have any alerts yet.
Enter alert name.
Select Log event as the event type.
Configure alert conditions:
- SQL filter: Query that defines which logs trigger the alert. See Write SQL filters for examples. When Loop is enabled, select Generate to create the filter from a natural-language description.
- Notify interval: The minimum time between notifications for this alert (5 min, 30 min, 1 hr, 4 hr, 12 hr, 24 hr).
  Braintrust evaluates logs in batches, so alerts aren’t instantaneous and a single alert can cover more than one matching log. When a batch triggers the alert, the notification counts only the matching logs in that batch, then the alert is suppressed for the length of the interval. Matching logs that arrive during that window don’t trigger another alert and aren’t reported later, so choose an interval short enough that you don’t miss what you need to act on.
Select an action type:
- Webhook: Enter the webhook URL to send a JSON payload to. Click Test next to the Webhook URL field to verify the configuration. See Test an alert.
- Slack: Select a Slack channel using the searchable dropdown. The channel list refreshes automatically every 7 days. To trigger a manual refresh, click Refresh channels. If your channel doesn’t appear, you can enter its manually.
Click Create.

You can also create log alerts directly from the Logs or Monitor pages. Apply filters to your logs, then select > Create alert from filters in the toolbar. The SQL filter automatically populates with your current filters.

Create an environment alert

Environment alerts notify you when prompt or dataset environments are updated. Use them to track deployments, monitor version changes, or trigger downstream workflows when prompts or datasets are promoted across environments.

Go to Settings > Alerts.
Click Alert, or Create alert if you don’t have any alerts yet.
Enter alert name.
Select Environment update as the event type.
Optionally filter by specific environments (e.g., only alert on “production” changes).
Select an action type:
- Webhook: Enter the webhook URL to send a JSON payload to.
- Slack: Select a Slack channel using the searchable dropdown. The channel list refreshes automatically every 7 days. To trigger a manual refresh, click Refresh channels. If your channel doesn’t appear, you can enter its manually.
Click Create.

Environment alerts trigger immediately when environments are updated. Unlike log alerts, they don’t have intervals or SQL filters.

Test an alert

Testing is available for log alerts that use the Webhook action. Click Test next to the Webhook URL field. Braintrust runs the SQL filter on recent logs. If matching logs exist, a test payload is sent. Check your webhook endpoint for the test message. Testing is not available for Slack actions or for environment alerts.

Edit or delete an alert

Go to Settings > Alerts.
Select the alert you want to change.
Update the alert configuration, then click Update.

If you close an alert with unsaved changes, Braintrust asks you to confirm before discarding them. To delete an alert, open it and click Delete in the footer, or click the delete icon in the alert’s row. Confirm the deletion in the dialog that appears.

Deleting an alert immediately stops its notifications. This cannot be undone.

Common alert patterns

Log alerts
Environment alerts

Error monitoring: Catch production issues immediately.

SQL filter: error IS NOT NULL AND metadata.environment = 'production'
Action: Post to Slack #incidents channel or create tickets in issue trackers

Quality thresholds: Track when model performance drops below acceptable levels.

SQL filter: scores.factuality < 0.8 AND metadata.environment = 'production'
Notify interval: 1 hr
Action: Send to monitoring systems or trigger automated remediation

Cost monitoring: Alert on expensive requests.

SQL filter: estimated_cost() > 1.0
Action: Webhook to cost tracking systems or budget management tools

Model-specific issues: Alert on problems with a particular model.

SQL filter: metadata.model = 'gpt-4o' AND (error IS NOT NULL OR scores.accuracy < 0.8)
Action: Post to team channel for model performance investigation

Feature-specific monitoring: Track specific workflows or user segments.

SQL filter: metadata.user_tier = 'enterprise' AND metadata.feature = 'summarization'
Action: Post to team channel for priority investigation

Combined conditions: Alert on multiple conditions for critical requests.

SQL filter: (scores.accuracy < 0.7 OR error IS NOT NULL) AND metadata.priority = 'high'
Action: Immediate Slack notification to on-call team

Usage spikes: Monitor when request volume exceeds normal levels.

Action: Use external systems to track historical rates and webhook alerts to capture spikes

Deployment tracking: Monitor when prompts are promoted to production.

Environment filter: production
Action: Post to Slack #deployments channel

Multi-environment monitoring: Track changes across all environments.

Environment filter: Leave empty to monitor all environments
Action: Webhook to deployment tracking system

Critical environment protection: Get immediate notification of staging or production changes.

Environment filter: staging, production
Action: Slack notification to team channel

Audit trail: Log all environment changes to external systems.

Environment filter: Leave empty
Action: Webhook to audit logging service

Webhook payloads

Webhooks can route alerts to any third-party incident, on-call, or automation tool that accepts inbound webhooks (for example, PagerDuty’s Events API, Opsgenie, or workflow platforms like Zapier). Braintrust delivers a fixed JSON payload (shown below) with the Content-Type: application/json header, and doesn’t support custom headers or payload formats. If the destination requires a specific payload shape or authentication headers, add a lightweight forwarder that transforms the Braintrust webhook payload before delivering it.

Webhook URLs must use http or https. To protect against server-side request forgery (SSRF), Braintrust blocks webhook delivery to private, internal, or otherwise reserved network addresses. Use Test to confirm your endpoint is reachable.

Log alerts
Environment alerts

When a log alert triggers a webhook, it sends this JSON structure:

{
  "organization": {
    "id": "org_123",
    "name": "your-organization"
  },
  "project": {
    "id": "proj_456",
    "name": "your-project"
  },
  "automation": {
    "id": "c5b32408-8568-4bff-9299-8cdd56979b67",
    "name": "High-Priority Factuality",
    "description": "Alert on factuality scores for priority logs",
    "event_type": "logs",
    "btql_filter": "metadata.priority = 0 AND scores.Factuality < 0.9",
    "interval_seconds": 3600,
    "url": "https://braintrust.dev/app/your-org/p/your-project/configuration/alerts?aid=..."
  },
  "details": {
    "is_test": false,
    "message": "High-Priority Factuality: 5 logs triggered this alert. You won't be notified again for at least 1 hour.",
    "time_start": "2025-05-12T10:00:00.000Z",
    "time_end": "2025-05-12T11:00:00.000Z",
    "count": 5,
    "related_logs_url": "https://braintrust.dev/app/your-org/p/your-project/logs?search=..."
  }
}

When an environment alert triggers a webhook, it sends this JSON structure:

{
  "organization": {
    "id": "org_123",
    "name": "your-organization"
  },
  "project": {
    "id": "proj_456",
    "name": "your-project"
  },
  "automation": {
    "id": "c5b32408-8568-4bff-9299-8cdd56979b67",
    "name": "Production Environment Changes",
    "description": "Alert when production environment is updated",
    "event_type": "environment_update"
  },
  "details": {
    "environment": {
      "slug": "production"
    },
    "object_type": "prompt",
    "prompt": {
      "id": "prompt_789",
      "slug": "summarizer"
    },
    "new_version": "8234923849293849...",
    "action": "update"
  }
}

For dataset environment changes, object_type is "dataset" and the payload includes a dataset field instead of prompt:

{
  "organization": { "id": "org_123", "name": "your-organization" },
  "project": { "id": "proj_456", "name": "your-project" },
  "automation": {
    "id": "c5b32408-8568-4bff-9299-8cdd56979b67",
    "name": "Production Environment Changes",
    "description": "Alert when production environment is updated",
    "event_type": "environment_update"
  },
  "details": {
    "environment": {
      "slug": "production"
    },
    "object_type": "dataset",
    "dataset": {
      "id": "dataset_789",
      "name": "golden-set"
    },
    "new_version": "8234923849293849...",
    "action": "update"
  }
}

The action field will be either "update" (environment assigned to object) or "delete" (environment removed from object). When action is "delete", the new_version field will be null.

Limitations

For self-hosted deployments:

Alerts are available starting with v0.0.72.
The Slack integration is available starting with v1.1.29.

For Slack workspaces with more than 100,000 channels:

The channel list shows the most recently active channels. If your target channel doesn’t appear, use the manual entry option.

Next steps

Export to cloud storage and configure data retention
View logs to understand alert triggers
Monitor deployments with dashboards
SQL reference for advanced filter queries

Start

Instrument

Observe

Annotate

Evaluate

Deploy

Admin

Best practices

Alert types

Create a log alert

Create an environment alert

Test an alert

Edit or delete an alert

Common alert patterns

Webhook payloads

Limitations

Next steps

​Alert types

​Create a log alert

​Create an environment alert

​Test an alert

​Edit or delete an alert

​Common alert patterns

​Webhook payloads

​Limitations

​Next steps

Alert types

Create a log alert

Create an environment alert

Test an alert

Edit or delete an alert

Common alert patterns

Webhook payloads

Limitations

Next steps