Stop LLM output regressions before they ship.

Promptguard runs prompt test cases in CI and validates outputs against a JSON Schema. If your agent/tool expects JSON, Promptguard turns silent breakage into a failed PR.

What it does

Runs prompt test cases on every PR
Strict JSON parsing (no extra text)
Validates output against your JSON Schema
Fails CI when required fields/types change

Works with

OpenAI + Anthropic (v0)
GitHub Actions
Agent builders + AI app devs

GitHub Action

name: Promptguard
on: [pull_request]

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./action
        with:
          config: promptguard.yaml
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Next: pinned versions + PR comments + baselines & diffs.

Get early access

$49

planned per repo / month

1 min

to add to a workflow

We’ll only email about Promptguard. No spam.

Example config

version: 1
suites:
  - name: support_agent_json
    provider: openai
    model: gpt-4.1-mini
    temperature: 0
    prompt: |
      Output ONLY JSON.
      User: {{message}}
    schema:
      type: object
      required: [category, priority, response]
      properties:
        category: { type: string }
        priority: { type: string, enum: ["low","medium","high"] }
        response: { type: string }
      additionalProperties: false
    cases:
      - name: refund
        vars: { message: "I want a refund." }