Gate pull requests on a pass rate

1. Create an API key

   In the web app: Settings → API keys. It is shown once.

2. Store it as a secret

   In your repository: Settings → Secrets and variables → Actions → New repository secret, named VVDEX_API_KEY.

3. Add the workflow

   Save this as .github/workflows/vvdex.yml:

name: VVDex gate
on:
  pull_request:
  workflow_dispatch:

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install https://vvdexops.com/downloads/vvdex-0.2.0-py3-none-any.whl
      - name: Evaluate
        env:
          VVDEX_API_KEY: ${{ secrets.VVDEX_API_KEY }}
        run: |
          vvdex eval run --forge vvdex.ops.gateway-incident-1 --attempts 3 \
            --model con_…:gpt-4.1 \
            --wait --fail-under 0.6

To test your own prompts instead, replace --forge … --attempts 3 with --dataset YOUR_DATASET_ID --grader exact (or the graders your dataset needs). Repeat --model to test several models; every one must clear the threshold.

What the job does

• Starts the run, prints progress, and waits (up to an hour by default; change it with --timeout).
• Prints each model’s pass rate with its 95% interval, errors by kind, cost and latency.
• Writes the same table to the job summary, with a link to the run.
• Sets step outputs eval-id, status and gate (pass or fail).
• Exits 1 if any model is below the threshold or has no scored answers, and 2 if the run failed, was cancelled or timed out.

How many attempts?

Each Forge attempt is a full working session in a fresh sandbox, so a few attempts take minutes, not seconds. Three attempts give a first signal; more narrow the 95% range. An attempt that fails because the model’s server broke is an endpoint error: shown apart and never counted as a wrong answer. A run with no scored attempt fails the gate.

Composite action

The SDK source also contains a composite action, sdk/github-action/action.yml, that wraps the same steps. Its inputs are api-key, models (one per line), forge (with attempts) or dataset (with graders, one per line), name, fail-under (default 0.8), timeout and base-url; its outputs are eval-id, status and gate.