VVDex
Open the app
Menu

Quickstart · Python

Python SDK and vvdex CLI

One package, vvdex: a client that mirrors the REST API, and a command line for your terminal and CI.

Install

shell
$ pip install https://vvdexops.com/downloads/vvdex-0.2.0-py3-none-any.whl                 # Python 3.10 or newer; depends only on httpx
$ export VVDEX_API_KEY=vvx_…        # Settings → API keys in the web app

From Python

python
from vvdex import Client

client = Client()   # reads VVDEX_API_KEY; base_url defaults to https://vvdexops.com

# one prompt, several models, side by side
out = client.playground("Name three prime numbers.",
                        models=["con_…:gpt-4.1", "con_…:claude-sonnet-4-5"])
for r in out["results"]:
    print(r["model"], r["ok"], r["text"], r["cost_usd"])   # cost_usd is None when unknown

# a Forge exam, three attempts, waited for
run = client.start_evaluation(forge="vvdex.ops.gateway-incident-1", attempts=3,
                              models=["con_…:gpt-4.1"])
run = client.wait_for_evaluation(run["id"])
print(run["status"], run.get("summary"))

# the receipt, checked by VVDex
receipt = client.evaluation_receipt(run["id"])
print(client.verify_receipt(receipt)["valid"])

A model is "CONNECTION:MODEL" (split at the first colon, so con_…:llama3:8b works) or a dict {"connection_id": …, "model": …}. Every method returns the server’s JSON unchanged.

Methods

AreaClient methods
Accountme() · usage() · plans() · audit() · status()
Modelsproviders() · connections() · create_connection(provider, api_key=, label=, base_url=) · check_connection(id, model) · models() · add_model(ref, display_name=, input_usd_per_m=, output_usd_per_m=)
Playgroundplayground(prompt, models=, messages=, system=, temperature=, max_tokens=, json_mode=) · playground_history() · share_playground(id)
Datasetsdatasets() · create_dataset(name) · add_rows(id, rows) · upload_dataset(id, path) · dataset(id) · graders()
Evaluationsstart_evaluation(models=, forge= | dataset_id=, attempts=, graders=, judge=, repeats=) · evaluation(id) · wait_for_evaluation(id, timeout=) · evaluation_results(id, status=) · compare_evaluations(a, b) · cancel_evaluation(id) · evaluation_receipt(id) · evaluation_csv(id)
Proofverify_receipt(receipt) · public_key() · forge_exams() · leaderboards()

Refusals raise AuthError (401/403), NotFound (404), LimitReached (429) or VVDexError; each carries status and the server’s detail.

From the command line

shell
$ vvdex models                                  # your connections and saved models
$ vvdex forge                                   # the certified Forge exams
$ vvdex playground --model con_…:gpt-4.1 --model con_…:claude-sonnet-4-5 "Say hi in five words"
$ vvdex eval run --forge vvdex.ops.gateway-incident-1 --model con_…:gpt-4.1 --attempts 3 --wait
$ vvdex eval run --dataset ds_… --grader exact --model con_…:gpt-4.1 --fail-under 0.8
$ vvdex eval get RUN_ID
$ vvdex compare RUN_A RUN_B
$ vvdex receipt verify receipt.json
$ vvdex status

A dataset run needs at least one --grader: a type such as exact, or a type with JSON settings such as 'regex:{"pattern": "^[0-9]+$"}'; GET /api/v1/graders lists them. Forge exams bring their own hidden tests.

Add --json to any command for the raw reply. --api-key and --base-url override VVDEX_API_KEY and VVDEX_BASE_URL.

Gating on a pass rate

--fail-under 0.8 waits for the run and checks every model: each must have a pass rate of at least 80%. A model with no scored answers fails the gate, because nothing was measured. Errors (auth, network, timeout, rate limit) are never counted as wrong answers, but they can leave a model with nothing scored.

Exit codeMeaning
0Done; every model cleared --fail-under, or no gate was set
1A model scored below --fail-under, or a receipt did not verify
2The request could not be done: bad input, an API refusal, or a run that failed, was cancelled or timed out

Inside GitHub Actions the CLI also writes a results table to the job summary and sets the outputs eval-id, status and gate. See GitHub Action.

Checking receipts offline

shell
$ pip install "vvdex[offline] @ https://vvdexops.com/downloads/vvdex-0.2.0-py3-none-any.whl"
$ vvdex receipt public-key > vvdex-receipts.pem
$ vvdex receipt verify receipt.json --offline --public-key vvdex-receipts.pem

Offline, the CLI recomputes the digest, every result’s hash and their root, and checks the Ed25519 signature on your machine. Re-grading answers happens on the online check. Details on the receipts page.