Getting Started

What is Vouqis Verify?

Vouqis Verify is a CLI that runs your evals on every pull request that touches AI code — prompts, evals, model config, agent logic — and posts a clear verdict to the PR before your team merges.

One GitHub Action step. No SDK changes. No infrastructure to run. Eval evidence on every merge.

Installation

# macOS / Linux
pip3 install vouqis-verify

# Windows
pip install vouqis-verify

Quick Start

vouqis init          # create vouqis.yml
vouqis doctor        # verify config + GitHub token
vouqis verify        # run evals and post to PR

How it works

PR opened or updated
  ↓
Vouqis detects changed AI paths
  ↓
Your eval script runs against the diff
  ↓
Verdict posted to PR (BLOCK / WARN / SAFE)

Verdicts

Three outcomes. No ambiguity.

Every Vouqis run produces one of three verdicts. The verdict is posted as a GitHub check status so the merge button reflects the eval result automatically.

BLOCK_MERGEBlock Merge

The eval score fell below the configured threshold. The PR comment is posted as a failing check — the merge button is blocked until the score improves or the threshold is explicitly overridden.

MERGE_WITH_WARNINGMerge With Warning

The eval passed but the score dropped from the baseline. The PR is not blocked, but the comment flags the regression with the delta so the team can decide whether to proceed.

SAFE_TO_MERGESafe to Merge

The eval score meets or exceeds the threshold and shows no regression from baseline. The PR comment shows the evidence and marks the check as passed.

CLI Reference

vouqis commands

vouqis initCreate a vouqis.yml config file in the current directory. Prompts for eval script path and threshold.

vouqis verifyDetect changed AI paths, run evals, compute the verdict, and post the result to the open PR.

vouqis doctorCheck that the config file, eval script, and GitHub token are all valid before running in CI.

Configuration

vouqis.yml

Place vouqis.yml at the repository root. Run vouqis init to generate it interactively.

ai_paths:
  - prompts/
  - evals/
  - src/agents/

eval:
  run: python evals/run.py
  score_key: accuracy
  threshold: 0.80

report:
  post_to_pr: true
  block_on_fail: true

ai_paths lists the directories Vouqis watches. A PR that touches none of these paths skips the eval entirely. score_key is the key Vouqis reads from your eval output JSON.