From install
to your first eval.

Install

You need Node.js 20+. Git is needed only for delivery checks. MIT licensed, no runtime dependencies.

npm install -g jevtool
jevtool --help
jevtool demo

No install at all: the online runner evaluates a suite in your browser with your own key. Or run any command without installing: npx jevtool eval --examples. Semantic commands need a key from TypeSafe:

export TYPESAFE_API_KEY=...

The demo creates a temporary Git repository and runs a real failing login test. It needs no key and sends no network request.

Evaluate a judgment

Start from one of the ten bundled suites, run it, then replace its cases with your own labeled data.

jevtool eval --examples
jevtool eval example:support-triage
jevtool eval example:support-triage --save my-suite.json

A suite is one JSON file: the questions exactly as your application sends them, and cases with the answers you expect.

{
  "name": "support-triage",
  "model": "jev-1.13.0",
  "target": 0.95,
  "questions": {
    "category": { "type": "choice", "instructions": "What does the customer who wrote `email` mainly want?",
      "criteria": { "refund": "Money back for a specific past charge.", "howto": "Learn how to do something.", "other": "Anything else." } },
    "escalate": { "type": "noul", "instructions": "Does `email` threaten a chargeback or legal action?" }
  },
  "cases": [
    { "id": "s1", "state": { "email": "Charged twice, refund one please." },
      "expect": { "category": "refund", "escalate": false }, "tags": ["en"] }
  ]
}

expect takes an option or a list of acceptable options for a Choice, true/false for a Noul, and a 0-based level for a Score. Use tags to split accuracy, for example by language or source.

jevtool eval my-suite.json --compare jev-1.14.0     # answers that change on a new version
jevtool eval my-suite.json --order-check            # Choices re-asked with options reversed
jevtool eval my-suite.json --min-accuracy 0.9 --json --output eval.json   # CI gate

For each question you get accuracy, accuracy per tag, every miss with its certainty, and a suggested cutoff: the lowest certainty at which the automatically handled cases still reach target. Send the cases below it to a person or a larger model. Confident misses usually mean the criteria are read literally; add the missing boundary and re-run the whole suite to check nothing regressed.

Each case's state and the questions are sent to TypeSafe. Labels stay on your machine.

Decisions API

One endpoint for every System One decision model: Jev direct from TypeSafe, plus everything on OpenRouter's Decisions catalog (Mercury Decide, Clef, Solar Decide, Kev, Decider and more). Bring your own key: send your TypeSafe or OpenRouter API key, and it is forwarded to that provider for this request only. jevtool does not resell model access, store keys or log requests. Free.

curl https://jevtool.dev/v1/decisions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "inception/mercury-decide",
    "models": ["typesafe/jev-1.13", "cloudflare/clef-flash"],
    "state": {"ticket": "Charged twice, please refund one."},
    "questions": {
      "team": {"type": "choice", "instructions": "Which team owns `ticket`?",
               "criteria": {"billing": "Money", "support": "How-to", "bugs": "Broken"}},
      "urgent": {"type": "noul", "instructions": "Is `ticket` urgent?"}
    }
  }'

• Routing. A model ID with a slash (inception/mercury-decide, typesafe/jev-1.13) goes to OpenRouter and needs an OpenRouter key. One without (jev-latest, jev-1.14.0) goes to TypeSafe and needs a TypeSafe key. Override with X-Provider: typesafe|openrouter.
• Fallbacks. models lists up to four backups, tried in order if a model errors. The response's fallback field says which one answered. A rejected key or an empty balance stops the chain.
• Response. The provider's answers and usage, plus id and provider. Errors are {"error": {"code", "message"}} with 400, 401, 402, 404, 413, 415, 424 or 429.
• TypeSafe SDKs. POST /v1/systemone is the same endpoint, so the official Python and JavaScript SDKs work by setting TYPESAFE_BASE_URL=https://jevtool.dev.
• Models. GET https://jevtool.dev/v1/models lists every callable model and its input price. The leaderboard shows how each one does on the same labeled suites.
• Browsers. CORS is open, so a front end can call it with a key the user supplies. Never ship your own key in client code.

Claude Code plugins

claude plugin marketplace add jevtool/claude-plugins
claude plugin install jev-guard@jevtool
claude plugin install jev-eval@jevtool
claude plugin install jev-verify@jevtool

jev-guard scores every shell command, file write and URL fetch before it runs and blocks or asks about risky ones. It never auto-approves, fails open without a key, and logs decisions to ~/.jevtool/guard.jsonl. Set JEV_GUARD_MODE=ask to turn blocks into prompts. jev-eval and jev-verify are skills that teach Claude to run jevtool eval and jevtool verify at the right moments. Details: github.com/jevtool/claude-plugins.

Check an agent delivery

1. Open your project's Git repository. It must have at least one commit.
2. Create a checklist, then edit its requirements, paths and test commands.
3. Run the checks after your agent finishes the task.

jevtool init --task "Reject blank passwords without changing styles"
# Edit .jevtool/acceptance.json
jevtool verify
jevtool verify --run-tests

Download an example acceptance file and adapt its paths to your project. A requirement can ask for semantic evidence, expected file changes, unchanged paths or successful tests.

Add Jev judgments

Set TYPESAFE_API_KEY in your environment using a key from TypeSafe, then run:

jevtool verify --run-tests --judge
jevtool verify --run-tests --judge --json --output .jevtool/report.json

--run-tests executes the commands you configured. --judge sends your task, selected diff excerpts and linked test output to TypeSafe. Common credentials are redacted, but this is not a comprehensive secret scanner. Local checks make no model request.

Tests must be freshly executed. A failing linked test overrides model approval. Missing evidence and unavailable judgments stay open for review. The report applies to the recorded code snapshot.

Checking committed changes

The default base is HEAD, so it checks your current uncommitted delivery. For a committed PR, supply its base commit:

jevtool verify --base <base-commit-sha> --run-tests --judge --json

Filter records by meaning

Download the sample payment events, set TYPESAFE_API_KEY, and try:

jevtool grep "Payment succeeded but membership was not activated" payments.jsonl --format jsonl

cat events.jsonl | jevtool grep "Customer requests a refund" --format jsonl --json

The filter sends each supplied record to TypeSafe. Results retain the original text and line numbers. Uncertain records appear separately for review. Text and JSONL are supported; multiline CSV and directory scanning are not included.

Start small: up to 1000 records per run, 2000 characters per record. Prefilter large logs by time and fields before using semantic checks.

Read the result

A supported check describes evidence support, not a guarantee of code correctness. Initial thresholds need testing on your own examples. Start with advisory checks before making them a CI gate.

Eval accuracy describes your suite on the model version you ran. It is a measurement, not a guarantee for unseen data.

Full CLI reference · npm · Offline archive · Checksum