Give a support request to classify Billing, Access or Shipping under the stated policy. Compare clear issues with mixed requests and missing details before using the rule.
Inspect a classification rubricLive API
The decision
How does Jev classification rubric work?
Give the model a policy with distinct labels and an explicit fallback. This original demo uses support requests to make disagreements easy to inspect. The chosen label is the highest-probability option for the question; it is not a human-approved training label.
Sutro's jev-align inspired this example. Its documented workflow evaluates rows, asks for labels and uses a separate reflection model with GEPA to propose a changed definition. A person can accept or reject that proposal. This page exercises only the typed classification step; it does not run the CLI, optimize a prompt or update model weights.
Change one rule at a time and rerun the same difficult request. Keep your intended label separately and compare it with the output. A concentrated distribution can still be wrong if the policy is incomplete. Mixed requests need a stated tie-breaking rule or a review outcome; confident mistakes should remain in an audit sample.
Original workflow diagram; not a screenshot of a third-party application.
How it works
Try the workflow in four steps.
Keep the supplied data, the model decision and the application action separate.
01
Write the rubric
Name each category and say what belongs in it. Include an outcome for insufficient context.
02
Try difficult rows
Test a clear case, mixed request and incomplete text before using a larger dataset.
03
Inspect the distribution
Read the leading label and alternatives. Do not treat a probability as a verified accuracy rate.
04
Record human corrections
Keep reviewed labels outside this demo. Re-test a changed rubric before accepting it.
Our API test
What the six original inputs returned.
What the six original inputs returned.
Original input
Observed label
Leading probability
Duplicate charge
Billing
100%
Sign-in failure
Access
100%
Missing parcel
Shipping
100%
Two equal issues
Needs review
100%
Missing context
Needs review
88%
Quoted instruction
Billing
93%
Six live Jev requests on September 26, 2026. Synthetic examples, not an independent accuracy benchmark. Fresh requests may return different distributions.Verified
Understand the results
The two equally important issues returned Needs review
The two equally important issues returned Needs review.
Read the example notes
The vague account message also returned Needs review. The quoted command to choose Shipping did not change the billing label in this one test. These outcomes illustrate the supplied fallback policy; six synthetic requests cannot establish protection against every misleading input or measure general accuracy.
Try a case with two equal issues
To repeat the check, choose Two equal issues, inspect Decision question and Candidate labels, then run the request.
Read the example notes
Now edit the question to give login problems explicit priority and run again. Compare both distributions against the rule you intended to express. Record the reviewed outcome separately; the site does not save corrections.
A change in output is not proof that the new rubric is better
A change in output is not proof that the new rubric is better.
Read the example notes
Keep previously correct examples in the comparison and separate data used to revise a rule from data reserved for evaluation. In the full jev-align workflow, feedback and proposal acceptance happen outside this page.
The optional local script turns the saved distributions into a three-row review file
The optional local script turns the saved distributions into a three-row review file.
Read the example notes
It ranks by one minus the largest Choice probability and also takes one seeded random audit row. This is our explicit sample-selection rule, not the jev-align algorithm or a universal threshold. Fill reviewed_label yourself after inspection; the script leaves every annotation blank.
# Standard library only. Saved real outputs; no new API call.
import csv
import json
import random
from pathlib import Path
rows = json.loads(Path(__file__).with_name("observed-rows.json").read_text())
if len({row["id"] for row in rows}) != len(rows):
raise ValueError("Duplicate row ID")
for row in rows:
probabilities = row["probabilities"]
if not probabilities or any(not 0 <= p <= 1 for p in probabilities.values()):
raise ValueError("Invalid probabilities")
row["uncertainty"] = round(1 - max(probabilities.values()), 6)
ranked = sorted(rows, key=lambda row: (-row["uncertainty"], row["id"]))
review = ranked[:2]
remaining = [row for row in ranked if row not in review]
audit = random.Random(7).sample(remaining, min(1, len(remaining)))
output = Path.cwd() / "review-queue.csv"
# Exclusive creation avoids replacing someone's completed annotations.
with output.open("x", newline="", encoding="utf-8") as file:
writer = csv.DictWriter(file, fieldnames=["id", "text", "predicted", "uncertainty", "selection", "reviewed_label"])
writer.writeheader()
for kind, selected in [("uncertainty", review), ("audit", audit)]:
for row in selected:
writer.writerow({**{k: row[k] for k in ["id", "text", "predicted", "uncertainty"]}, "selection": kind, "reviewed_label": ""})
print(json.dumps({"review_ids": [r["id"] for r in review], "audit_ids": [r["id"] for r in audit], "output": output.name}))
The local script selects two uncertain rows plus one seeded audit row from saved API results. It leaves reviewed_label blank and never accepts a model proposal or changes weights. No API request or key is needed.
Scope of this demo
What this example helps you test.
Supplied choices
What you can test
Editable categories and fallback rules
Ambiguous and quoted instructions
Comparing repeated decision requests
Application responsibilities
What this demo does not do
Automatic labeling or retraining
A working GEPA optimization loop
A benchmark of the original CLI
Example FAQ
Questions before you use the result.
Understand the choices, the source and what a live result means.
Is this the jev-align application?
No. This is an original, small classification exercise inspired by its documented workflow. Use the linked Sutro repository for the complete CLI and its separate reflection-model setup.
Does this demo learn from my corrections?
No. Changing the prompt changes the next request. The page does not save labels, train a model or accept a revised function. Human feedback storage and optimization belong in your application.
Should I review only uncertain rows?
Do not infer that a confident answer is correct. An audit sample and separately reviewed labels can expose systematic errors. This page does not choose a universal confidence threshold.
Can I change the input and see the request cost?
Yes. Edit the text, question and choices before running. A successful request displays elapsed request time and a provider-usage cost estimate when available. Verification must finish first if a security check appears. The estimate is not a charge to you.
Test your own decision
Try another input here or build a different Choice question in the Playground.