Jev classification rubric

Try a written support-ticket classification rule

Give a support request to classify Billing, Access or Shipping under the stated policy. Compare clear issues with mixed requests and missing details before using the rule.

Inspect a classification rubricLive API
Try an example

Original synthetic input. Edit it to run a fresh API decision.

Source: Original workflow on X

Source and example scope

Workflow inspiration only. Inputs and this mini-demo are original to Jevai; they do not reproduce or benchmark the source application.

Using the supplied rubric, which one label should this request receive?

Decision rules and labels
BillingAccessShippingNeeds review
A real response from the Jev API

Edit any input to explore a different decision. No request is sent until you run it.

Your next decision

Not run yet

Choose an example or write your own input, then run it to see the ranked choices.

Results are not stored by this site. Your submitted text is processed by the API provider.

The decision

How does Jev classification rubric work?

Give the model a policy with distinct labels and an explicit fallback. This original demo uses support requests to make disagreements easy to inspect. The chosen label is the highest-probability option for the question; it is not a human-approved training label.

Sutro's jev-align inspired this example. Its documented workflow evaluates rows, asks for labels and uses a separate reflection model with GEPA to propose a changed definition. A person can accept or reject that proposal. This page exercises only the typed classification step; it does not run the CLI, optimize a prompt or update model weights.

Change one rule at a time and rerun the same difficult request. Keep your intended label separately and compare it with the output. A concentrated distribution can still be wrong if the policy is incomplete. Mixed requests need a stated tie-breaking rule or a review outcome; confident mistakes should remain in an audit sample.

Write the rubric in the original Jev classification rubric workflow.
Original workflow diagram; not a screenshot of a third-party application.
How it works

Try the workflow in four steps.

Keep the supplied data, the model decision and the application action separate.

  1. Write the rubric in the original Jev classification rubric workflow.

    Write the rubric

    Name each category and say what belongs in it. Include an outcome for insufficient context.

  2. Try difficult rows in the original Jev classification rubric workflow.

    Try difficult rows

    Test a clear case, mixed request and incomplete text before using a larger dataset.

  3. Inspect the distribution in the original Jev classification rubric workflow.

    Inspect the distribution

    Read the leading label and alternatives. Do not treat a probability as a verified accuracy rate.

  4. Record human corrections in the original Jev classification rubric workflow.

    Record human corrections

    Keep reviewed labels outside this demo. Re-test a changed rubric before accepting it.

Our API test

What the six original inputs returned.

What the six original inputs returned.
Original inputObserved labelLeading probability
Duplicate chargeBilling100%
Sign-in failureAccess100%
Missing parcelShipping100%
Two equal issuesNeeds review100%
Missing contextNeeds review88%
Quoted instructionBilling93%

Six live Jev requests on September 26, 2026. Synthetic examples, not an independent accuracy benchmark. Fresh requests may return different distributions.Verified

Understand the results

The two equally important issues returned Needs review

The two equally important issues returned Needs review.

Read the example notes

The vague account message also returned Needs review. The quoted command to choose Shipping did not change the billing label in this one test. These outcomes illustrate the supplied fallback policy; six synthetic requests cannot establish protection against every misleading input or measure general accuracy.

Try a case with two equal issues

To repeat the check, choose Two equal issues, inspect Decision question and Candidate labels, then run the request.

Read the example notes

Now edit the question to give login problems explicit priority and run again. Compare both distributions against the rule you intended to express. Record the reviewed outcome separately; the site does not save corrections.

A change in output is not proof that the new rubric is better

A change in output is not proof that the new rubric is better.

Read the example notes

Keep previously correct examples in the comparison and separate data used to revise a rule from data reserved for evaluation. In the full jev-align workflow, feedback and proposal acceptance happen outside this page.

The optional local script turns the saved distributions into a three-row review file

The optional local script turns the saved distributions into a three-row review file.

Read the example notes

It ranks by one minus the largest Choice probability and also takes one seeded random audit row. This is our explicit sample-selection rule, not the jev-align algorithm or a universal threshold. Fill reviewed_label yourself after inspection; the script leaves every annotation blank.

Recorded data, reproducible code

Prepare a small review queue locally.

View the code example
prepare-review.py
# Standard library only. Saved real outputs; no new API call.
import csv
import json
import random
from pathlib import Path

rows = json.loads(Path(__file__).with_name("observed-rows.json").read_text())
if len({row["id"] for row in rows}) != len(rows):
    raise ValueError("Duplicate row ID")
for row in rows:
    probabilities = row["probabilities"]
    if not probabilities or any(not 0 <= p <= 1 for p in probabilities.values()):
        raise ValueError("Invalid probabilities")
    row["uncertainty"] = round(1 - max(probabilities.values()), 6)
ranked = sorted(rows, key=lambda row: (-row["uncertainty"], row["id"]))
review = ranked[:2]
remaining = [row for row in ranked if row not in review]
audit = random.Random(7).sample(remaining, min(1, len(remaining)))
output = Path.cwd() / "review-queue.csv"
# Exclusive creation avoids replacing someone's completed annotations.
with output.open("x", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(file, fieldnames=["id", "text", "predicted", "uncertainty", "selection", "reviewed_label"])
    writer.writeheader()
    for kind, selected in [("uncertainty", review), ("audit", audit)]:
        for row in selected:
            writer.writerow({**{k: row[k] for k in ["id", "text", "predicted", "uncertainty"]}, "selection": kind, "reviewed_label": ""})
print(json.dumps({"review_ids": [r["id"] for r in review], "audit_ids": [r["id"] for r in audit], "output": output.name}))

The local script selects two uncertain rows plus one seeded audit row from saved API results. It leaves reviewed_label blank and never accepts a model proposal or changes weights. No API request or key is needed.

Scope of this demo

What this example helps you test.

Supplied choices

What you can test

  • Editable categories and fallback rules
  • Ambiguous and quoted instructions
  • Comparing repeated decision requests
Application responsibilities

What this demo does not do

  • Automatic labeling or retraining
  • A working GEPA optimization loop
  • A benchmark of the original CLI
Example FAQ

Questions before you use the result.

Understand the choices, the source and what a live result means.

Is this the jev-align application?

No. This is an original, small classification exercise inspired by its documented workflow. Use the linked Sutro repository for the complete CLI and its separate reflection-model setup.

Does this demo learn from my corrections?

No. Changing the prompt changes the next request. The page does not save labels, train a model or accept a revised function. Human feedback storage and optimization belong in your application.

Should I review only uncertain rows?

Do not infer that a confident answer is correct. An audit sample and separately reviewed labels can expose systematic errors. This page does not choose a universal confidence threshold.

Can I change the input and see the request cost?

Yes. Edit the text, question and choices before running. A successful request displays elapsed request time and a provider-usage cost estimate when available. Verification must finish first if a security check appears. The estimate is not a charge to you.

Test your own decision

Try another input here or build a different Choice question in the Playground.