Jev evaluation question dedup

Check whether two evaluation questions test the same thing

Give both questions and their pass conditions. Compare equivalent tests with questions a candidate could answer differently before removing an item.

Do these questions define the same test?Live API
Try an example

The wording changes but the required evidence does not.

Source: API-collected workflow inspiration; original demonstration below

Source and example scope

The Albedo post names this task. A separately located public Albedo repository contains corresponding decision code, pinned here to a commit. The post’s short link points to its photo, not that repository; shared authorship is unverified. This is an original synthetic adaptation, not an Albedo integration or reproduction of its speed and cost claims.

Compare evaluation questions and their pass conditions.

Decision rules and labels
Same testDifferent testsNeed questions
A real response from the Jev API

Edit any input to explore a different decision. No request is sent until you run it.

Your next decision

Not run yet

Choose an example or write your own input, then run it to see the ranked choices.

Results are not stored by this site. Your submitted text is processed by the API provider.

The decision

Similar words can hide different pass conditions.

“Was the test run?” and “Did the test pass?” require different observations. An execution that fails satisfies only the first condition.

The paraphrase preset changes the wording while keeping the required runner invocation the same. Optional detail, such as a line number, also need not create a distinct test when both conditions accept the same underlying finding.

A vague standard such as good or acceptable cannot be compared without its definition. This demonstration supplies a relationship label; it does not score candidate work or remove questions from an evaluation set.

The Albedo post names this task. A separately located public Albedo repository contains corresponding decision code, pinned here to a commit. The post’s short link points to its photo, not that repository; shared authorship is unverified. This is an original synthetic adaptation, not an Albedo integration or reproduction of its speed and cost claims.

Illustration of matching a question to supplied passages and identifying missing evidence
Illustration of the decision flow.
How it works

Try the example in four steps

Use the sample to understand the rule, then test your own input for evaluation question overlap.

  1. Illustration of adding the text or evidence for a decision

    Add the information to check

    Give both questions and their pass conditions.

  2. Illustration of defining a question and its allowed choices

    Confirm the question and labels

    Open the decision rules and keep the intended distinction between Same test, Different tests, Need questions. Edit the choices only when testing your own rule.

  3. Illustration of sending one input and comparing the returned choices

    Run the decision

    Use a sample first, then try your own input. Compare the selected label with the other returned probabilities.

  4. Illustration of reviewing a result and an uncertain item before using it

    Review the next step

    Review Need questions before using a result. The label is a decision about the supplied information, not an executed action.

Our API test

Recorded API observations

Recorded API observations
Original inputObserved labelLeading probability
Equivalent wordingSame test98.0%
Read versus repairDifferent tests98.0%
Run versus passDifferent tests93.0%
Optional detailSame test85.0%
Conflicting conditionDifferent tests61.0%
Missing standardsDifferent tests49.0%

These are six original synthetic inputs, tested once per request revision. The percentage is the leading option probability, not measured accuracy. Authored expectations and any disagreements remain in the downloadable evidence. 4 of 6 results matched the authored fixture expectations. For Conflicting condition, the expectation was Need questions; the observed choice was Different tests. For Missing standards, the expectation was Need questions; the observed choice was Different tests.Verified

Understand the results

Replay the saved synthetic requests to inspect each recorded decision

Replay the saved synthetic requests to inspect each recorded decision. Authored expectations and model disagreements remain available; these examples are not a production accuracy benchmark.

Using the result in your application

The single-request SDK runner uses your private TYPESAFE_API_KEY. The offline consumer reads saved observations, checks request hashes and prints a pending output. It does not call the model or change any external system.

Runnable offline example

Run one sample, then inspect the local packet

View the code example
Single-request replay
npm install @typesafe-ai/sdk@0.6.0
# Set TYPESAFE_API_KEY privately in your shell.
node run-choice-sample.mjs samples.json case-1
python preview_choice_workflow.py samples.json saved-api-results.json
Scope of this demo

Where this decision fits

Useful scope

  • Finding redundant evaluation questions
  • Distinguishing execution from successful execution
  • Checking whether optional detail changes a test

Outside this demonstration

  • Scoring a candidate trajectory
  • Deleting evaluation questions automatically
  • Inferring undefined quality standards
Example FAQ

Questions about evaluation question dedup.

Input scope and downstream handling determine what this judgment means.

Why are run and pass different tests?

The runner can execute a test and report a failure. That candidate satisfies the execution condition but not the success condition.

Does different wording always mean a different test?

No. If both conditions require the same evidence, the questions can be equivalent despite their wording.

Is the leading probability an accuracy measurement?

No. It is one value in the returned distribution over this request’s choices. The six synthetic fixtures have authored expectations; they do not measure production accuracy.

What happens on an API failure?

The window displays an error without a fresh judgment. An incomplete-input outcome describes the supplied evidence and is not used as a fallback for provider failure.

Try your own evaluation question dedup input

Edit the input above, or open the Playground to explore another decision.