209 comments sorted in 21 seconds: the exact Jev setup

The result

I gave Jev one question: does this YouTube comment ask the creator a question, or ask for help?

Then I gave it every top-level comment from three public videos about Jev itself. Sam Witteveen's "Jev - The Ultimate Classification Model?" had 143, Gary Explains had 29 and Jack Roberts had 37. That's 209 comments, pulled on 26 September 2026, each one sent in full.

It answered all 209 in 21.2 seconds, eight requests at a time. The bill was $0.003339. That number is the usage.cost field OpenRouter sends back with every answer, added up, not a price list.

• 63 scored 0.7 or higher, so they were flagged as questions.
• 130 scored under 0.3, so they were flagged as not questions.
• 16 landed in between. Those are the ones you read yourself.

I ran it a second time with the script further down: 63, 15 and 131, in 21.5 seconds, for the same $0.003339. One comment moved from the middle to "not a question". That drift is normal for Jev, and it's why you set cut-offs instead of trusting an exact score.

Jev never writes a word. Every answer is a probability, and your code (or you) decides what to do with it.

Did it get them right?

Before I looked at a single score, I labelled 20 random comments by hand.

I was sure about 18 of them. Jev agreed with me on 17. The 18th was a rant it scored 0.41, so it went to the unsure pile, not the wrong one.

The two I couldn't decide on myself ("I wonder what would happen if..." and "What's the point of this??") scored 0.63 and 0.77. Fair enough, honestly.

Twenty is a spot check, not a benchmark. Label twenty of your own before you trust it on a pile.

The cost math

209 comments, one request each
79,507 input tokens in total
$0.003339 billed (usage.cost, summed)
= about $0.042 per million input tokens
output tokens are free

OpenRouter bills Jev by input tokens only. The current rate is on the Jev model page; my figure is what I was actually charged.

See it before you write any code

OpenRouter's Jev Lab runs Jev demos in the browser, ticket triage among them. It shows the input, the question and the raw probabilities that come back.

The setup I used

1. An OpenRouter account and an API key from openrouter.ai/settings/keys. No TypeSafe account, no waitlist.
2. Put the key in your terminal:

export OPENROUTER_API_KEY='your-key-here'

3. Send one comment and one question. This is the exact request, tested on 26 September:

curl -sS --fail-with-body https://openrouter.ai/api/v1/systemone \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "typesafe/jev-1.13",
    "state": "How did you make that animation at the start of your videos?",
    "questions": {
      "answer": {
        "type": "noul",
        "instructions": "Does this YouTube comment ask the creator a question or ask for help?",
        "criteria": {
          "true": "it asks a question or asks for help: how, can it, does it, what about, is there",
          "false": "praise, opinion, a joke, a link, or a statement with no question in it"
        }
      }
    }
  }'

What came back:

{
  "answers": { "answer": { "type": "noul", "noul": 0.99 } },
  "usage": { "input_tokens": 352, "output_tokens": 20, "cost": 0.000014784 }
}

noul is Jev's name for a yes or no answer, and the number is the probability of yes. 0.99 means yes. With --fail-with-body, a missing or wrong key stops with a 401 instead of looking like it worked.

The whole pile: the script I ran

Put one comment per line in comments.txt (squash any line breaks inside a comment into spaces), then run this with Python 3. It needs nothing installed beyond Python itself.

# One Jev yes/no question for every line of comments.txt, eight at a time.
import json, os, time, urllib.error, urllib.request
from concurrent.futures import ThreadPoolExecutor

QUESTION = {
    "type": "noul",
    "instructions": "Does this YouTube comment ask the video's creator a question or ask for help?",
    "criteria": {
        "true": "it asks a question or asks for help: how, can it, does it, what about, is there",
        "false": "praise, opinion, a joke, a link, or a statement with no question in it",
    },
}

def ask(comment):
    body = {"model": "typesafe/jev-1.13", "state": comment, "questions": {"answer": QUESTION}}
    req = urllib.request.Request(
        "https://openrouter.ai/api/v1/systemone", data=json.dumps(body).encode(),
        headers={"Authorization": "Bearer " + os.environ["OPENROUTER_API_KEY"],
                 "Content-Type": "application/json"})
    for attempt in range(4):
        try:
            with urllib.request.urlopen(req, timeout=60) as r:
                out = json.load(r)
            return out["answers"]["answer"]["noul"], out["usage"]["cost"]
        except urllib.error.HTTPError as e:
            if e.code not in (429, 500, 502, 503) or attempt == 3:
                raise
            time.sleep(2 ** attempt)

comments = [line.strip() for line in open("comments.txt", encoding="utf-8") if line.strip()]
start = time.time()
with ThreadPoolExecutor(max_workers=8) as pool:
    results = list(pool.map(ask, comments))
seconds = time.time() - start

buckets = {"question": [], "read it yourself": [], "not a question": []}
for comment, (p, _) in zip(comments, results):
    key = "question" if p >= 0.7 else "not a question" if p < 0.3 else "read it yourself"
    buckets[key].append((p, comment))
for key, items in buckets.items():
    print(f"\n== {key}: {len(items)}")
    for p, c in sorted(items, reverse=True):
        print(f"{p:.2f}  {c[:100]}")
print(f"\n{len(comments)} comments in {seconds:.1f} s, ${sum(c for _, c in results):.6f}")

It prints three piles, highest score first, then the time and the total cost. Change QUESTION to ask something else of your own pile; Jev also does choice (pick one of several options) and score (place something on a scale you define). The Jev docs on OpenRouter cover both.

The catch

• It writes nothing. No reply, no reason, just a number. If you need words, pair it with a chat model, or with yourself.
• It needs an API key and a terminal. Jev Lab is the no-install way to look at it.
• Choose your cut-offs on labelled examples. 0.5 means unsure, not "medium". I used 0.7 and 0.3 and read the middle myself.
• Try a plain search first. On one of my own reels, 215 of 232 comments were just the keyword or just emoji. A text match cleared those before Jev saw anything.
• The same comment can score slightly differently on a rerun. Compare against your cut-offs, not against the last run's exact number.

What I would test next

Send me a redacted example of the pile you're stuck on (comments, emails, support messages, with names, addresses and account details taken out) and the one question you'd ask of each. I'll tell you whether Jev is worth wiring up for it, or whether a search box already does the job.

If you want more measured AI tests, The Rabbit Hole is free to join. The course library is free; optional Premium adds weekly office hours.