Fast workflows with code and models in zeroshot

The retrieval task

Retrieval asks which passages in a document collection help answer a question. BM25 word matching and BGE embeddings find candidates, but a passage can match the topic without supplying useful evidence. We use Jev, a decision model that returns probabilities for the fixed options relevant and irrelevant, to score each question and passage pair. These independent judgments can run concurrently, and the model's low cost makes it practical to assess several candidates for every request.

Building the graph with zeroshot

One zeroshot graph runs the complete request. A script node retrieves eight candidates from a warm local service. One map evaluates all eight concurrently through a native decision node that calls Jev directly. A second script node sorts the relevance probabilities and returns the ranked passages.

The retrieval node's runtime binding specifies an executable and its arguments:

{
  "retrieve": {
    "kind": "script",
    "command": [
      "python3",
      "script_client.py",
      "--url",
      "http://127.0.0.1:37877/retrieve"
    ]
  }
}

A script node starts the command, sends the step's input as JSON on standard input and reads JSON from standard output. Here, script_client.py forwards that input to the service over HTTP. The service performs BM25, dense retrieval and document routing while keeping indexes and embedding weights loaded across requests.

The decision node declares the relevance instruction and answer options, and its runtime selects jev-1.14.0. Each call receives the question and one passage. Zeroshot passes the resulting probabilities to the sorting script through the graph's typed inputs and outputs.

The benchmark

We tested all 300 SciFact and 323 NFCorpus queries from the official BEIR test splits, plus a 128-query sample from LegalBench-RAG. Each domain's full corpus remained searchable. BM25 word matching and BGE dense embeddings supplied eight candidates per query, combining results from document routing and global search.

The main comparison holds those eight candidates constant and ranks them by hybrid retrieval scores or Jev's relevance probabilities. NDCG@3 rewards placing relevant evidence in the first three positions. The complete graph retrieved its candidates afresh, matching the frozen pools on all 751 queries.

Ranking quality

Jev reranking improved NDCG@3 in each domain. With equal weight for each of the three domains, average NDCG@3 rose from 0.3838 to 0.4416, an absolute improvement of 0.0579. Its paired 95% bootstrap interval was 0.0404 to 0.0765.

Standalone BM25 scored 0.3496 on the same average, and dense retrieval scored 0.3780. These are local baselines.

Speed and cost

Across 751 queries, the graph made 6,008 Jev relevance decisions. It used 4.64 million input tokens, for an estimated model charge of $0.195. That is about $0.26 per 1,000 queries with eight candidates each, excluding local compute costs.

We measured latency separately on 60 queries, with 20 per domain. The retrieval service kept indexes and embedding weights loaded, and one graph ran at a time on a shared development host.¹

The eight concurrent Jev requests completed in 90 ms median and 140 ms at p95, including network and provider time. The complete CLI invocation, including retrieval and sorting, took 607 ms median and 759 ms at p95.

Conclusion

Retrieval is a practical example of a workflow that combines deterministic code with model judgments. Here, that combination improved evidence ranking with a median execution time below one second. A similar graph could select documentation for a support assistant or retrieve evidence from an internal knowledge base.

The approach could also support workflows beyond search. A support workflow could fetch account information through a script, classify the issue with a decision model and ask an agent to draft a response. Document intake could combine code that validates structured fields with a model that interprets free-text entries. Zeroshot lets these operations share one graph, with explicit data flow and concurrent execution where the steps are independent.