<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://gjusev.github.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://gjusev.github.io/" rel="alternate" type="text/html" hreflang="en" /><updated>2026-11-16T20:03:53+00:00</updated><id>https://gjusev.github.io/feed.xml</id><title type="html">Youssef, AI product engineer</title><subtitle>I build open tools that test whether AI does what it claims, and I publish every result. AI product engineer, founder of mokka.</subtitle><author><name>Youssef</name></author><entry><title type="html">The cheap way to check an agent’s work</title><link href="https://gjusev.github.io/writing/2026/who-checks-the-agent/" rel="alternate" type="text/html" title="The cheap way to check an agent’s work" /><published>2026-11-16T00:00:00+00:00</published><updated>2026-11-16T00:00:00+00:00</updated><id>https://gjusev.github.io/writing/2026/who-checks-the-agent</id><content type="html" xml:base="https://gjusev.github.io/writing/2026/who-checks-the-agent/"><![CDATA[<p>A coding agent hands you a finished change. Do you trust it as it is, check it yourself, run the tests, or pay a second agent to review it? I ran all four options on the same tasks and compared two things: how much the pass rate moved, and what the check cost in tokens.</p>

<h2 id="the-setup">The setup</h2>

<p>One Python coding agent solves HumanEval tasks. Between its first solution and submission, four things can happen, and that is the whole experiment:</p>

<table>
  <thead>
    <tr>
      <th>Arm</th>
      <th>What happens before submission</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>none</td>
      <td>The solution goes in as it is</td>
    </tr>
    <tr>
      <td>selfcheck</td>
      <td>The same agent instance rereads its own solution, once, and may revise it</td>
    </tr>
    <tr>
      <td>reviewer</td>
      <td>A second, fresh-context agent reviews it and may ask for one revision</td>
    </tr>
    <tr>
      <td>gate</td>
      <td>The solution runs against the task’s tests; on failure the agent sees the output and retries, twice at most</td>
    </tr>
  </tbody>
</table>

<p>Solver and reviewer are the same model (glm-4.5-flash), temperature 0. Tokens are the ones the API reported, real calls only.</p>

<h2 id="the-result">The result</h2>

<p>On the first 20 tasks the model scored 20 out of 20 in every arm, so the interesting numbers come from the harder back half, tasks 144 to 163:</p>

<table>
  <thead>
    <tr>
      <th>Arm</th>
      <th>Pass rate</th>
      <th>Tokens per task</th>
      <th>Cost</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>none</td>
      <td>85%</td>
      <td>325</td>
      <td>1.00×</td>
    </tr>
    <tr>
      <td>selfcheck</td>
      <td>90%</td>
      <td>887</td>
      <td>2.73×</td>
    </tr>
    <tr>
      <td>reviewer</td>
      <td>90%</td>
      <td>1,155</td>
      <td>3.56×</td>
    </tr>
    <tr>
      <td>gate</td>
      <td>90%</td>
      <td>487</td>
      <td>1.50×</td>
    </tr>
  </tbody>
</table>

<p>Every check bought the same five points. The gate bought them by running the code and letting reality complain. The reviewer bought them by paying for a second full context on every turn.</p>

<h2 id="what-this-number-can-and-cannot-say">What this number can and cannot say</h2>

<p>Twenty tasks is small. One flipped task moves the pass rate by exactly five points, so these deltas show a direction, not a size. The paper this replicates, Rethinking Verification in Agent Harnesses (Alibaba and Cornell), found the same direction across seven frontier models; my setup can only check that the direction holds at small scale, and it did.</p>

<p>Two honest footnotes. The reviewer’s cost (3.56×) is much worse here than in the paper (1.13× to 1.25×), because my base arm is one short call, so every review turn pays the full context again. And the gate sees the bundled tests, which rewards fitting those exact asserts; against a hidden test suite its edge would shrink.</p>

<h2 id="what-i-take-from-it">What I take from it</h2>

<p>If the task has executable checks, a gate is the cheap default. It never pays for a second opinion that mostly restates the code. A reviewer earns its cost only where no test can run, and the price of that opinion should be measured, not assumed. Self-check landed in between, and it did better here than the paper expected, which is exactly the kind of result a small replication exists to surface.</p>

<p>Raw rows, per-task CSVs and the console logs are in the <a href="https://github.com/Gjusev/trust-but-verify">trust-but-verify</a> repo.</p>]]></content><author><name>Youssef</name></author><summary type="html"><![CDATA[Three ways to check a coding agent's work, measured side by side. All three bought the same five points; the gate cost 1.5× the tokens, the reviewer 3.56×.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://gjusev.github.io/assets/projects/trust-but-verify/og.jpg" /><media:content medium="image" url="https://gjusev.github.io/assets/projects/trust-but-verify/og.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">74 comparisons instead of 3,000</title><link href="https://gjusev.github.io/writing/2026/what-navigation-retrieval-costs/" rel="alternate" type="text/html" title="74 comparisons instead of 3,000" /><published>2026-11-12T00:00:00+00:00</published><updated>2026-11-12T00:00:00+00:00</updated><id>https://gjusev.github.io/writing/2026/what-navigation-retrieval-costs</id><content type="html" xml:base="https://gjusev.github.io/writing/2026/what-navigation-retrieval-costs/"><![CDATA[<p>Flat retrieval scores every vector in the corpus. On my test corpus that is 3,000 comparisons per query, and it is the reason retrieval gets expensive at scale. treeseek walks instead: the query descends a two-level k-means tree and only scores the passages in the branch it picked. 74 comparisons, one per branch decision and one per candidate passage in the final subtree.</p>

<h2 id="what-it-bought">What it bought</h2>

<table>
  <thead>
    <tr>
      <th>Retriever</th>
      <th>Recall@1</th>
      <th>Hit@5</th>
      <th>Vectors compared per query</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>flat</td>
      <td>0.613</td>
      <td>0.812</td>
      <td>3,000</td>
    </tr>
    <tr>
      <td>treeseek</td>
      <td>0.650</td>
      <td>0.750</td>
      <td>74</td>
    </tr>
  </tbody>
</table>

<p>Top-rank accuracy went up slightly; a walk is good at not being distracted by near-misses living in other topics. The comparison count fell 40×.</p>

<h2 id="what-it-cost">What it cost</h2>

<p>Hit@5 dropped six points, and all eight queries that flat solved and treeseek missed trace to one mechanism: a branch taken is a branch never revisited. My favorite failure is the query “Age of Reason Was: Who has read Rushdie’s” (topic alt.atheism). The walk descended into a mixed subtree of medicine, baseball and politics. No atheism passage existed in that subtree, so none could appear at any rank. Close in meaning, wrong topic, and greedy descent has no undo.</p>

<h2 id="when-the-tree-is-worth-it">When the tree is worth it</h2>

<p>When the corpus has real topic structure and top-1 matters more than top-5 recall. When queries arrive as short subject lines, the way they do in triage and routing. When it is not worth it: when recall matters more than cost, or when the corpus is a tangle without clusters; then the tree prunes knowledge it cannot afford to lose.</p>

<p>The run, per-query CSVs and the build cost are in the <a href="https://github.com/Gjusev/treeseek">treeseek</a> repo. It is a measured comparison against flat retrieval, not a reproduction of DecisionRAG; the navigation idea is theirs.</p>]]></content><author><name>Youssef</name></author><summary type="html"><![CDATA[A retrieval walk that scores 74 vectors instead of 3,000 keeps the same top-1 accuracy. What the pruning costs, measured, and when the tree is worth it.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://gjusev.github.io/assets/projects/treeseek/og.jpg" /><media:content medium="image" url="https://gjusev.github.io/assets/projects/treeseek/og.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Your agent already made this decision</title><link href="https://gjusev.github.io/writing/2026/your-agent-already-made-this-decision/" rel="alternate" type="text/html" title="Your agent already made this decision" /><published>2026-11-09T00:00:00+00:00</published><updated>2026-11-09T00:00:00+00:00</updated><id>https://gjusev.github.io/writing/2026/your-agent-already-made-this-decision</id><content type="html" xml:base="https://gjusev.github.io/writing/2026/your-agent-already-made-this-decision/"><![CDATA[<p>Watching an agent rerun the same workload and call the model four times for decisions it had already made was expensive to watch. zerocall is my attempt to skip those calls safely. It stores each decision under a signature of what the agent was asked, and pins it with guards that are re-checked against the live environment before any replay.</p>

<h2 id="the-measurement">The measurement</h2>

<p>The same workload, run cold and warm, with real API tokens:</p>

<table>
  <thead>
    <tr>
      <th>Run</th>
      <th>Model calls</th>
      <th>Tokens</th>
      <th>Wall time</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Cold: the model decides every task</td>
      <td>4</td>
      <td>1,092</td>
      <td>35.86 s</td>
    </tr>
    <tr>
      <td>Warm: nothing changed, cache active</td>
      <td>0</td>
      <td>0</td>
      <td>0.08 s</td>
    </tr>
  </tbody>
</table>

<h2 id="the-part-that-matters-the-guard-drill">The part that matters: the guard drill</h2>

<p>A cache that cannot tell when it is wrong is worse than no cache. So after the warm run I edited <code class="language-plaintext highlighter-rouge">demo_pkg/utils.py</code> by hand and reran the task that touches it. The signature still matched. The file hash guard did not. That single failed guard forced exactly one fresh decision (293 tokens), which was recorded again, and the next run was back to zero calls.</p>

<p>That is the whole contract: replay is only ever offered while every guard holds. One changed byte and the agent pays for one decision, not for the whole workload.</p>

<h2 id="where-i-would-not-use-it">Where I would not use it</h2>

<p>On tasks whose inputs never repeat, the cache is dead weight. On decisions whose guards are hard to pin, anything that depends on a live API’s mood rather than on file bytes, a replay would be a guess, and a wrong replay is worse than an expensive call. But for agents working over files, where “did anything change” has a real answer, the warm path is free.</p>

<p>The signature function, the guard types and the demo are in the <a href="https://github.com/Gjusev/zerocall">zerocall</a> repo. Sixteen tests, standard library only.</p>]]></content><author><name>Youssef</name></author><summary type="html"><![CDATA[Context signatures with guards turned a warm rerun from 4 model calls into 0, and 35.9 seconds into 0.08. When a file changed, exactly one call came back.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://gjusev.github.io/assets/projects/zerocall/og.jpg" /><media:content medium="image" url="https://gjusev.github.io/assets/projects/zerocall/og.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">A small model that refuses</title><link href="https://gjusev.github.io/writing/2026/a-small-model-that-refuses/" rel="alternate" type="text/html" title="A small model that refuses" /><published>2026-11-05T00:00:00+00:00</published><updated>2026-11-05T00:00:00+00:00</updated><id>https://gjusev.github.io/writing/2026/a-small-model-that-refuses</id><content type="html" xml:base="https://gjusev.github.io/writing/2026/a-small-model-that-refuses/"><![CDATA[<p>Big models guess with confidence. For a decision that leads to an action, the guess is the expensive part: the wrong write that went through because nothing said stop. MK-SH-V is my attempt at the other end of that trade. It is a 260M visual decision model that returns calibrated probabilities, a choice, and a refusal when its confidence sits below a gate whose risk was validated before release.</p>

<h2 id="what-it-does">What it does</h2>

<p>Hand it a document or a screenshot, a question and the valid options. One encoder pass reads pixels and text together, each option is scored by a shared head, and the softmax runs over the valid options only. A photographed receipt asked “is the total at least 45.662 IDR?” comes back true at 0.978. Asked which of three ranges the total (302.016) falls into, it picked the wrong bin at 0.757, cleared the gate, and answered wrong. On real receipts about a quarter of the answers the gate lets through are wrong, and I published that too.</p>

<h2 id="the-gate">The gate</h2>

<p>The gate is a threshold on confidence, fitted on a calibration split by a rule frozen in advance: the smallest threshold whose 95% Clopper-Pearson upper bound on selective risk stays within a 5% budget. Frozen in advance means I could not tune it after seeing the test. The value it ships with (0.55) was fixed by protocol, not by taste. On the held-out synthetic test it answers 98.3% of the cases with a 3.2% risk bound.</p>

<h2 id="small-against-big">Small against big</h2>

<p>On held-out real screenshots, MK-SH-V scores 0.839. A roughly 15× larger open model, run zero-shot on the same test, scores 0.731. A small trained model with a real calibration pass beats a much bigger generalist on a narrow job, and unlike the generalist it can say “below my gate, this one is yours” and hand the decision back.</p>

<p>The weights, the inference policy and a live demo (with a small booth game where every refusal comes from the model’s real probabilities) are linked from the <a href="https://github.com/Gjusev/MK-SH-V">MK-SH-V</a> repo.</p>]]></content><author><name>Youssef</name></author><summary type="html"><![CDATA[A 260M visual model with calibrated odds and a validated refusal gate. On held-out real screenshots it scores 0.839; a model 15× larger scores 0.731 zero-shot.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://gjusev.github.io/assets/projects/MK-SH-V/og.jpg" /><media:content medium="image" url="https://gjusev.github.io/assets/projects/MK-SH-V/og.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>