Evaluation & quality

After an agent finishes a task, an independent reviewer scores it 0–100. The principle is simple: no agent grades its own work, the reviewer reads the artifacts the task produced rather than taking the assignee’s word for it, and a confused verdict fails closed rather than rubber-stamping.

The reviewer reads the work, not the report

The judge doesn’t just get the assignee’s summary. Alongside it, the runtime assembles a work manifest — the files the task actually wrote (in the agent’s working directory and the org’s shared docs, captured over the run’s own time window) with a bounded slice of each file’s real content — and puts it in front of the reviewer. The instruction is explicit: grade the work itself against those files, and if the report describes work the artifacts don’t back up, or a needed deliverable simply isn’t there, score it low and say what’s missing. So “independent review” means reviewing the artifacts, not a self-authored claim about them. (The manifest is a bounded aid — newest files first, capped in size — not a full snapshot; the deliverable gate below is the objective backstop.)

Who reviews

Review is manager-routed. The evaluator is chosen in order:

1. A delegated task is reviewed by the manager who handed it down.
2. Otherwise a Reviewer/QA-role agent, if one exists.
3. Otherwise the assignee’s manager.
4. Otherwise any other agent — never the assignee itself.

With no other agent available (for example, a solo CEO), evaluation returns null and the runtime accepts, relying on you as the external check. It never asks an agent to grade its own output.

Unparseable verdicts fail closed

A missing SCORE is retried once with a strict reprompt. If it’s still unparseable, the task is sent back for rework — not auto-passed (task.eval.unparsed is audited). This is the opposite of fail-open behavior, which would rubber-stamp exactly the confused, low-quality outputs most likely to omit the token. The score parser tolerates wrapping (SCORE: **72**) and decimals.

Echo/no-judge setups (like the fake brain) can opt back into fail-open with failOpenOnUnparsedEval: true.

Rework carries feedback

Below evalThreshold (default 60) the task returns to todo with the reviewer’s feedback. On the next attempt the judge is shown the prior feedback and asked to verify each point was addressed — so rework is a real loop, not a re-roll.

Rework holds the task’s dependents until it lands, so downstream work never builds on output that’s still being fixed.

Deliverables are verified, not just reviewed

The grounded review above reads the artifacts, but it’s still a model judging quality. When a task’s whole point is a deliverable, the plan can also anchor it to one — an outcome kind on the task — and the completion gate then verifies the artifact itself with an objective check (a parse, a stat), no model involved, before “done” counts:

Outcome	What must exist
code	real files written or changed on disk in the agent’s working directory
doc	a document created or updated in the org’s shared docs
decision	a decision recorded in the shared DECISIONS.md
service	a service registered on the service registry whose URL actually responds (2xx/3xx) — a dead or invented link doesn’t count
send	an outbound message sent on an org channel
contact	a contact recorded on the org CRM

The assignee is told up front that its completion is checked against the artifact. A passing review without the artifact takes the same rework path a low score does — with what’s missing (and the marker or file that produces it) as the feedback. File-backed kinds are checked against the current attempt: work found stale on disk doesn’t count, and the runtime’s own bookkeeping files never satisfy a code outcome.

Two boundaries keep it honest:

• The check is objective — a marker parse and a file check (and, for a service, a real HTTP request to the registered URL), never a model call — so it holds even with --no-eval, and in a solo org with no independent reviewer. Prose alone can’t complete artifact-anchored work, and a URL that 404s or fails to connect can’t stand in for a live deploy. A SERVE line reporting an outage (status=down) is exempt — there the registration itself is the deliverable.
• The gate never demands the impossible. An outcome the assignee is denied the means to produce (a capability or channel it doesn’t have) is waived and audited (task.outcome.waived), not failed.

Force-accept is honest

After maxAttempts (default 2 — one initial run plus one rework) a still failing task is not left to deadlock. It’s marked done, but:

• flagged qualityFlag: 'below-threshold' with its score — or 'missing-artifact' when it was the deliverable, not the score, that failed,
• a distinct task.forceAccept audit event is emitted,
• the responsible manager is flagged in-house — the delegator, else the assignee’s manager, gets a note (in the assignee’s voice) to re-scope, reassign, or confirm on their next turn. Whether below-bar work is redone or accepted is a management call, so the operator is not asked to make it; the flag and the Health panel keep it visible for inspection. At the very top of the org (no manager) the work simply stands, flagged, and
• delegations do not fan out off rejected or unverified work.

Downstream tasks that depend on it get their upstream context prefixed with [ACCEPTED BELOW QUALITY BAR, score N] (or [ACCEPTED WITHOUT ITS DECLARED DELIVERABLE]), so a dependent can hedge rather than trust it blindly. Accepted-clean and accepted-below-the-bar are always distinguishable — to the runtime and to you.

The plan is reviewed too

The same “no one grades their own work” principle guards planning, one step earlier. When the lead decomposes a goal, an independent reviewer judges whether the plan actually covers the goal before any task is created; a plan judged thin or rushed is sent back for one fuller re-plan. This plan-coverage gate runs on the same evaluate switch as task scoring and picks a non-lead reviewer the same way — see the autonomous cycle. It’s what stops the org from racing a big mission into a few shallow tasks.

Turning it off or tuning it

Knob	Default	Effect
evaluate (EVAL=off, --no-eval)	on	run the quality gate at all (task and plan review)
evalThreshold (EVAL_THRESHOLD, --eval-threshold)	60	minimum passing score
maxAttempts	2	total executions (initial + reworks)
maxTasks	12	most subtasks the lead may split one goal into
planReviewConfidence	60	reviewer certainty that a plan is thin before it’s sent back for one re-plan
failOpenOnUnparsedEval	false	auto-pass an unparseable verdict (echo/no-judge only)

EVAL_THRESHOLD=70 quorum      # hold a higher bar
quorum --no-eval              # ship whatever agents produce, unreviewed

Reliability around execution

Evaluation sits on top of a side-effect-safe execution model. A task turn runs real tools, so it gets a single attempt — a blind retry would re-run Bash/writes/sends. Only genuinely pre-effect blips (spawn/ENOENT) retry once; a clean non-zero exit or unparseable output fails fast. The review-gated rework loop is the correct layer to re-execute real work, which is exactly what it does.