Aggregation - choose the right approval_method

You’ve decided to call consult(). Now: how many humans should answer, how do you combine their answers, and what do you do if they disagree or don’t show up?

This is what approval_method, target_responses, wait_seconds, bypass_on_timeout, and require_human together control. Pick them based on the criticality of the decision point.

Note

This page is about aggregating free-text answers via approval_method. If you asked a typed question (a card — multiple-choice, ranking, rating, …), the response instead carries a typed aggregate computed by the method that fits the answer type (plurality, Borda, median, …); you don’t pick an approval_method for those.

The four approval_method values, by use case

"fastest" - first valid response wins

consult(
    question=...,
    target_responses=1,
    approval_method="fastest",
    wait_seconds=60,
)

Latency-optimised. Asks a single human, returns the first valid response, no aggregation. The Hub still routes to up to 3 candidates internally to absorb no-shows, but you only see the winner.

Use when:

• The question is informational, not judgment-heavy
• A reasonable answer from any qualified human is fine
• You want sub-minute latency

Don’t use for:

• Irreversible actions (pair with consensus or all_must_agree)
• Niche domains where one expert is much better than another (use best_of_n)

Shorthand: @hc/1/fastest/60s/bypass

"consensus" - N voters, agreement ratio surfaced

consult(
    question=...,
    target_responses=3,
    approval_method="consensus",
    wait_seconds=60,
)

Routes to 3 humans (or whatever you set). Waits for all of them, then returns a response only if the agreement ratio across their answers is ≥ 0.6 (the Hub does the semantic-similarity check). Below that, provenance flips to "escalated" and you can decide what to do.

Use when:

• Medium-stakes guidance where you want sanity-checking
• The question has a defensible “right answer” 2+ experts would converge on
• You can absorb 2 minutes of latency in exchange for confidence

Trade-off: 3 credits per call, but you get a confidence floor under the answer.

Shorthand: @hc/3/consensus/120s/bypass

"best_of_n" - N voters, highest-ranked wins

consult(
    question=...,
    target_responses=3,
    approval_method="best_of_n",
    wait_seconds=60,
)

Routes to 3 humans, waits for all, returns whichever one has the highest expertise grading score on the matched tags. Like fastest but with a quality bias.

Use when:

• Niche domain where one expert is meaningfully sharper than the others (a top oncologist vs. a generalist; a specialist tax lawyer vs. a generalist counsel)
• You want a quality bar but agreement isn’t the right metric (two experts can give different correct answers in nuanced fields)

Trade-off: Same cost as consensus (3 credits) but uses expertise grading rather than agreement to pick the surfaced answer.

Shorthand: @hc/3/best_of_n/120s/bypass

"all_must_agree" - N voters, unanimity required

consult(
    question=...,
    target_responses=3,
    approval_method="all_must_agree",
    wait_seconds=600,
    require_human=True,
)

The strictest option. Routes to N humans, waits for all of them, and surfaces a response only if they unanimously agree. Anything less and provenance becomes "escalated".

Use when:

• Irreversible actions: deploys, sends, transactions, contract signatures, public statements
• High-stakes binary decisions where being wrong is much worse than being slow
• Regulated workflows where you need an audit trail of multiple human approvers

Trade-off: Highest latency, highest cost, highest quality floor. Often paired with require_human=True so the LLM fallback doesn’t override the strict gate.

Shorthand: @hc/3/all_must_agree/600s/strict

A decision tree

Is this an irreversible action? (wire, send, deploy, sign, delete)
├── Yes → all_must_agree, N=3, wait=600s, require_human=True
└── No  → ...
    │
    Is this a high-stakes judgment? (medical/legal/financial advice surfaced to user)
    ├── Yes → consensus, N=3, wait=120s
    └── No  → ...
        │
        Is this a niche domain where one expert >> others?
        ├── Yes → best_of_n, N=3, wait=120s
        └── No  → fastest, N=1, wait=60s   ← the default

How to handle each provenance value

The other half of aggregation is what your agent does with the response, given its provenance:

answer = consult(...)

if answer.provenance == "human":
    # Ground truth. Use it directly.
    apply_answer(answer.response_text)

elif answer.provenance == "llm_fallback":
    # The LLM filled in because no human responded in time.
    # Use it, but flag for async review.
    apply_answer(answer.response_text)
    flag_for_async_review(answer.consult_id)

elif answer.provenance == "mixed":
    # Consensus across human + LLM responses (only happens with
    # target_responses>1 and approval_method="consensus"). Treat as
    # somewhat-trusted; weaker than pure human.
    apply_answer(answer.response_text)
    log_mixed_provenance(answer.consult_id)

elif answer.provenance == "escalated":
    # all_must_agree failed unanimity, OR consensus dipped below 0.6.
    # Do NOT proceed. Pause the agent, surface to the user / on-call.
    pause_and_escalate(answer)

The provenance field is the single most important branch in your integration. Build that switch once and the rest of the wiring stays simple.

On wait_seconds and human reality

Picking a wait_seconds value is a calibration exercise. Some rough benchmarks from production:

wait_seconds	Use case	What it actually means
30	“instant gratification” interactive	Only people already in the queue answer; falls back to LLM often
60	Standard interactive	Catches online experts; some fallback
120	Tolerable for chat / agent flows	Wakes mobile-notified experts; low fallback rate in active chains
300	Acceptable for background tasks	Allows multi-attempt routing; near-zero fallback rate
600	High-stakes blocking gate	Truly waits for humans; pair with require_human=True
1800	Overnight batch	Maxes out; expert can sleep, wake, answer

There’s no single right answer - it depends on how active your chain is and how much latency your agent’s workflow can absorb. Start with the suggestion in the tree above and adjust based on the fallback rate you observe.

Cost mechanics

Every voter that produces a valid response costs 1 credit. So:

Configuration	Cost per call
target_responses=1, fastest	1
target_responses=3, consensus	3
target_responses=3, best_of_n	3
target_responses=3, all_must_agree	3 (or less if it short-circuits early on disagreement)

LLM fallback is free during the free-tier window. After that, fallback counts as a half-credit (rounded up to 1 for billing simplicity).

approval_method="all_must_agree" does short-circuit if it sees an early disagreement - once it’s mathematically impossible to reach unanimity, the call returns with provenance="escalated" and you’re only billed for the responses already in.

Worked examples

Example 1: A legal Q&A bot

Most questions are informational. A handful per day are stakes-heavy.

def legal_assistant(question: str, user_tier: str) -> str:
    answer = consult(
        question=question,
        domain_tags=["legal", "contracts"],
        target_responses=1 if user_tier == "free" else 3,
        approval_method="fastest" if user_tier == "free" else "consensus",
        wait_seconds=90,
        bypass_on_timeout=True,
    )

    if answer.provenance == "llm_fallback":
        return f"{answer.response_text}\n\n*(LLM-generated; for binding legal advice, consult a licensed attorney.)*"
    return answer.response_text

Free users get the fast cheap path. Paid users get consensus across three lawyers.

Example 2: An infra agent doing prod deploys

Deploys are irreversible. Don’t ship one without three humans unanimously approving.

def gated_deploy(plan: dict) -> bool:
    answer = consult(
        question=f"Approve this prod deploy?\n{plan['summary']}",
        context_brief=plan["full_runbook"],
        domain_tags=["ops", "prod-deploy"],
        target_responses=3,
        approval_method="all_must_agree",
        wait_seconds=600,
        require_human=True,
    )
    if answer.provenance != "human":
        return False  # escalated or human-unavailable → don't deploy
    return answer.response_text.strip().lower().startswith("approve")

Example 3: A medical-symptom triage bot

User-facing, stakes-heavy, but soft-real-time. Use consensus.

def triage(symptoms: list[str]) -> dict:
    answer = consult(
        question=f"Symptoms: {', '.join(symptoms)}. Urgency level?",
        domain_tags=["medical", "triage"],
        target_responses=3,
        approval_method="consensus",
        wait_seconds=180,
        bypass_on_timeout=True,
    )

    if answer.provenance == "escalated":
        return {"urgency": "see_a_doctor_now", "reason": "expert_disagreement"}
    return parse_urgency(answer.response_text)

When experts disagree on triage, the safest move is escalation, not guessing. The escalated branch handles that automatically.

Next step

You know when to call and how to aggregate. Now: where do the humans actually come from?

Nominated chains (your team) → Public marketplace (vetted experts) →