Part 3: Knowing when your agent doesn’t know: the confidence layer

[kasenur]

The most important number an agent produces isn’t its answer — it’s how sure it is. Compose that number from independent signals, check it’s calibrated, grade the high-stakes calls with a second model, and route the rest to humans well.This is Level 3 of a six-level maturity model for running LLM systems in production. Levels 1 and 2 got you to where the system works and you can see it working. Level 3 is confidence: the system acts on its own only when its calibrated confidence is high, grades the decisions that matter with an independent judge, and routes everything it’s unsure about to a human in a way that actually catches errors.These three things are one mechanism. Confidence is the dial that decides what runs automatically. A judge is one of the signals that dial is built from — and the trigger that pulls a decision off the automated path. The human handoff is where the dial sends everything below the line. Get all three right and you have a system that knows when it doesn’t know, and can therefore be trusted with more.Composed confidence: never trust a single sourceAsk most LLM systems “how sure are you?” and you get nothing useful — either no number, or the model’s self-reported confidence, which is notoriously miscalibrated (models are cheerfully certain when wrong). So you have to engineer a confidence signal. A production score is composed from independent signals, never lifted from the model’s own claim😡dataclass
class Signals:

# the model's own score — useful but weighted DOWN (overconfident)

# fraction of deterministic checks passed (0..1)

# True/False if this decision was judged; None if not sampled (most aren't)

# this slice's past accuracy (e.g. 0.99 vs 0.80)

WEIGHTS = {"model_self": 0.25, "verification": 0.35, "judge": 0.25, "historical": 0.15}

def compose(s: Signals) -> float:

# only ~8% of decisions are judged. When judge_agreed is None, DROP the judge term and

# renormalize the rest — an unjudged decision must not be penalized (False) or faked (True).

parts = {"model_self": s.model_self, "verification": s.verification, "historical": s.historical}

parts["judge"] = 1.0 if s.judge_agreed else 0.0

return sum(WEIGHTS[k] * v for k, v in parts.items()) / sum(WEIGHTS[k] for k in parts)The exact weights matter less than the principle: a single source of confidence is a single point of failure. Note model_self is deliberately the smallest weight — a model that’s confidently wrong gets dragged down by failed verification or a disagreeing judge, so “confidently wrong” can’t on its own clear the bar. The three other signals are also the ones you can check: deterministic verification either passed or didn’t, the judge either agreed or didn’t, and historical accuracy is a measured fact about this slice. The model’s opinion of itself is the one input you can’t independently verify, which is exactly why it gets the least say.Calibration: the number has to mean somethingA score is only useful if it’s calibrated — if “0.9” means “right ~90% of the time.” Check it: bucket decisions by predicted confidence and measure real accuracy per bucket (a reliability diagram).predicted bucket mean predicted actual accuracy verdict
---------------- -------------- --------------- ---------------------------------------
0.90 – 1.00

 

[YaseminTuran_34]

Forum bu tür içeriklerle daha değerli oluyor.

 

[OzlemBaran27]

Harika bilgiler, bu tür içerikleri çok faydalı buluyorum.