Puan
28,728
Çözümler
0
- Konum
- Freanchey
- Mesajlar
- 3,049
- Katılım
- 2 Eki 2026
- Tepkime puanı
- 3,565
- Puan
- 28,728
- Web sitesi
- glomaxgpt.com
New StoryDesigning Agent-Assisted Abuse Detection With Classifier Routing and Feedback LoopsbySumitbySumit|@sumitzSenior Staff Software Engineer and Technical Lead at WhatsApp with over seven years of experience building AI/ML systems at scaleOctober 9th, 2026TLDR Your browser does not support the audio element.Speed1xVoiceDr. One Ms. Hacker bySumit@sumitzbySumit|@sumitzSenior Staff Software Engineer and Technical Lead at WhatsApp with over seven years of experience building AI/ML systems at scaleStory's CredibilitybySumit|@sumitzSenior Staff Software Engineer and Technical Lead at WhatsApp with over seven years of experience building AI/ML systems at scaleStory's CredibilityAn LLM agent will write you a convincing abuse report on almost any account you point it at. Give it tools to pull the account’s history and its connections, and a few minutes later you have a tidy paragraph explaining why this looks like a scam ring. The first time I watched one do it, I assumed the hard part was over.
It wasn’t. Writing an explanation of why something looks abusive is usually the easy part. Whether an agent provides real value in production depends on the system around the agent: when it gets called, whether its findings ever improve conventional classifiers, whether anyone can explain a ban the agent recommended six months later, and what an attacker can do to circumvent agents.
Abuse detection is the job of finding accounts, content, and coordinated campaigns that go against a platform’s rules and policies (fraud, scams, misinformation, etc.), all while malicious actors actively work to stay ahead of the abuse detection system.
Start With Routing, Not Reasoning
Picture a wave of freshly created accounts posting scam comments under popular videos. Any single account looks unremarkable - a genuine-looking name, a stock photo, a few posts. But the graph of accounts, the domains they link to, the devices they share, and the timing of their activity looks coordinated in a way no set of unrelated users ever would.
That’s a good case for an agent: unfamiliar, relational, and worth a careful look. The mistake is making the agent the first stop for every suspicious account. Most suspicious accounts aren’t novel at all - they're the same abuse you've seen ten thousand times, and you already have classifiers that catch them.
So before you build the agent, build the router that decides which cases the agent handles.
Here’s roughly how I route them. (In what follows, “entity” just means the thing under investigation - an account, a piece of content, or a cluster of them.)
classifier_score = known_abuse_model(entity)
novelty_score = anomaly_model(entity)
if classifier_score >= 0.95:
send_to_known_abuse_queue()
elif classifier_score <= 0.05:
mark_likely_benign()
elif novelty_score >= 0.90:
send_to_agent_investigation()
else:
send_to_sampling_or_human_review()
Treat those numbers as placeholders. I tune real ones against precision-recall curves and whatever reviewers can absorb, weighing what a missed scam costs against what a wrongful ban costs.
The rule underneath it is simple: the agent runs only when the fast classifier is unsure and the entity is unusual. Everything else has a simpler solution.
Known abuse belongs to a fast classifier - a gradient-boosted tree or a small transformer trained on behavioral features. This catches everything that resembles labeled abuse you've already seen, at a fraction of the latency and cost of an agent call.
The novelty path asks a different question: how unusual is this entity compared to everything else you’ve seen? You can answer that with an isolation forest, with nearest-neighbor distance from known abuse clusters, or with community detection on a graph of accounts and the signals they share. The choice depends on your data: what matters is producing a single anomaly score per entity.
The failure to watch for is the agent drifting into an expensive classifier. Running it on traffic a cheap classifier could have handled burns money for no gain, because an agent invocation costs far more than classifier inference.
Make the Output Useful After the Investigation
Here's a failure I've seen in practice. An agent investigates a cluster of malicious accounts and writes a clean, readable summary. A reviewer reads it and applies an enforcement action - a ban, a takedown, a rate limit. Two weeks later, a slightly modified version of the same attack shows up, and the entire process runs again from scratch.
The agent did the work twice, but the classifier learned nothing either time.
This is the most expensive failure mode I know of, and it almost always comes from a single design choice: the agent’s output is unstructured text instead of a structured training signal. A paragraph is something a human reads once and forgets. A structured record is something the rest of your system can actually consume.
The agent should emit something closer to a training row:
case_id: case_123
entities: account_1, account_2, domain_7
suspected_policy: scam
classifier_score: 0.58
novelty_score: 0.96
distinguishing_features:
- recent account creation cluster
agent_findings:
- claim: accounts share infrastructure
- evidence: graph_edge_1, domain_lookup_4
reviewer_label: confirmed_positive
importance_weight: 0.8
Concretely, the agent emits the entity IDs involved, the features that separate those accounts from benign traffic, a label (confirmed positive, confirmed negative, or ambiguous) attributed to the process that assigned it, and the evidence trail behind that label. Attach a sample weight, and you have a row in the classifier's training set rather than a text report. The features record what made the case different; the evidence IDs make the label auditable long after anyone remembers the details.But you can’t retrain on all of it. You need a strategy for deciding which rows are worth learning from, because retraining on everyth
It wasn’t. Writing an explanation of why something looks abusive is usually the easy part. Whether an agent provides real value in production depends on the system around the agent: when it gets called, whether its findings ever improve conventional classifiers, whether anyone can explain a ban the agent recommended six months later, and what an attacker can do to circumvent agents.
Abuse detection is the job of finding accounts, content, and coordinated campaigns that go against a platform’s rules and policies (fraud, scams, misinformation, etc.), all while malicious actors actively work to stay ahead of the abuse detection system.
Start With Routing, Not Reasoning
Picture a wave of freshly created accounts posting scam comments under popular videos. Any single account looks unremarkable - a genuine-looking name, a stock photo, a few posts. But the graph of accounts, the domains they link to, the devices they share, and the timing of their activity looks coordinated in a way no set of unrelated users ever would.
That’s a good case for an agent: unfamiliar, relational, and worth a careful look. The mistake is making the agent the first stop for every suspicious account. Most suspicious accounts aren’t novel at all - they're the same abuse you've seen ten thousand times, and you already have classifiers that catch them.
So before you build the agent, build the router that decides which cases the agent handles.
Here’s roughly how I route them. (In what follows, “entity” just means the thing under investigation - an account, a piece of content, or a cluster of them.)
classifier_score = known_abuse_model(entity)
novelty_score = anomaly_model(entity)
if classifier_score >= 0.95:
send_to_known_abuse_queue()
elif classifier_score <= 0.05:
mark_likely_benign()
elif novelty_score >= 0.90:
send_to_agent_investigation()
else:
send_to_sampling_or_human_review()
Treat those numbers as placeholders. I tune real ones against precision-recall curves and whatever reviewers can absorb, weighing what a missed scam costs against what a wrongful ban costs.
The rule underneath it is simple: the agent runs only when the fast classifier is unsure and the entity is unusual. Everything else has a simpler solution.
Known abuse belongs to a fast classifier - a gradient-boosted tree or a small transformer trained on behavioral features. This catches everything that resembles labeled abuse you've already seen, at a fraction of the latency and cost of an agent call.
The novelty path asks a different question: how unusual is this entity compared to everything else you’ve seen? You can answer that with an isolation forest, with nearest-neighbor distance from known abuse clusters, or with community detection on a graph of accounts and the signals they share. The choice depends on your data: what matters is producing a single anomaly score per entity.
The failure to watch for is the agent drifting into an expensive classifier. Running it on traffic a cheap classifier could have handled burns money for no gain, because an agent invocation costs far more than classifier inference.
Make the Output Useful After the Investigation
Here's a failure I've seen in practice. An agent investigates a cluster of malicious accounts and writes a clean, readable summary. A reviewer reads it and applies an enforcement action - a ban, a takedown, a rate limit. Two weeks later, a slightly modified version of the same attack shows up, and the entire process runs again from scratch.
The agent did the work twice, but the classifier learned nothing either time.
This is the most expensive failure mode I know of, and it almost always comes from a single design choice: the agent’s output is unstructured text instead of a structured training signal. A paragraph is something a human reads once and forgets. A structured record is something the rest of your system can actually consume.
The agent should emit something closer to a training row:
case_id: case_123
entities: account_1, account_2, domain_7
suspected_policy: scam
classifier_score: 0.58
novelty_score: 0.96
distinguishing_features:
- recent account creation cluster
agent_findings:
- claim: accounts share infrastructure
- evidence: graph_edge_1, domain_lookup_4
reviewer_label: confirmed_positive
importance_weight: 0.8
Concretely, the agent emits the entity IDs involved, the features that separate those accounts from benign traffic, a label (confirmed positive, confirmed negative, or ambiguous) attributed to the process that assigned it, and the evidence trail behind that label. Attach a sample weight, and you have a row in the classifier's training set rather than a text report. The features record what made the case different; the evidence IDs make the label auditable long after anyone remembers the details.But you can’t retrain on all of it. You need a strategy for deciding which rows are worth learning from, because retraining on everyth