arXiv:2611.23074v1 Announce Type: cross
Abstract: AI agents are governed by hundreds of natural language criteria, yet existing verifiers cannot efficiently evaluate an agent’s output against a large set of such criteria and report which are satisfied, violated, or inapplicable. We introduce the VFM, a tree-attention verifier that encodes the shared context and target text, and scores N natural language criteria in parallel, returning independent per-criterion ternary verdicts. The VFM is pretrained on CriteriaBank, a corpus of 394K (context, target, criterion, verdict) tuples spanning over 321K distinct criterion strings. A finetuned VFM-4B outperforms GPT-5.4 on a synthetic governance benchmark. On the DynaBench multi-criterion benchmark, it outperforms DynaGuard-4B on trace-level failure detection, and its per-criterion verdicts localize the violated rule and support focused escalation to a larger chain-of-thought judge. We show that CriteriaBank pretraining improves cross-domain transfer, particularly in the low-data regime, while matching a strong direct-finetuning control at full data. On a single H100, VFM-4B evaluates 1,000 criteria against a 4K-token context in under 2s at <11 GB VRAM, making it suitable for runtime verification.
Trending
- Best Laptops for Travel (2026): Which to Take on the Road
- [2611.25796] How Reliable Are Explanations of Language Models That Detect Insider Threats?
- Lifting Classifier Training with Latent Gaussian Mixture Models
- New open model Inkling 2 Mini thinks in loops to solve problems with fewer words and still runs on a laptop
- RouteVerify: Routing Interventions for Step-Level Verification in Mixture-of-Experts Models
- US technology stocks suffer their worst weekly loss since the pandemic as AI doubts pile up
- [2609.04910] Fast Gauss Sums via Flash Attention
- 17 Best Gifts for Women in the Enshittocene (2026)
