Why producing 75,000 tokens to resolve a easy logic puzzle proves that Reasoning is NOT Rule Adherence.
Should you ask a frontier AI mannequin to resolve a posh math downside, it shines. However what occurs in the event you pressure it to behave as a strict, zero-tolerance compiler for a totally made-up language?
For the Google DeepMind Govt Features observe, I made a decision to search out out. I constructed SymboLang — an artificial, zero-contamination symbolic language — and deployed a 170-case progressive stress take a look at.
What I discovered was a important blind spot in trendy LLMs: Syntax Drift through Overthinking.
The Premise: Testing True Cognitive Limits
Present benchmarks (like MMLU or HumanEval) reward open-ended reasoning or sample matching. They don’t take a look at inhibitory management — the power of a mannequin to suppress its pure urge to talk and strictly comply with a inflexible protocol below excessive cognitive load.
I created SymboLang with a strict grammar (prefixes, tenses, operators) and constructed a “Gauntlet” of 100 adversarial instances. The principles have been easy: output the precise symbolic code. One misplaced character equals failure.
The Large Reveal: The Effectivity Paradox
In Section 1 (easy sentences), fashions like Claude and GPT-5.4 aced the take a look at. However in Section 3 (The Gauntlet), introducing multi-clause conjunctions and temporal scopes brought on chaos amongst reasoning-optimized fashions.
Right here is the info that shocked me: Qwen 3 Subsequent 80B Considering achieved excessive accuracy, but it surely paid an enormous operational tax. It burned over 75,000 output tokens to resolve 100 deterministic instances. That’s a median of 750 tokens per case simply to output a single line of code!
That is the Effectivity Paradox: Extreme deliberation actively degrades inhibitory management. The mannequin brute-forced the syntax guidelines via sheer computational overhead.
The Failure Mode: Preamble Leakage
One other extreme situation emerged with fashions like DeepSeek-R1. Underneath the cognitive stress of the Gauntlet, the mannequin suffered from Preamble Leakage. Regardless of strict “No preamble” system prompts, it generated verbose, hallucinated English textual content, misplaced management of the syntax, and hallucinated invalid operators (like != as a substitute of !).
When reasoning fashions “suppose tougher,” they neglect the specific grammar guidelines they parsed simply seconds in the past.
The Repair: Engineering the NSE Normalizer
To make sure my benchmark graded true reasoning and never simply formatting errors, I couldn’t simply fail fashions for being chatty. I engineered a customized extraction algorithm: the NSE Normalizer (Normalized String Equivalence).
It deterministically strips out Chain-of-Thought traces, Markdown blocks, and conversational noise to isolate and rating the pure logic beneath.
Conclusion: Compilers vs. Reasoners
This benchmark proves that reasoning doesn’t mechanically produce rule adherence. As duties turn into extra complicated, the act of “considering tougher” can erode a mannequin’s govt operate. For strict deterministic pipelines (like API routing or code technology), compact, instruction-following fashions (like Claude 4.5 or Gemini 3.1 Flash-Lite) are far superior and infinitely cheaper than heavy reasoning fashions.
SymboLang makes the chief operate hole measurable.
Take a look at the complete knowledge, Kaggle pocket book, and the NSE Normalizer code on my GitHub: [https://github.com/meowmet/SymboLang-AGI-Benchmark/]
Kaggle: [https://www.kaggle.com/competitions/kaggle-measuring-agi/writeups/meowmet-synthetic-protocol]
