When “Competency” in Reasoning Opens the Door to Vulnerability: Jailbreaking LLMs via Novel Complex Ciphers
Abstract
Shows that as LLMs get better at reasoning they become more susceptible to novel jailbreaks. Introduces ACE and LACE — attacks that encode malicious queries with custom and layered ciphers — and CipherBench to measure cipher-decoding ability.
The paradox
LLM safety training has mostly learned to recognize harm written in natural language or in common ciphers like Base64 and ROT-13. But a stronger reasoner can decode ciphers it has never been trained on — and that same ability is exactly what makes it exploitable. Encode a harmful request in a novel, user-defined cipher and it sails past the safety filter as meaningless text; then the model's reasoning quietly reconstructs the request and answers it. The paper's uncomfortable finding: the better a model is at decoding ciphers, the more vulnerable it becomes.
ACE — Attacks using Custom Encryptions
ACE is a zero-shot attack: no optimization, no gradients, no model access beyond a prompt. It encodes the harmful query in a cipher, hands the model the decoding scheme, and adds a short priming sentence to nudge toward a completed answer. The ciphers span three tiers by how well-known they are — the insight being that safety training can only pre-empt ciphers it has already seen:
- Common — Base64, ROT-13, Pig Latin, LeetSpeak. Easy to decode (high DSR) but easy to defend: low attack success, likely already in safety training.
- Uncommon — Keyboard Cipher (shift one key right on QWERTY), Upside-Down, Word-Reversal. Rarely in the literature, so rarely defended.
- Novel — user-designed schemes absent from cryptographic literature: Word-Substitution, Grid Encoding (a 5×5 chess-notation grid), ASCII-Art. Impossible to pre-train against.
CipherBench — can the model even decode it?
An attack only works if the model can reconstruct the request, so the paper first measures raw decoding skill. CipherBench is 240 safe sentences (short/long, question/declarative) across ten ciphers, scored by Decryption Success Rate (DSR). A telling control: when the same sentences are replaced with random characters, decoding collapses — showing the models lean on reconstructing plausible English, not on mechanically inverting the cipher.
ACE results
On the 50-sample AdvBench subset (14 harm categories), measured by Attack Success Rate (ASR). Common ciphers decode easily but rarely jailbreak; uncommon and novel ciphers match or beat them despite being zero-shot:
| Attack (ACE cipher) | Tier | GPT-4o ASR | gpt-oss-20b ASR | Llama-3.1-70B ASR |
|---|---|---|---|---|
| Base64 | common | 0.10 | 0.36 | 0.44 |
| ROT-13 | common | 0.24 | 0.48 | 0.28 |
| Upside-Down | uncommon | 0.30 | 0.60 | 0.06 |
| Word Reversal | uncommon | 0.40 | 0.16 | 0.32 |
| Grid Encoding | novel | 0.36 | 0.48 | 0.08 |
| Word Substitution | novel | 0.40 | 0.26 | 0.54 |
Notably, zero-shot ACE matches the optimization-based AutoDAN on Llama-3.1-8B and ranks second-best on Llama-3.1-70B — no search required.
LACE — layering the ciphers
LACE stacks a second encoding on top of a Word-Substitution Cipher, pushing complexity past what safety training anticipates. Layering lowers decode rates for weak models — but strong reasoners hold their decoding and get jailbroken more often. On gpt-oss-20b, ASR climbs from 60% (ACE) to 72% (LACE); on GPT-4o, easier layered ciphers like Word-Reversal reach 78% ASR at 92% decode — nearly double the single-cipher rate.
| LACE (Word-Sub + …) | GPT-4o DSR | GPT-4o ASR | gpt-oss-20b DSR | gpt-oss-20b ASR |
|---|---|---|---|---|
| Word Reversal (sentence) | 0.92 | 0.78 | 0.74 | 0.38 |
| ROT-13 (sentence) | 0.80 | 0.72 | 0.74 | 0.56 |
| Keyboard (both) | 0.00 | 0.08 | 0.86 | 0.72 |
| Grid Encoding (sentence) | 0.26 | 0.30 | 0.78 | 0.70 |
Two mechanics fall out of the data. Encrypting the whole query is easier to crack than encrypting only the cipher's mappings — the model self-corrects from surrounding words when it has full context. And the layering only pays off on models whose decode rate survives it: weak models can't decode the layers, so the extra complexity is wasted on them. That is the paper's central trade-off — decoding competence and jailbreak vulnerability rise together.
Over-defensiveness
The attacks also expose the opposite failure: models sometimes refuse perfectly benign ciphered text simply because it looks obfuscated. The paper quantifies this over-refusal across the ACE encodings — a reminder that bolting on cipher-blind filters trades one failure mode for another.
Why it matters
The defensive implication is uncomfortable: you cannot enumerate every user-invented cipher in advance, and the models most worth protecting — the strong reasoners — are precisely the ones that can decode novel schemes. Safety training needs to generalize to unseen encodings, and cipher complexity may need to be matched to a model's decoding ability rather than applied blindly.
Resources
Paper: OpenReview — NeurIPS 2025 Workshop · Extended version: arXiv:2402.10601 · Code & data: github.com/DivijH/jailbreak_cryptography
Content warning: the paper contains model-generated content that may be offensive.