How nice that state-of-the-art LLMs reveal their reasoning ... for miscreants to exploit

Analysis AI models like OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking can mimic human reasoning through a process called chain of thought. That process, described by Google researchers in 2022, involves breaking the prompts put to AI models into a series of intermediate steps before providing an answer. It can also improve AI safety for some attacks while undermining it at the same time. We previously went over chain-of-thought reasoning here, in our hands-on guide to running DeepSeek R1 locally. Researchers affiliated with Duke University in the US, Accenture, and Taiwan's National Tsing Hua University have now devised a jailbreaking technique to exploit chain-of-thought (CoT) reasoning. They describe their approach in a pre-print paper titled: "H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking." Specifically, the boffins probed the aforementioned AI models in their cloud-hosted forms, including R1, via their respective web-based user interfaces or APIs. Associated data and code have been published to GitHub. The team – Martin Kuo, Jianyi Zhang, Aolin Ding, Qinsi Wang, Louis DiValentin, Yujia Bao, Wei Wei, Da-Cheng Juan, Hai Li, and Yiran Chen – devised a dataset called Malicious-Educator that contains elaborate prompts designed to bypass the AI models' safety checks – the so-called guardrails put in place to prevent harmful responses. They use th...