New Jailbreaks Allow Users to Manipulate GitHub Copilot

Researchers have discovered two new ways to manipulate GitHub's artificial intelligence (AI) coding assistant, Copilot, enabling the ability to bypass security restrictions and subscription fees, train malicious models, and more. The first trick involves embedding chat interactions inside of Copilot code, taking advantage of the AI's instinct to be helpful in order to get it to produce malicious outputs. The second method focuses on rerouting Copilot through a proxy server in order to communicate directly with the OpenAI models it integrates with. Researchers from Apex deem these issues vulnerabilities. GitHub disagrees, characterizing them as "off-topic chat responses," and an "abuse issue," respectively. In response to an inquiry from Dark Reading, GitHub wrote, "We continue to improve on safety measures in place to prevent harmful and offensive outputs as part of our responsible AI development. Furthermore, we continue to invest in opportunities to prevent abuse, such as the one described in Issue 2, to ensure the intended use of our products." "Copilot tries as best as it can to help you write code, [including] everything you write inside a code file," Fufu Shpigelman, vulnerability researcher at Apex explains. "But in a code file, you can also write a conversation between a user and an assistant." In the screenshot below, for example, a developer embeds within their code a chatbot prompt, from the perspective of an end user. The prompt carries ill intent, asking Copilot ...