Open Source AI Models: Perfect Storm for Malicious Code, Vulnerabilities

Attackers are finding more and more ways to post malicious projects to Hugging Face and other repositories for open source artificial intelligence (AI) models, while dodging the sites' security checks. The escalating problem underscores the need for companies pursuing internal AI projects to have robust mechanisms to detect security flaws and malicious code within their supply chains. Hugging Face's automated checks, for example, recently failed to detect malicious code in two AI models hosted on the repository, according to a Feb. 3 analysis published by software supply chain security firm ReversingLabs. The threat actor used a common vector — data files using the Pickle format — with a new technique, dubbed "NullifAI," to evade detection. While the attacks appeared to be proofs-of-concept, their success in being hosted with a "No issue" tag shows that companies should not rely on Hugging Face's and other repositories' safety checks for their own security, says Tomislav Pericin, chief software architect at ReversingLabs. "You have this public repository where any developer or machine learning expert can host their own stuff, and obviously malicious actors abuse that," he says. "Depending on the ecosystem, the vector is going to be slightly different, but the idea is the same: Someone's going to host a malicious version of a thing and hope for you to inadvertently install it." Companies are quickly adopting AI, and the majority are also establishing internal projects using op...