The Guardian Shows Hidden Text Can Manipulate ChatGPT’s Search Results

The British newspaper The Guardian revealed that AI-powered search engines can be manipulated by websites with hidden content.

In a test using OpenAI’s ChatGPT search engine feature, researchers asked for a summary of a fake website containing malicious information to alter the AI’s response—a vulnerability known as prompt injection—and the AI was susceptible to it, even favoring the third party’s instructions.

To prove this, The Guardian’s team considered a fake website of a camera’s product page—featuring good and bad reviews—with hidden instructions to give a positive review and disregard the bad reviews, and ChatGPT included only positive reviews in its summary. They also proved that AI can return malicious codes.

“The simple inclusion of hidden text by third parties without instructions can also be used to ensure a positive assessment, with one test including extremely positive fake reviews which influenced the summary returned by ChatGPT,” wrote the newspaper.

A cybersecurity researcher at CyberCX, Jacob Larsen, said that this vulnerability could be of “high risk” as people could create websites specifically to deceive users, especially once it reaches a wider audience. OpenAI was warned about this security risk.

The journal also highlighted the case of a cryptocurrency enthusiast who used ChatGPT to write the code for a crypto project and stole their credentials, making the programmer lose over $2,000.

“They’re simply asking a question, receiving an answer, but the model is producing and sharing content that has basically been injected by an adversary to share something that is malicious,” said Larsen.

OpenAI warns about possible mistakes and errors in its use, but researchers are concerned about future web practices with AI-powered search engines.