As large language models (LLMs) gain prominence, concerns concerning their security weaknesses are being raised. Simon Willison, the maintainer of the open source Datasette project, is concerned about prompt injection, a severe security problem that might damage LLMs.
Willison noted that when developers create applications on top of language models, prompt injection becomes a problem. The developer creates a human-readable English description of what they want, integrates it with user input, and feeds it into the model. The issue emerges when the user input contradicts what the developer intends the model to perform in the initial section of the message.
Prompt injection is a concern not only for ChatGPT but also for other LLMs such as OpenAI’s chat.openai.com and Google’s Bard playground. Because of the security risk, ChatGPT may report incorrect information and take actions that violate its ethical training.
According to experts, quick injection is a long-standing security vulnerability that compromises application security. Willison pointed out that such issues in application security have existed for decades.
The sources for this piece include an article in TheRegister.