Weakness · High
CWE-1427: Improper Neutralization of Input Used for LLM Prompting
What it is
The product uses externally-provided data to build prompts provided to large language models (LLMs), but the way these prompts are constructed causes the LLM to fail to distinguish between user-supplied inputs and developer provided system directives.
- Severity (our rating)
- High
- Likelihood of exploit
- Not rated by MITRE
- Introduced during
- Architecture and Design, Bundling, Implementation, Integration, System Configuration
- Exploited in the wild
- No CVE in CISA's catalog
In more detail
When prompts are constructed using externally controllable data, it is often possible to cause an LLM to ignore the original guidance provided by its creators (known as the "system prompt") by inserting malicious instructions in plain human language or using bypasses such as special characters or tags. Because LLMs are designed to treat all instructions as legitimate, there is often no way for the model to differentiate between what prompt language is malicious when it performs inference and returns data. Many LLM systems incorporate data from other adjacent products or external data sources like Wikipedia using API calls and retrieval augmented generation (RAG). Any external sources in use that may contain untrusted data should also be considered potentially malicious.
What an attacker can do
- Execute Unauthorized Code or Commands (Confidentiality, Integrity, Availability)
- Varies by Context (Confidentiality, Integrity, Availability)
- Read Application Data (Confidentiality, Access Control)
- Modify Application Data (Integrity, Access Control)
- Gain Privileges or Assume Identity (Access Control)
How to prevent it
Architecture and Design
- LLM-enabled applications should be designed to ensure proper sanitization of user-controllable input, ensuring that no intentionally misleading or dangerous characters can be included. Additionally, they should be designed in a way that ensures that user-controllable input is identified as untrusted and potentially dangerous.
- LLM-enabled applications should be designed to ensure proper sanitization of user-controllable input, ensuring that no intentionally misleading or dangerous characters can be included. Additionally, they should be designed in a way that ensures that user-controllable input is identified as untrusted and potentially dangerous.
Implementation
- LLM prompts should be constructed in a way that effectively differentiates between user-supplied input and developer-constructed system prompting to reduce the chance of model confusion at inference-time.
- Ensure that model training includes training examples that avoid leaking secrets and disregard malicious inputs. Train the model to recognize secrets, and label training data appropriately. Note that due to the non-deterministic nature of prompting LLMs, it is necessary to perform testing of the same test case several times in order to ensure that troublesome behavior is not possible. Additionally, testing should be performed each time a new model is used or a model's weights are updated.
Installation
- During deployment/operation, use components that operate externally to the system to monitor the output and act as a moderator. These components are called different terms, such as supervisors or guardrails.
System Configuration
- During system configuration, the model could be fine-tuned to better control and neutralize potentially dangerous inputs.
How to find it
- Dynamic Analysis with Manual Results Interpretation. Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.
- Dynamic Analysis with Automated Results Interpretation. Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.
- Architecture or Design Review. Review of the product design can be effective, but it works best in conjunction with dynamic analysis.
H2 Security
Found it in your code?
We review where it appears, fix it with your team, and check it was not already used against you.
Talk to our teamT Academy
Stop writing it in the first place
Learn to build software and systems without weaknesses like this one, with practice on real hosts in your browser, every objective checked on the machine.
Start freeSource: MITRE Common Weakness Enumeration, CWE version 4.19.1. CWE is a trademark of The MITRE Corporation.