Weakness · Medium
CWE-1434: Insecure Setting of Generative AI/ML Model Inference Parameters
What it is
The product has a component that relies on a generative AI/ML model configured with inference parameters that produce an unacceptably high rate of erroneous or unexpected outputs.
- Severity (our rating)
- Medium
- Likelihood of exploit
- Not rated by MITRE
- Introduced during
- Build and Compilation, Installation, Patching and Maintenance
- Exploited in the wild
- No CVE in CISA's catalog
In more detail
Generative AI/ML models, such as those used for text generation, image synthesis, and other creative tasks, rely on inference parameters that control model behavior, such as temperature, Top P, and Top K. These parameters affect the model's internal decision-making processes, learning rate, and probability distributions. Incorrect settings can lead to unusual behavior such as text "hallucinations," unrealistic images, or failure to converge during training. The impact of such misconfigurations can compromise the integrity of the application. If the results are used in security-critical operations or decisions, then this could violate the intended security policy, i.e., introduce a vulnerability.
What an attacker can do
- Varies by Context (Integrity, Other)
- Unexpected State (Integrity, Other)
- Alter Execution Logic (Other)
How to prevent it
Implementation
- Develop and adhere to robust parameter tuning processes that include extensive testing and validation.
- Implement feedback mechanisms to continuously assess and adjust model performance.
Documentation
- Provide comprehensive documentation and guidelines for parameter settings to ensure consistent and accurate model behavior.
How to find it
- Automated Dynamic Analysis. Manipulate inference parameters and perform comparative evaluation to assess the impact of selected values. Build a suite of systems using targeted tools that detect problems such as prompt injection (CWE-1427) and other problems. Consider statistically measuring token distribution to see if it is consistent with expected results.
- Manual Dynamic Analysis. Manipulate inference parameters and perform comparative evaluation to assess the impact of selected values. Build a suite of systems using targeted tools that detect problems such as prompt injection (CWE-1427) and other problems. Consider statistically measuring token distribution to see if it is consistent with expected results.
H2 Security
Found it in your code?
We review where it appears, fix it with your team, and check it was not already used against you.
Talk to our teamT Academy
Stop writing it in the first place
Learn to build software and systems without weaknesses like this one, with practice on real hosts in your browser, every objective checked on the machine.
Start freeSource: MITRE Common Weakness Enumeration, CWE version 4.19.1. CWE is a trademark of The MITRE Corporation.