Book a Security Review

Weakness · High

CWE-1426: Improper Validation of Generative AI Output

What it is

The product invokes a generative AI/ML component whose behaviors and outputs cannot be directly controlled, but the product does not validate or insufficiently validates the outputs to ensure that they align with the intended security, content, or privacy policy.

MITRE CWE 4.19.1

Severity (our rating)
High
Likelihood of exploit
Not rated by MITRE
Introduced during
Architecture and Design, Implementation
Exploited in the wild
No CVE in CISA's catalog

What an attacker can do

  • Execute Unauthorized Code or Commands (Integrity)
  • Varies by Context (Integrity)

How to prevent it

Architecture and Design

  • Since the output from a generative AI component (such as an LLM) cannot be trusted, ensure that it operates in an untrusted or non-privileged space.

Operation

  • Use "semantic comparators," which are mechanisms that provide semantic comparison to identify objects that might appear different but are semantically similar.
  • Use components that operate externally to the system to monitor the output and act as a moderator. These components are called different terms, such as supervisors or guardrails.

Build and Compilation

  • During model training, use an appropriate variety of good and bad examples to guide preferred outputs.

How to find it

  • Dynamic Analysis with Manual Results Interpretation. Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.
  • Dynamic Analysis with Automated Results Interpretation. Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.
  • Architecture or Design Review. Review of the product design can be effective, but it works best in conjunction with dynamic analysis.

CWE-1426 on MITRE's CWE site

H2 Security

Found it in your code?

We review where it appears, fix it with your team, and check it was not already used against you.

Talk to our team

T Academy

Stop writing it in the first place

Learn to build software and systems without weaknesses like this one, with practice on real hosts in your browser, every objective checked on the machine.

Start free

Source: MITRE Common Weakness Enumeration, CWE version 4.19.1. CWE is a trademark of The MITRE Corporation.