The same image input, different consequences downstream

An image AI that finds defects in factory visual inspection, or a person-detection AI that watches for entry into a work area, returns a judgment from the image: a pass/fail result or a detection (a decision system). An AI assistant that reads an incoming document or a screenshot and then summarizes it or enters it into a business system produces text and function calls (a generative system).

In both, you put in an image and a result comes out, but threat analysis does not start from the same place. In a decision system, the judgment leads to rejecting a defective part or stopping equipment. In a generative system, the text or function call that was produced leads to a record being created in a business system or a command being executed.

When what happens automatically differs, so do the inputs to watch for and the metrics to measure. Threat analysis starts not with the accuracy of the judgment or the generation, but with what happens downstream of the result.

“Degradation,” “adversarial input,” and “instructions inside the image” are not the same thing

Problems involving image input tend to be lumped together as “attacks” or “prompt injection.” In practice, three different problems are mixed together there, with different aims, different paths, and different metrics.

Ordinary variation in conditions, with no attack intent. Low light, compression, occlusion, focus drift. It also covers changes carried out as ordinary processing, such as correction or compression, that end up changing what is recognized. These are the subject of robustness evaluation: measuring how the detection rate and the false-detection rate move with conditions.

Changes made to target recognition. The input is altered with the aim of changing the judgment itself. The US NIST taxonomy of adversarial machine learning (NIST AI 100-2e2025) classifies attacks that modify the input at inference time to change the classification as evasion attacks. What you measure here are misses and false detections under the change conditions you assume.

Advanced methods are not the only case to plan for. In the case that document cites, a commercial phishing webpage detector, what was used was not an optimized perturbation but simple manipulation such as cropping, masking, or blurring. That is not a demonstration that holds for image or person detection in general, but it is a reason to include simple manipulation in the baseline of an evaluation.

Instructions written in the image. Here the problem is that a multimodal AI treats text or characters written into an image as instructions from the user. NIST notes that because generative AI processes data and instructions through the same channel, instructions can be injected through external resources controlled by a third party, and classifies this as indirect prompt injection.

What you measure here is not a detection rate but whether an illegitimate instruction was followed and whether a tool was executed. OWASP’s LLM01:2025 lists techniques for hiding instructions in images, and sets out least privilege, human approval for high-risk actions, segregating external content, and adversarial testing as countermeasures.

For the third, measures on the text side are not enough. NIST states that combining multiple modalities does not automatically make a model robust against an attack on a single modality.

Four questions to answer about your own system

Which of the three to assume, and how far, differs from system to system. Writing out the following four points gives you the skeleton of a threat model.

  1. Who can change which input, and how far? The people who can touch the lighting or the cameras on site, who can attach something to the object being inspected, who create the documents and screens that arrive, who administer the web pages the AI reads. Each of them can change a different range of things.
  2. Which decisions or actions have to be protected? Pass/fail judgments, granting passage or building access, reading amounts and quantities, opening a ticket, executing a command, and so on.
  3. What happens automatically after that decision? Does the judgment become an action directly, or does it pass through a check by a person or a policy?
  4. What would you have to measure to have evaluated that risk? Metrics for misses and false detections, or whether instructions were followed and tools executed?

What to measure for each change you assume

Input and the change you assume Possible impact Example verification and acceptance criteria
Degradation such as low light, compression, or occlusion (ordinary variation with no attack intent) More misses and false detections Grade the conditions and measure detection and false-detection rates per condition at an operating point fixed in advance
Something attached to the object, a pattern, or partial covering (a change aimed at recognition) Missing a specific object, misclassifying it as another class Keep simple changes as the baseline and compare the change conditions step by step on the same scenes
Text or characters inside the image (instructions to a multimodal AI) Following an illegitimate instruction, tampering with a summary or extraction result, tool calls Evaluate whether instructions are followed both with and without tool permissions, and list the actions that run without human approval
Downstream processing of the judgment or the generated result A miss becoming an action, a wrong instruction becoming an execution Confirm that decision, authorization, and execution are separated, and identify the actions that run without confirmation

This breakdown and these criteria are our design proposal, not requirements laid down by a standard.

Grading conditions and measuring them, as in the second row, is illustrated by our study Assessing Person-Detector Misses: An AI Security Perspective. It replaces part of the person region with a texture and compares detection rates on the same scenes while varying mask extent and texture conditions. It is not a study that shows a vulnerability in a particular product or the effectiveness of a defense; refer to it as an example of how to separate conditions and how to measure them.

Measures that improve recognition, and measures that limit the consequences

Both misses and instruction-following split into two directions of countermeasure. One is to evaluate and improve the recognition or the output itself. The other is to limit, at a permission boundary, the effect that result has on downstream actions. Both problems need both.

For misses and false detections, the measures that improve recognition are setting the operating point, adding sensor redundancy, and retraining per condition. On the side that limits the consequences, you do not let a judgment alone settle an action; you insert a confirmation.

For instruction-following, the measures that improve the output are separating and marking externally sourced content as data, validating the output format, and adding countermeasures for the weaknesses adversarial testing finds and then confirming their effect. On the side that limits the consequences are least privilege and human approval for high-risk actions. Least privilege limits the damage when injection succeeds, but it does not eliminate injection itself.

In both problems, the foundation is a design that keeps the AI’s output from going straight to execution. Separate the decision (the AI’s output), the authorization (a policy or a person’s confirmation), and the execution (what is reflected in a business system or an actuator). With those apart, you can trace afterwards what happened at which layer, and measure the effect of countermeasures layer by layer.

Where to start

First, list the systems in your organization that take images as input, and write one line for each: whether it is a decision system or a generative system, and what its result sets in motion automatically. Once that is filled in, you can answer the four questions above, and the changes to assume and the metrics to measure follow.

Build the evaluation in this order: ordinary variation, simple manipulation, and advanced manipulation if needed. Fix the operating point and the number of scenes before you measure, and do not skip the baseline of simple manipulation; that baseline is what lets you read the difference when you add conditions. For a generative system, draw up first how externally sourced content is handled and which actions require human approval.

Work through that and you can decide how to allocate countermeasures between improving recognition and output and restricting downstream actions. Set priorities not only on the degradation or instruction-following you observed, but together with the size of the impact, how likely it is, and the capability you assume of an adversary. You also end up able to explain, in a deployment decision or an audit, what you evaluated and how.

Talk to us

Deciding which conditions to measure and how, while you are still at the PoC stage, gives you evidence you can use directly when deciding whether to go to production. In our custom AI development, robustness testing and adversarial resilience assessment are built into the development process. To discuss the threat model and evaluation design for a system that takes image inputs, use the contact form.

Reference materials

The classification in this article follows the way the sources above organize the subject; the threat-model questions, the verification items, and the suggested first steps are U-Rec, Inc.’s general design proposals.