What are adversarial attacks?

How adversarial attacks work

Machine learning models learn by finding patterns in data. When you show a model thousands of pictures of stop signs, it learns to recognize the visual patterns that define ‘stop sign’—the red octagon, the white border, the letters S-T-O-P. Adversarial attacks exploit this learning process by adding carefully calculated noise or patterns to an input that push the model’s prediction in a specific direction [1].

The key insight is that these changes are often too small for humans to notice. A few pixels altered in a photo, a slight rotation of a traffic sign, or a carefully chosen word in a prompt—these subtle modifications can completely change how the model interprets the input [2].

Attack types and taxonomy

Security researchers categorize adversarial attacks along several dimensions:

By attacker knowledge:

• White-box attacks: The attacker has full access to the model’s internal parameters, architecture, and training data. This allows them to calculate exactly how to craft the most effective adversarial input [3].
• Black-box attacks: The attacker can only observe the model’s inputs and outputs—like sending queries and getting answers—without seeing inside. They use these observations to estimate how the model works and craft attacks accordingly [3].

By goal:

• Targeted attacks: The attacker wants the model to produce a specific wrong answer—for example, making a security camera classify an intruder as ’empty room.’ [3]
• Untargeted attacks: The attacker simply wants the model to fail in any way—any misclassification counts as success [3].

By method:

• Evasion attacks: The most common type. The attacker manipulates input data at inference time (when the model is making predictions) without modifying the model itself. The model’s training remains intact; only the input is corrupted [3].
• Poisoning attacks: The attacker manipulates the training data before the model learns from it, corrupting the model’s ‘education’ from the start [3].

Classic attack methods

Two foundational methods illustrate how adversarial attacks are constructed:

Fast Gradient Sign Method (FGSM)

Introduced by Goodfellow et al. in 2015, FGSM is a one-step white-box attack. It works by calculating which direction to nudge each pixel (or input feature) to make the model’s prediction more wrong, then adding a small perturbation in that direction. The formula is simple: x_adv = x + ε × sign(∇J), where ε controls the perturbation size. FGSM is fast and widely used for adversarial training—teaching models to resist attacks by exposing them to adversarial examples during training [1].

Projected Gradient Descent (PGD)

PGD is an iterative extension of FGSM—it takes multiple small steps instead of one big one, and after each step it ‘projects’ the result back into an allowed range to keep the perturbation small. Considered one of the strongest first-order attacks, PGD is the standard benchmark for evaluating model robustness [4].

Physical-world example: Autonomous vehicles

Adversarial attacks aren’t just theoretical. A 2025 study by researchers at UC Irvine and Drexel University tested physical adversarial attacks on five production vehicles (Tesla Model 3, Toyota Camry, Nissan Sentra, Mazda CX-30, Hyundai Tucson). By placing specially designed stickers on stop signs and speed limit signs, they demonstrated that even modern traffic sign recognition systems can be fooled [3].

The study categorized physical attacks into three types:

• Hiding attacks: Make a real sign undetectable—the car doesn’t see the stop sign at all
• Appearing attacks: Trigger a false detection where no sign exists—causing unexpected braking
• Misclassification attacks: Alter how a sign is interpreted—a stop sign reads as a 45 mph speed limit sign [3]

‘Appearing’ attacks were significantly more successful than ‘hiding’ attacks, partly because many commercial systems use ‘spatial memorization’ that remembers a sign’s location after first detection [3].

Defenses and mitigations

Defending against adversarial attacks is an active area of research. The primary defenses include:

Adversarial training: The most widely used defense. The model is trained on adversarial examples alongside normal data, teaching it to recognize and resist manipulation. This is computationally expensive but effective against known attack types [4].

Input preprocessing: Detecting and removing adversarial perturbations before they reach the model—like filtering noise from an image.

Certified defenses: Provide mathematical guarantees about a model’s robustness within specific bounds. If an adversarial perturbation stays within a defined ‘safe zone,’ the model is provably correct. These defenses are still limited to smaller models and specific attack types [6].

Ensemble methods: Using multiple models and aggregating their predictions to reduce the impact of attacks on any single model.

Adversarial attacks on LLMs

Large language models face their own form of adversarial attacks through carefully crafted prompts. ‘Jailbreaking’ prompts are designed to bypass safety filters and cause the model to generate harmful or prohibited content. These adversarial prompts exploit the model’s instruction-following patterns—just as pixel perturbations exploit image classifiers [5].

As AI systems become more autonomous through agentic architectures, adversarial attacks become more dangerous. An agent that can take actions in the real world—making purchases, sending emails, controlling devices—poses much higher stakes if manipulated by adversarial inputs than a model that only generates text.