Adversarial Machine Learning

Definition

A field studying attacks that manipulate ML model inputs or training data to cause misclassification, evade security detection systems, extract sensitive model information, or degrade model performance.

Technical Details

Adversarial ML attacks span several classes: evasion attacks (crafting inputs to fool a deployed model), poisoning attacks (corrupting training data), model extraction (reconstructing model weights through queries), and membership inference (determining if specific data was used in training). Defenses include adversarial training, input preprocessing, ensemble methods, and differential privacy in training.

Practical Usage

Malware authors craft adversarial samples that evade ML-based antivirus engines. Deepfake generators use adversarial techniques to produce realistic synthetic media. Security researchers use adversarial ML tools like CleverHans and ART to audit model robustness before deployment.

Examples

• Adding imperceptible pixel perturbations to malware binaries causes ML classifiers to label them as benign.
• Injecting adversarial traffic patterns causes an ML-based IDS to misclassify attack traffic as normal.
• A threat actor uses model extraction to replicate a competitor's fraud detection model and find its blind spots.

Related Terms