Red Teaming Large Language Models: A Study Of Jailbreaks, Prompt Injection and Data Leakage

Abstract

Large Language Models (LLMs) are now used in search engines, chat assistants, coding tools, document analysis systems, and autonomous agents. Their use creates new security and safety problems because natural language becomes both the input data and the instruction channel. An attacker can try to bypass safety rules, override system instructions, or extract hidden information from the model context. These risks are usually studied through jailbreak attacks, prompt injection, adversarial prompting, and data leakage. This thesis presents a theoretical study of red teaming for LLMs. It does not implement a complete practical framework. The goal is to review existing research, explain the main attack techniques, compare their assumptions, and discuss the limits of current evaluation methods. The study focuses on techniques such as Greedy Coordinate Gradient (GCG), Prompt Automatic Iterative Refinement (PAIR), AutoDAN, PromptInject, indirect prompt injection, JailbreakBench, and recent reinforcement learning based approaches such as PISmith. The thesis shows that LLM red teaming evolved from manual prompt attacks to automated and optimization-based methods. White-box attacks such as GCG can produce strong adversarial suffixes, but they require model access that is not always available. Black-box methods such as PAIR are more realistic for commercial APIs, but they depend on query budgets and judge quality. Prompt injection attacks create a separate risk because they tar-get the boundary between trusted instructions and untrusted content. Data leakage remains difficult to evaluate because success can be partial and context-dependent. The main finding is that current LLM red teaming still lacks stable evaluation standards. Attack success rates are hard to compare across papers because datasets, judges, threat models, and refusal criteria differ. Future work should focus on reproducible evaluation, clearer threat models, and safer automated testing. The thesis ends by suggesting two possible directions: using PAIR as a lightweight black-box optimization method, and designing a smaller adaptation of PISmith for prompt injection testing.