Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition

Official repository of the 2024 SaTML LLM Capture-the-Flag Competition led by Edoardo Debenedetti, Javier Rando and Daniel Paleka.

Competition report: https://arxiv.org/abs/2406.07954

Dataset: https://huggingface.co/datasets/ethz-spylab/ctf-satml24

Blogpost: https://spylab.ai/blog/results-competition/

---

Loading the dataset form HuggingFace

from datasets import load_dataset

defenses = load_dataset("ethz-spylab/ctf-satml24", "defense")["valid"]

teams = load_dataset("ethz-spylab/ctf-satml24", "teams")["defense_teams"]

chats = load_dataset("ethz-spylab/ctf-satml24", "interaction_chats")["attack"]

---

Analyzing the data

We provide a script chat_diversity.py to reproduce the basic analysis that we include in our official report. The folder raw_data_manipulation includes transformations we performed on the raw data we collected from the competition. Please, reach to us if you think you need the original raw data.