AI Control Hackathon - Sydney

This is the Sydney edition of the Apart AI Control Hackathon. We will run from Saturday morning to Monday afternoon local time. This event is supported by the Sydney AI Safety Space and the Sydney Knowledge Hub, the University of Sydney's startup incubator and research commercialisation hub. RSVP here and on the official page to be admissible. The AI Control Hackathon brings together researchers, engineers, security professionals, and AI safety enthusiasts to tackle one of the most pressing open problems: how do we keep AI systems safe when they might be actively working against us? This is the second edition of the hackathon, co-organized with Redwood Research (the group that founded the field of AI control). Over three days, participants will develop novel control protocols, build evaluation tools, and stress-test existing safety measures using infrastructure like ControlArena and SHADE-Arena. Top teams get ControlConf Tickets with full travel support, Fast-track to The Apart Fellowship, $2,000 in cash prizes, 🥇 1st Place $1,000 🥈 2nd Place $500 🥉 3rd Place $300 🏅 4th Place $100 🏅 5th Place $100. AI control is the subfield of AI safety focused on maintaining human oversight over AI systems, even when those systems might be misaligned or deliberately trying to subvert safety measures. Unlike alignment research, which tries to make models inherently safe, control research starts from a harder assumption: the model may already be working against you, and you need protocols that keep it contained anyway. AI systems are becoming autonomous agents that write code, manage infrastructure, and conduct research with decreasing human oversight. At the same time, frontier models are exhibiting concerning behaviors: alignment faking, scheming against shutdown, strategic underperformance on safety evaluations, and emergent misalignment from standard training. These are not hypothetical risks. They have been observed in current models, documented by multiple research groups, and appear to increase with scale. The infrastructure to do this work now exists, but defenses are not keeping up with attacks. Adaptive adversaries can reduce protocol safety from 99% to under 60%. Prompt injections bypass most monitoring at over 90% success rates. The field needs more researchers building stronger defenses. That is what this hackathon is for. Hackathon Tracks: 1. ControlArena Challenges: Work directly with ControlArena, the open-source library built by UK AISI and Redwood Research for running AI control experiments. The platform provides 14 evaluation settings, each pairing a legitimate task with a covert malicious objective, along with pre-built monitors, scorers, and analysis tools. Participants can create new evaluation environments, improve existing settings, develop novel monitors, or benchmark new approaches against standardized metrics. Current settings cover code sabotage, infrastructure compromise, training data poisoning, evaluation manipulation, financial system exploitation, and more. 2. Control Protocol Design: Design and implement novel control protocols that restrict AI systems from performing harmful actions while preserving useful capabilities. The best current protocols combine multiple techniques: trusted monitoring catches obvious attacks, resampling handles suspicious actions, and deferring on critical operations provides a safety net. This track covers runtime monitoring, factored cognition, adaptive deployment, chain-of-thought monitoring, scalable oversight mechanisms like debate, and shutdown and containment protocols for autonomous agents. 3. Red Teaming & Vulnerability Research: Identify and exploit weaknesses in existing control protocols and safety measures. Current protocols have known blind spots: adaptive attacks on monitors, feedback exploitation, steganographic collusion between agents, and strategic attack timing all represent real threats. This track covers adversarial inputs, attacks on monitoring systems, sandbagging strategies, jailbreaking techniques targeting reasoning models, and demonstrations of alignment faking or emergent misalignment in controlled settings. The goal is to push the frontier of what attackers can do, so defenders can build stronger systems. Who should participate? AI safety researchers working on alignment, interpretability, control, or governance, Machine learning engineers and researchers, Security researchers and red teamers, Software engineers with experience in agent systems or infrastructure, Policy researchers interested in technical safety measures, Students and early-career researchers exploring AI safety. No prior AI safety experience is required. The Resources tab has a curated reading list organized by track. Teams typically form at the start of the event. What you will do: Over three days, you will: Form teams and choose a challenge track, Research and scope a specific problem in AI control using the provided resources, Build a project developing tools, protocols, evaluations, or attack demonstrations, Submit a research report (PDF) documenting your approach, results, and implications, Present your work to judges from leading AI safety organizations. After the hackathon, all submitted projects will be reviewed by expert judges. Top projects receive prizes and are featured on the Apart Research website. Outstanding work may be invited for further development, publication, or presentation at upcoming events. ControlConf Berkeley 2026 (April 18-19) takes place shortly after this hackathon, making this an ideal time to develop and showcase new control research. Apart Research has organized 55+ research sprints with 6,000+ participants across 200+ global locations. Our hackathons produce real research output: published papers, new research collaborations, and contributions to open-source safety tools used across the field.