How to Conduct Advanced Security Audits with AI-Powered LLMs: A Step-by-Step Guide Based on Project Glasswing

Introduction

In the rapidly evolving landscape of cybersecurity, large language models (LLMs) are transforming how we find and fix vulnerabilities. Project Glasswing, a recent initiative involving Anthropic's Mythos Preview model, demonstrated that modern LLMs can do more than just identify bugs—they can chain multiple attack primitives into full exploit proofs, mimicking the work of a senior security researcher. This guide translates those insights into a practical, step-by-step approach for using advanced LLMs to audit your own codebases. Whether you're a security engineer or a curious developer, these steps will help you harness the power of AI to uncover hidden flaws and strengthen your systems.

What You Need

• Access to an advanced security-focused LLM – e.g., Anthropic's Mythos Preview or similar models capable of code analysis and reasoning.
• Your own source code repositories – at least one project with a decent codebase (the original test used over fifty repositories).
• A sandboxed execution environment – to safely compile and run proof-of-concept code without risking production systems.
• Basic familiarity with common vulnerability classes – use-after-free, buffer overflows, ROP chains, etc.
• Automation scripts or harnesses – to feed code snippets to the LLM and collect its responses efficiently.
• Time and patience – the iterative process may require multiple rounds of testing and refinement.

Step-by-Step Guide

Step 1: Select and Configure Your LLM

Start by choosing an LLM that specializes in code and security. Mythos Preview is a proven example, but other frontier models may also work. Ensure you have API access or a local instance. Configure the model to accept code snippets and return reasoning and executable code. For best results, set the temperature lower (e.g., 0.2) to encourage deterministic outputs. Prepare a system prompt that instructs the model to act as a security researcher looking for exploitable vulnerabilities.

Step 2: Feed Your Code Repositories to the Model

Do not dump entire repositories at once. Instead, break your code into logical units—functions, classes, or files—and send them one by one. Use an automated harness to iterate over your repositories. The original Project Glasswing sent over fifty repos to Mythos Preview. For each snippet, ask the model to identify potential vulnerabilities, including common bugs like use-after-free, integer overflows, or injection flaws. Record the model's initial findings as a list of suspected primitives.

Step 3: Analyze and Prioritize Bug Primitives

Review the model's output. Not every identified issue will be exploitable. Look for those that the model explicitly labels as 'primitives'—small atomic bugs (e.g., a read primitive, a write primitive) that can be combined. Prioritize bugs that appear in security-critical paths or that involve memory corruption. Mythos Preview excels at reasoning about how these primitives fit together, so pay extra attention to any chaining suggestions it offers.

Step 4: Instruct the Model to Build Exploit Chains

Once you have a set of primitives, ask the LLM to combine them into a coherent exploit chain. For example, prompt: 'Given a use-after-free vulnerability that gives an arbitrary read and a separate buffer overflow that gives arbitrary write, create a ROP chain to gain code execution.' The model should reason step-by-step, producing a sequence of operations. Mythos Preview demonstrated this ability to think like a senior researcher, linking several small bugs into a working proof. If the model fails to connect them, try rephrasing or providing additional context about the target system.

Step 5: Generate Proof-of-Concept Code

The real test is whether the exploit actually works. Instruct the LLM to write proof-of-concept code that triggers the vulnerability and executes the chain. The model should compile this code in a scratch environment (use your sandbox) and run it. If the expected behavior occurs—e.g., the program crashes in a controlled way or yields a shell—the proof is valid. If not, the model reads the error, adjusts its hypothesis, and tries again. This closed-loop iteration is crucial: a suspected flaw without a working proof remains speculation. In Project Glasswing, Mythos Preview independently looped through failure and refinement until it succeeded.

Step 6: Iterate and Refine

After generating proofs, go back to the model with the same or different repositories. Use the insights gained to improve your detection criteria. For instance, if the model consistently misses a certain class of bug, adjust the prompt or train it with examples. Also, compare results with other LLMs—the original test found that other frontier models identified similar bugs but struggled with chaining. Use your own judgment to validate all outputs; AI is a tool, not a replacement for human expertise. Repeat steps 2–5 for each new codebase update.

Tips for Success

• Start small – Begin with a single, well-understood repository to gauge the model's strengths and weaknesses.
• Automate the loop – Write scripts to send code, capture responses, and simulate the compile-and-run cycle. This makes the iterative process scalable.
• Use a dedicated sandbox – Always run generated exploit code in an isolated environment with no network access to prevent accidental damage.
• Validate everything – AI can hallucinate or produce code that looks correct but doesn't actually work. Manual review remains essential.
• Focus on chaining – The real power of advanced LLMs like Mythos Preview lies in connecting multiple bugs. Prioritize models that demonstrate strong reasoning ability.
• Document your findings – Keep a log of the bugs discovered, the chains built, and the proofs generated. This helps refine your future prompts and builds organizational knowledge.
• Stay within legal boundaries – Only test systems you own or have explicit permission to audit. Never deploy unverified exploits in production.

By following these steps, you can replicate the core methodology of Project Glasswing: using AI not just as a scanner, but as an autonomous security researcher that can find, chain, and prove vulnerabilities at scale. The technology is still evolving, but the approach outlined here gives you a head start in leveraging the next generation of LLMs for robust security auditing.