I Used Codex for Our Annual SOC 2 Pentest

DocsBot SOC 2 Type II badge

Our annual penetration test was due for DocsBot. I shared the first results on X, but the full workflow deserves more than a post.

I wrote about getting SOC 2 Type II certified last year. Certification was the milestone. This is the less glamorous part that comes after it: doing the work again, improving it, and collecting evidence that the controls are still real.

The traditional path is to hire a vendor, give them a target, and wait for a black-box scan. That can be useful, especially when you need independent validation. It can also be expensive, shallow, and disconnected from the people who have to fix what it finds.

This year I tried a different starting point.

I used Codex to test the system from both sides:

  1. Source code first
  2. Runtime behavior second
  3. Auditor report last

That order matters.

The entire scan used around 6–7 million tokens, ran for 6–7 hours, found 37 issues ranked by severity, and produced a 50-page vulnerability assessment and penetration testing report mapped to our SOC 2 controls.

Then I used agents to help fix and independently verify all 37 findings.

When I was CTO at WPMU DEV, a project like this could have taken one or two months. This took hours of machine time plus my review.

Why start with the source code?

A black-box tester sees what an outsider can see. That is valuable, but it also means the tester has to discover the shape of the application from the outside.

Codex could begin with the map.

I used its deep security scan workflow to inspect the full application surface across our repositories. Different agents could take different slices:

  • API routes
  • Authentication boundaries
  • File uploads and downloads
  • Webhooks
  • Billing flows
  • Widget endpoints
  • Server-side request forgery surfaces
  • CORS and cookies
  • Secrets and permissions
  • Dependency risk
  • Tenant isolation

This was not just grepping the codebase for scary strings and dumping everything into a report. The agents could trace a route, see how it was called, inspect the authorization checks, follow the data into storage, and compare the implementation with the intended security model.

The main agent then validated findings one by one.

That context makes a big difference. A suspicious pattern is not automatically exploitable. A route that looks protected in one file may be exposed by middleware somewhere else. A scanner can flag the pattern. An agent with repository context can investigate the path.

Then test the real application

Source review still leaves an important question: does production behave the way the code says it should?

After the repository scans, I had Codex test staging and production non-destructively. It checked the actual endpoints using the code analysis to decide what to probe.

The questions were practical:

  • Can this route be reached without authentication?
  • Can one tenant access another tenant's object?
  • Do the live cookies and headers match the intended security model?
  • Do upload and download flows expose anything they should not?
  • Does runtime behavior match the assumptions we made while reading the source?

That last question is where shallow audits often fall apart.

Code review can miss deployment configuration. Runtime scanning can miss the path that would make a strange behavior exploitable. Connecting the two gives you a much more useful answer:

code → runtime behavior → exploitability → remediation → evidence

The economics were strange

The scan consumed enough tokens to sound ridiculous: roughly 6–7 million.

If I had paid directly for all of them, I estimated the model cost at around $200–$500. In practice, it used a large part of my weekly Codex allowance plus one of the limit resets I had banked.

That is still dramatically cheaper than a typical external pentest.

Cost is only part of the story, though. The bigger advantage was continuity. The same system that found an issue could explain the code path, propose a fix, retest the exploit, check for regressions, and turn the result into evidence for the audit.

The output was not a scanner export somebody would ignore in a spreadsheet. It was an engineering queue.

We fixed all 37 findings

Finding issues is the easy part. The value comes from closing them without creating new ones.

For each issue, or related group of issues, Codex helped run the same pipeline:

  1. Create a separate worktree
  2. Assign the fix to an agent in the correct repository
  3. Keep the patch scoped to the finding
  4. Hand it to a separate verifier agent
  5. Have that verifier review the patch and test the exploit path
  6. Iterate until it passed
  7. Run our normal regression tests before merge
  8. Finish with human review

The important rule was simple: the agent that wrote the fix should not be the only agent that approved it.

Many fixes went through three or four layers of review: implementation agent, verifier agent, security reasoning pass, regression tests, then my merge review.

Find the issue. Prove it. Patch it. Verify it independently. Test for regressions. Merge.

That loop is much more interesting to me than "AI found 37 vulnerabilities." Finding a pile of possible problems is easy. Turning them into verified fixes is the work.

What this does not prove

I would not tell every company to cancel its independent pentest vendor tomorrow.

An external tester brings independence, specialized experience, and credibility that your own agent run does not automatically provide. Your auditor, customer contracts, cyber-insurance policy, or compliance scope may specifically require a qualified third party. AI can also share the same blind spots across scanning, fixing, and reporting if you do not deliberately separate those roles.

For us, this was an affordable, repeatable security review and a strong evidence package for our annual SOC 2 work. It let us go much deeper before deciding where outside validation adds the most value.

The safest use of this workflow is not "the AI says we're secure."

It is:

  • Give the agent broad visibility into the authorized system
  • Keep runtime tests non-destructive and in scope
  • Separate implementation from verification
  • Require evidence for every finding
  • Keep a human responsible for the final decision
  • Bring in independent expertise where the risk or requirement calls for it

Coding agents are becoming security infrastructure

I used to think of coding agents mainly as a way to write features faster.

This project changed that.

A capable agent can become part of a repeatable security review system: map every surface, test each assumption, verify the findings, fix the code, retest the exploit, and produce an auditor-ready record of what happened.

It does not remove responsibility. If anything, it makes the responsibility more explicit because the human has to define authorization, scope, evidence, and the standard for "fixed."

But it changes what a small team can afford to examine.

Our first SOC 2 effort proved that a lean company could build a serious compliance program. This year's pentest made me realize the ongoing security work can become much more continuous too.

Not a point-in-time scan that goes stale in a folder. A loop we can run again.

That might be the biggest security benefit of coding agents: not one magical audit, but making careful review cheap enough to repeat.