Continuous code health auditing at scale

Most engineering organizations know whether their production systems are healthy.

They track latency, errors, deployments, incidents, and test failures.

Ask whether the codebase itself is getting healthier or harder to change, and the answer gets fuzzy.

Where is complexity accumulating? Who owns it? Is it getting better or worse? How big would a cleanup be?

AI coding agents give us a new way to answer these questions. Aaron Francis has shown a great example with agents auditing code for drift, unnecessary abstractions, dead code, weak error handling, and naming problems.

But how much of a code health audit actually needs AI?

Our rule at Codemod is simple: if a question can be answered reliably with deterministic analysis, do that first.

We tested this on Backstage, a popular production-scale open-source codebase used by many enterprises. A TypeScript mining codemod, typescript-impossible-states-mining, scanned 839,957 LOC across 7,636 files in 5.33 seconds, finding 70 candidates across 51 files.

Here's what that makes possible.

Consider:

Loading code sample...

The type system allows contradictory states.

Compare that with:

Loading code sample...

You don't need a frontier model to spot this structural difference. An AST-based miner can find patterns like this across an entire codebase, quickly and consistently.

We built mining codemods for TypeScript and Python.

The TypeScript miner looks for three patterns:

• Boolean flag clusters that may represent one underlying state.
• Duplicated state, such as status: "loading" alongside isLoading: true.
• Nested linear lookups, such as .find() inside .map().

These aren't declarations that the code is bad. They're candidates for review.

A miner can establish that a pattern exists. Engineering judgment decides whether it's worth changing.

The Backstage results:

• 839,957 LOC
• 7,636 files
• 5.33s scan time
• 70 findings across 51 files
• 62 high-confidence collections findings
• 8 medium-confidence state_representation findings

That's more useful than a "code health score."

A score of 73/100 doesn't tell me what to do. Seventy concrete findings across 51 files gives me something I can inspect, scope, and potentially fix.

Confidence matters too. A .find() inside a .map() is straightforward to detect. A boolean cluster requires more interpretation.

The dashboard keeps those differences visible instead of hiding them behind one number.

Three hundred findings spread across 80 repositories is a different problem from 300 findings concentrated in one subsystem.

That's why we attach metadata to findings.

In Backstage, findings were concentrated in plugins including catalog-backend-module-gitlab and scaffolder.

Excluding the catchall @backstage/maintainers, ownership was concentrated under @backstage/tooling-maintainers and @backstage/framework-maintainers.

Now you can see the blast radius before writing the first migration PR.

Run the same miner every week and you can ask:

Are we introducing this pattern faster than we're removing it?

After a cleanup, the count should fall. If it climbs again, the problem is returning.

And because the detector is deterministic, the measurement stays stable between runs.

Some questions need context:

• Is this abstraction earning its keep?
• Does this module boundary make sense?
• Is this naming misleading?
• Has the architecture drifted?

AI is useful there.

But "is .find() being called inside .map()?" doesn't need an agent.

Neither does "does this type encode the same state with both strings and booleans?"

So split the audit.

Use parsers and static analysis to establish facts. Spend AI reasoning on intent, architecture, context, and tradeoffs.

There's another hypothesis worth testing.

When code represents state through several booleans, both humans and agents have to infer which combinations are valid.

A discriminated union puts those rules directly in the type system.

My hypothesis is that as agents make larger changes across unfamiliar codebases, explicit models will require less inference and make changes safer.

That's something worth measuring rather than assuming.

The useful model isn't deterministic analysis or AI.

It's both.

Deterministic miners tell you what can be measured precisely. AI handles questions that require understanding the surrounding system.

Insights then answers the questions tech leads actually care about:

Where is the problem?

Who owns it?

How big is the cleanup?

Is it getting better or worse?

That's more actionable than an audit report saying the codebase has "architectural inconsistencies."

Our rules are just examples.

You could mine deprecated internal APIs, error-handling patterns, old framework conventions, state modeling, or company-specific architectural rules.

If you can define the pattern deterministically, you can find it, measure it, track it, and scope the work required to change it.

You can build these mining codemods yourself. If you want help defining one for your codebase, reach out to the Codemod team.

The goal isn't to remove AI from code auditing.

It's to give AI fewer obvious things to figure out.