Teams now have two ways to find problems in a codebase: deterministic static analysis, and semantic analysis that can judge what the code means. They are often treated as competing approaches. The useful question is when each one helps.
We ran both against codemod-app, our own product codebase, as evidence for that question. One side used deterministic AST analysis (JSSG, our JS ast-grep engine). The other used semantic judgments powered by Jev, the TypeSafe primitive behind Codemod Smart Insights.
I expected the semantic approach to find roughly the same problems as the AST miner, just with less noise. It didn't.
Same brief, same commit
This started as a practical exercise, not a benchmark. We wanted to find performance problems worth fixing, and running the same review two ways was a way to see what each approach actually catches.
Same codebase. Same commit. Same performance brief: N+1 I/O, independent async waterfalls, unbounded concurrency, expensive work inside loops, and missing cleanup. We audited every medium- and high-confidence hit by hand, on our own code, because the findings had to be useful enough to change production. We are already acting on them.
JSSG required agent-assisted authoring and several precision passes against false-positive audits. Jev consumed the same brief as natural-language checks. After the shared research, the Jev side was close to zero extra authoring cost.
Fifteen percent overlap, and the misses were different bugs
After that audit we had 20 files with performance problems worth changing. Only three were found by both approaches. That 15% overlap changed how I think about this class of tooling. These were not two filters over the same list. They were missing different bugs.
The clearest case was campaign creation. A path downloaded and extracted the same package archive once for every repository in a campaign. Jev caught it. The AST miner saw the loop, but ranked the problem weakly. The expensive part wasn't obvious from local syntax; you had to understand what the operation meant.
The reverse happened in our sign-in auth hook. Two independent GitHub REST calls were awaited sequentially. JSSG caught the waterfall. Jev inspected the same file, focused on an N+1, and walked past the waterfall a few lines away.
Task sync was one of the three they shared: per-task Jira and Linear HTTP, including a states fetch that didn't depend on the loop. Neither approach was simply smarter. They were failing for different reasons.
Candidate volume tells a different story, and the units aren't comparable. JSSG produced a broad set of line-level candidates and a higher false-positive rate. Jev returned a short file list. Jev's short list is denser; JSSG's net is broader.
An interpretation problem, and an attention problem
AST miners have an interpretation problem. Semantic checks have an attention problem.
Once you see that distinction, asking which approach is "better" stops being useful.
Deterministic analysis is broad and exhaustive, and it struggles with meaning. An AST is good at telling you what the code is doing structurally. It doesn't automatically know what your domain abstractions mean.
Our domain layer puts db as a second argument and names functions for business intent (incrementPackageExecutionCount, hasEmailBeenSent). A callee-name I/O heuristic cannot see through that. Lazy SDK properties (await issue.state) are not call expressions at all. Domain verbs the miner has never heard of (generateObject as LLM I/O) score low while readFile in a build script scores high.
The noise clustered into learnable classes: notify-after-write sequences misread as waterfalls, pagination and retry loops, build scripts, among others. Once you see the class, you can usually encode an exclusion and remove it everywhere. Somebody still has to do that work.
Semantic analysis understands meaning better, and it may not inspect every relevant site. When Jev looked at the right code, its judgment was usually useful. The noisy cases were mostly scope: offline repo tooling treated as live load, a page-lifetime singleton flagged as a leak.
Sometimes a detector didn't run. Sometimes a file with textbook per-item I/O never opened. Sometimes Jev opened the right file, found one meaningful problem, and effectively stopped paying attention — exactly what happened with the auth hook.
JSSG struggles to decide whether a suspicious site is actually a problem. Jev struggles to decide which sites deserve its attention.
Discover, codify, enforce
That suggests a lifecycle, and it is not strictly one-way. Semantic discovery, then deterministic coverage, then semantic judgment again where context remains: cardinality, domain meaning, whether a suspicious site is actually a problem.
Discover
Use semantic analysis while the requirement still sounds like something you'd tell a senior engineer. "We've had a few incidents caused by unbounded concurrency. Find other places where we're taking that risk." At that point you have a problem description, not a mature static-analysis rule. Some cases are easy AST patterns. Others sit behind internal abstractions, or they're safe only because the collection has a known max size.
Encoding all of that up front is possible, and you may encode the wrong assumptions before you've seen enough real examples. Semantic checks let you start with the brief and learn what the bug looks like in your codebase.
Codify
Once the same shape keeps surviving review, the economics change. You've seen the bad pattern across several services. You know the syntax and the false-positive classes. Engineers agree it shouldn't be introduced again.
Asking a model to rediscover that judgment forever stops making sense. Write the deterministic rule. You get cheap, repeatable coverage, behavior doesn't change between runs, and when it gets something wrong you can inspect exactly why.
A first pass will still be wrong in recognizable ways. Until the miner has survived real triage, treat it as a candidate generator, not a merge gate. And static analysis can tell you a loop performs I/O without telling you whether that loop runs over 3 items or 300,000. Sometimes the next input is one number from production, not another heuristic.
Enforce
Once the deterministic rule is boring and predictable, run it continuously, or gate on it. Where a candidate still needs context, apply semantic judgment before the finding reaches an engineer.
What changes at enterprise scale?
On a mid-sized product codebase like ours, either path keeps up. Both were fast enough here to drive commit-over-commit time series for the same five checks.
At monorepo scale the economics split. Deterministic miners stay cheap to re-run on every commit across huge trees. Semantic checks get expensive as file count and history length grow. Use Jev to discover and prioritize. Lean on JSSG once the chart has to update continuously.
The infrastructure should let you move between them
We started this experiment expecting to compare two ways of finding the same performance problems. That's not what happened. Only three of the 20 actionable files were shared. Semantic analysis and deterministic analysis are useful at different stages of understanding a problem.
Codebase intelligence shouldn't force a team to choose between exhaustive static analysis and contextual semantic reasoning. The infrastructure should let them move between the two as an insight matures: discover with semantic checks, codify what repeats, enforce what is predictable, and keep semantic judgment where context still matters.
That's the workflow we're building into Codemod Smart Insights. If you want to try it against your own codebase, build a Smart Insight or talk to us about running it across an enterprise codebase.