Your scanner's LLM triage isn't dumb, it's underfed

Point a static scanner at a bunch of MCP servers and you get a big pile of maybes. The problem isn't that it misses things. It's that a person still has to read every single flag to figure out which ones actually matter. If your triage step just forwards almost all of them to a human anyway, it hasn't really saved anyone any work.

Sift is a thin layer on top of SARIF. For each finding it asks a model one question: is this real, is it designed_behavior, or does it need a human? It also gives you a dry-run cost estimate and a default limit, so it can't quietly run up 500 API calls behind your back. That's the whole thing.

The run

I took four well-known MCP servers: server-memory, server-everything, server-sequential-thinking, and server-filesystem. That's 37 tools between them. I scanned all of them with Snyk's real snyk-agent-scan engine, which gave me nine findings. Then I ran those nine through sift twice on the same day. First on the raw SARIF, then again with each finding carrying the actual tool descriptions from the server it came from.

	designed_behavior	needs_human	real
bare SARIF	4	5	0
enriched	6	1	2

Both runs together cost about thirteen cents.

Here's what's going on. Snyk's W0xx codes are capability flags. They say "this tool can delete files," not "this tool did something bad." And Snyk flags at the server level, so the SARIF message is a generic template with none of the actual tool text in it. On the bare run, sift threw out the obviously fine capabilities (a filesystem server modifying files is kind of the point) and deferred everything else. Its reasoning kept saying the same thing: not enough evidence, a human should go read the real description. That's honestly the correct answer to the question it was given. It just wasn't a useful one.

The fix isn't clever at all. mcp-scan inspect already spits out signature.tools, so the descriptions were sitting right there in the scan output the whole time, one join away from the findings. So I attached them, bumped the message cap so the longer findings don't get cut off, and ran it again.

The one that mattered

server-everything ships a tool called get-env. Its description says it "returns all environment variables, helpful for debugging MCP server configuration." It takes no arguments, and the handler is literally one line: JSON.stringify(process.env, null, 2).

On the bare SARIF, Snyk's W016 (untrusted content retrieval) and W017 (sensitive data exposure) for that server both came back as needs_human at 55 to 60 percent confidence. Nothing in the finding said what the tools actually did, so there was nothing to judge.

Once the descriptions were attached, both flipped to real: secret and credential exposure. I didn't just trust the verdict, I went and read the source. No allowlist, no redaction, it dumps the entire process environment and hands it back. So this wasn't a hallucination. It was a correct catch that the bare pipeline could never have made, because the one fact that made it a real problem was never in the finding to begin with.

What this doesn't prove

Nine findings, four servers, and all of them are reputable. In an earlier round I used a dumb keyword filter instead of Snyk. It flagged 9 of 37 tools and sift correctly dismissed all 9. That's good specificity on real, benign tool text, but a sample with zero actual threats in it only tells you about true negatives. Two real verdicts is not a recall number, and I'm not going to pretend it is.

The idea also isn't new. Cisco's mcp-scanner already has an LLM judge for contextual triage. MCP-Guard ends in LLM arbitration. Datadog runs false-positive filtering on SAST in production. The whole industry is moving toward scanners cleaning up their own output, which leaves a standalone triage tool sitting on the side with less and less to do.

So the takeaway isn't the tool. It's this: "the LLM triage is miscalibrated, it defers too much" and "the LLM triage is being handed a keyword and told to guess the intent" look exactly the same from the outside, and only one of them gets fixed by rewriting the prompt. Before you go tuning the judge, check what you're actually feeding it. In my case it was a rule ID and a server name, and that was the whole problem.