Making an NHI Scanner Production-Ready: What a 600-Identity Run Taught the Tool

In a companion piece, I wrote about running my open-source tool nhi-scan against a real Microsoft Entra tenant with 614 non-human identities — and the lessons in the findings. This is the other half of that story: the tenant didn’t just produce findings, it exposed three places where the tool wasn’t ready for the real world. I fixed all three upstream. Here’s what broke and why it matters for anyone building security tooling.

1. It has to run where practitioners actually are

The tool was born on a POSIX machine. The tenant I scanned was administered from Windows — which is where a huge share of enterprise identity admins actually live. Out of the box, the tool failed on Windows for two boring-but-fatal reasons:

• CLI invocation. The collectors shell out to cloud CLIs (az, aws, gcloud). On Windows those are .cmd shims, which don’t launch the same way they do on Linux/macOS — so the very first collector died.
• The UTF-8 BOM. Windows PowerShell writes a byte-order mark at the front of files (and older PowerShell writes UTF-16 for redirected output). The tool’s readers weren’t expecting it, so perfectly valid inventory files failed to parse.

The fix was unglamorous and important: a single run_cli() helper that launches CLIs correctly across platforms, and reading everything as utf-8-sig (tolerating the BOM) throughout — files and piped stdin.

Lesson: a security tool that only runs on the author’s laptop isn’t a security tool, it’s a demo. “Cross-platform” isn’t a nice-to-have for anything meant to touch enterprise identity — it’s table stakes.

2. At fleet scale, N+1 API patterns are the enemy

The enriched scan — the one that pulls each identity’s granted permissions to reason about over-privilege — originally made per-identity calls to Microsoft Graph. On a ~600-service-principal tenant, that meant a long sequence of round-trips: the enriched gather took roughly 40 minutes.

Forty minutes is the difference between a tool people run and a tool people mean to run. The fix was to batch permission expansion through the Microsoft Graph $batch endpoint (up to 20 sub-requests per POST), with throttling-aware retries and pagination fallback. Same data, far fewer round-trips:

~40 minutes → ~2 minutes. About a 20× speedup.

Lesson: anything that inventories a fleet — and non-human identities are always a fleet — has to assume hundreds or thousands of objects and design for batching from the start. A correct tool that’s too slow to run regularly doesn’t get run regularly, which means the inventory goes stale, which defeats the point.

3. Context-aware classification (the false-positive fix)

This is the one that mattered most, and it came straight out of the findings.

The scan initially flagged 53 identities with long-lived secrets. On investigation, 52 were platform-managed identities whose “credentials” are cloud-issued certificates that rotate automatically — never stored, never touched by a human, structurally incapable of being the stale-secret risk the finding implied. Only one was a real application credential.

A tool that reports 53 criticals when there’s 1 is worse than no tool: it trains the security team to distrust the output. The fix was to make the risk engine credential-type aware — classify managed identities as managed so they no longer generate stored-secret or credential findings, while still surfacing the genuine ones.

Lesson: for non-human identity, the existence of a credential isn’t the risk — the kind of credential is. Managed identities, federated workload identities, and stored client secrets carry completely different risk. A risk model that flattens them into “has a credential” will cry wolf until nobody listens.

The meta-lesson: dogfood on something real

None of these three problems showed up in unit tests or on a handful of sample identities. They showed up the moment the tool met a real tenant with real scale, on a real admin’s real operating system. All three fixes are now merged upstream, with tests added to keep them fixed.

That’s the argument for building in the open and running your own tools against real environments: the environment tells you the truth about your tool. The findings made the tenant safer; running against the tenant made the tool better. Both halves came from the same afternoon.

---

nhi-scan is open source (MIT): github.com/rpmsft9/nhi-scan. Risk-tiers non-human and agent identities against the OWASP NHI Top 10, with drift detection for agent reach — now cross-platform, batched, and credential-type aware.