I scanned the 600 most-downloaded npm CLI packages for Windows bugs

Why do this at all

I maintain a small static checker called winbreak. It looks for the handful of patterns that make Node code work on macOS and fail on Windows: spawning a .cmd without a shell, spawning an extensionless node_modules/.bin shim, shelling out to a command Windows does not have, hardcoding /tmp.

A checker with no calibration is worthless. Anyone can write a regex that reports something in every file, and the resulting number tells you nothing except that the regex is loose. So the question I wanted answered was not "how much npm code is broken." It was "is my threshold sane?"

The method, so you can re-run it

I searched the npm registry for packages tagged cli, devtools, build-tool, scaffold, generator and process, which gave 1,417 unique names. I ranked those by real weekly download counts from the npm downloads API — not by the registry's own popularity score, which returned an identical value for every package and is useless for ranking — and took the top 600.

Each one was fetched as its published tarball, extracted, scanned including dist/, and deleted before the next batch. One fetch failed, leaving 599.

The harness is in the repository, so you can check this rather than believe it: github.com/Hackierz/winbreak/survey. python run_survey.py 600 and it does the whole thing. The exact ranked list of packages is checked in too — without it you would be scanning a different sample and could not reproduce anything, because download counts move. I have had to correct this write-up three times in public, and those corrections are only worth something if somebody else can reach the numbers independently.

The headline result

That 91.7% is the number I actually wanted. A tool that lit up on most of the ecosystem would have a broken threshold, and a tool that found nothing anywhere would be doing nothing. Mature, widely-used packages are mostly fine, which is what you would hope and is worth saying out loud.

This table is about JavaScript only, which is what I was scanning at the time. Counting the package.json findings from the last section as well, 459 of the 599 packages (76.6%) are clean on both axes.

Then I read all forty-eight

This is where a survey usually stops and publishes the percentage. It is also where it stops being honest, because a count of findings is not a count of bugs.

3 real   38 deliberately Linux-only   7 my checker being wrong

The correction, because it is the interesting part

My first pass through these findings put eleven in the “real” column. I published that. Then I went back to write each one up properly, which meant opening the actual source rather than the single line my own tool had shown me. Four of the eleven evaporated:

• create-storybook — the ps call is there, but one function up getProcessAncestry dispatches on os.platform() === "win32" and only reaches it on Unix.
• pi-subagents — the caller sets posixGroupOwned = process.platform !== "win32" and returns "unsupported-platform" before the call.
• metaharness — an if (platform === 'win32') { …powershell…; return } sits directly above the ps call.
• disk — the /tmp path is set as an environment variable for a remote process. It is not a path on this machine at all.

Three of those four hide the guard in a different function. That is cross-function analysis, and a tool that reads text one file at a time is not going to do it. I fixed the fourth — the block-form early return — and left the rest as a stated limitation rather than pretending otherwise.

The three that survived

agent-cli-detector, at 4.5M downloads a week, accounts for two. It calls execFileSync("ps", …) inside a try/catch that returns an empty string, and there is no mention of platform, win32, darwin or linux anywhere in the 326-line file. On Windows the call throws, the catch swallows it, and the entire process-tree detection strategy quietly finds nothing.

The third is pmx, part of the pm2 family, and it is the whole article in five lines:

childProcess.execFile(/^win/.test(process.platform) ? 'npm.cmd' : 'npm',
  ['ls', '--json', '--production'],
  { windowsHide: true, maxBuffer: 1024 * 1024 },
  function (error, stdout, stderr) {
    // if we can't spawn the npm binary, stop here
    if (error) return

Someone thought carefully about Windows here — there is a platform check, and windowsHide is a Windows-only option. But since Node 18.20.2 / 20.12.2 shipped the fix for CVE-2024-27980, handing a .cmd to execFile without shell: true throws EINVAL. The callback catches it and returns. So on Windows this quietly collects no dependency data, forever, and nothing anywhere logs a thing.

The thirty-eight that are not bugs

The largest group by far, and the most interesting one. This code is POSIX-only on purpose, and my checker has no way to know that.

There is no clever fix for this. execSync('rm -rf ...') in a script that only ever runs on a Linux build agent is indistinguishable, as text, from the same call in a cross-platform CLI. One is fine and one is a bug, and the difference is intent. A checker that reads text cannot see intent, and pretending otherwise is how you get a tool nobody trusts.

The seven that were my fault

The four from the correction above, plus firebase-tools, flagged for "/home/firebase/app/…" — a path inside a Docker image, not on the machine running the code — and generify, flagged for a /tmp path in its example.js, which is documentation.

That is a 15% false-positive rate on findings, and I would rather print it than have you discover it. Every checker has one. Most do not tell you what theirs is.

Seven bugs in my own checker

Getting from the first run to the numbers above meant fixing the checker seven times. This was the survey's actual yield.

1. whoami was on my list of commands Windows does not have. It ships with Windows, and has since Vista. Two packages were reported for perfectly good code. The general trap is worth naming: "this is a Unix command" is not the same as "Windows does not have it." Windows also has find, sort, more, where, tasklist and taskkill.
2. Guard detection did not understand a platform name. It looked for process.platform and isWindows. agent-browser stores the platform in a local variable and writes if (os === 'linux'), with an os === 'win32' branch immediately below. Correct code, reported as a bug.
3. And when I fixed that, it still failed. To stop braces inside strings from confusing the brace counter, the code stripped string literals before recording a block header — and then recorded the stripped version. So the guard arrived as if (os === '') with the platform name deleted. It now counts on the stripped line and keeps the original.
4. An early return is a guard. if (win32) return; followed by the POSIX call does not enclose anything, so walking the enclosing blocks never found it.
5. Handing a .cmd to cmd.exe was reported as a bug. That is the recommended fix. projen does exactly the right thing and got told off for it.
6. A platform branch that returns is a guard too, even spread over a block. The single-line if (win32) return; was handled; the same thing written as if (win32) { …; return } above the call was not, because that block does not enclose the call, it only leaves before it. metaharness was reported for exactly this.
7. rm -rf matched anywhere in 2,000 characters of extracted call text. sails-generate was flagged because a console.log("rm -rf node_modules && npm install") — advice printed for a human to read — fell inside a nearby call's window.

There was an eighth, of a different kind: findings on minified lines. Next.js ships cross-spawn compiled to a single line, and my checker reported it as a .cmd spawning bug. cross-spawn is the library that exists to fix .cmd spawning. A finding on line 1 of a 200KB line is useless even when it is correct, so those are suppressed now.

Then I checked the file I had been ignoring

Everything above is about JavaScript. Three real bugs in 599 packages is a reassuring number, and I was ready to publish it as the finding.

Then it occurred to me that the most common Windows portability bug in the Node ecosystem does not live in JavaScript at all. It lives in the scripts block of package.json. The evidence is cross-env: a package that exists for no purpose other than working around NODE_ENV=production node build.js not being valid on Windows, and which is downloaded 25 million times a week. Nobody installs that many copies of a workaround for a rare problem.

My checker read .js files. It had never opened a package.json. So I taught it three rules — inline environment variables, POSIX-only commands in command position, and $VAR shell expansion — and re-ran the same 600.

17.4% against 3.0%. The bug I could not see was roughly six times more common than the one I could, and it was sitting in a file I had walked past 599 times.

What it looks like in packages you already have installed

These are not obscure. Ordered by weekly downloads, the first few are:

The fix, for 142 of the 214

Naming a problem 214 times without offering the repair is half a tool, so the checker now applies it. winbreak --fix rewrites only the scripts block, and only the problems the ecosystem has already settled on an answer to:

- "clean": "rm -rf dist"
+ "clean": "rimraf dist"

- "build": "rm -rf dist && NODE_ENV=production rollup -c"
+ "build": "rimraf dist && cross-env NODE_ENV=production rollup -c"

- "assets": "cp -R ./src/assets ./lib/"
+ "assets": "shx cp -R ./src/assets ./lib/"

Run over every finding in this survey, that repairs 142 of the 214 (66%). The rest are $npm_package_config_*, docker invocations, || true, and shell programs.

What it refuses is the more important half. A for loop, $(...), ${VAR/a/b} or a pipe into sed is a shell program. There is no mechanical translation of one into something cmd.exe runs, and a wrong repair to a build script fails later, in a different place, in a file nobody re-reads. So it lists those by name with the reason and leaves them alone. It also rewrites only the changed values rather than reserializing your file, reparses before saving, and does not run npm install for you.

You can try it with nothing installed: paste a package.json into the browser checker and it hands back the repaired file to copy, along with what it would not touch.

Why this is less alarming than 17.4% sounds

This is the part it would be dishonest to leave out, and it is the reason I am not calling this "104 broken packages."

Almost none of these ever run on your machine. npm executes a package's own scripts for you only at install time — preinstall, install, postinstall, prepare. Everything else runs only for someone who has cloned the repository and is working on it. The most-flagged script names are clean (31), build (27) and test (20). Exactly two of the 214 are install-time scripts, and both are the || true idiom, whose only consequence on Windows is failing to swallow an error it was meant to swallow.

So the correct reading of 17.4% is not "one in six npm packages is broken on Windows." It is:

One in six of the most-downloaded npm CLI packages ships perfectly well to Windows users, and cannot be contributed to from a Windows machine without the contributor first repairing the build.

That is a real cost, but it falls on would-be contributors, silently, and it never arrives as a bug report — because the person who hit it was not a contributor yet, and gave up.

How confident am I in the 214

I read 28 of them in full: the top twelve by download count, and a random sample of sixteen from the middle of the distribution. All 28 were genuine. That is a far better hit rate than my JavaScript rules managed, and the reason is that an npm script gives you almost no context to misread. There is no enclosing if (process.platform === 'win32') to miss, because package.json has no if. Either the string runs in cmd.exe or it does not.

The command list is deliberately narrow for the same reason. echo, mkdir, find and test all exist in cmd.exe in some form, so none of them are flagged, even though a POSIX purist would point out that find is a completely different program there. Commands match only in command position, and cross-env, rimraf, shx, del and npm-run-all are allowlisted — reaching for the workaround is the fix, not the bug.

One caution I ran into while checking this, worth knowing if you try to reproduce it: on my Windows machine sed, grep and true all resolve perfectly happily from a shell — to C:\Program Files\Git\usr\bin. That directory is on Git Bash's PATH. It is on neither the machine PATH nor the user PATH, so it does not exist for cmd.exe, which is what npm uses to run scripts on Windows. I nearly talked myself out of an entire rule on the strength of a test run in the wrong shell.

I scanned tarballs. Contributors run the repository.

This survey reads each package's published tarball. But the person this bug actually hurts has cloned the repository, and those are not guaranteed to be the same file — tools like clean-publish can strip the scripts block on the way to npm. If that were common, I would be measuring the wrong artefact entirely.

So I checked twenty of them, comparing the scripts in the tarball against the scripts in the repository's package.json on its default branch:

Nobody stripped anything. The published scripts block is a fair proxy for what a contributor would find after cloning, so the 17.4% is measuring the thing it claims to. Twenty is a small check, but stripping is the kind of failure that would show up as an obvious zero, and there were none.

What this sample leaves out, and whether it matters

Not one of the 599 packages is a scoped @scope/name package. Zero, out of all 1,412 names the keyword search returned — so this is a property of how I gathered the list, not a filter I applied on purpose. Scoped packages are a large and growing part of npm, so that is a real limitation and you should know about it before trusting the headline.

So I measured it rather than leaving it as a caveat. I took 34 scoped CLI and build-tooling packages — @babel/cli, @typescript-eslint/parser, @swc/core, @playwright/test, @nestjs/cli and so on, matched to this survey's subject area — and ran the same scan:

The intervals overlap heavily, so there is no evidence that leaving scoped packages out moved the headline number. That is the useful conclusion, and it is about as much as 34 packages can support — the scoped interval is wide enough to drive a bus through, and the sample is hand-picked rather than random, because the registry's search endpoint rate-limited me and I stopped rather than hammer it.

One thing I am not claiming: the scoped sample also produced more raw JavaScript findings (5 of 34 against 18 of 599). I have not hand-read those the way I read all 48 above, and the whole lesson of this write-up is that a raw finding count is not a bug count — the 48 became three. So that number stays here as an observation and does not become a claim.

What I would take away from this

Most of what a text-based checker flags in mature code is deliberate. Four fifths of my findings were people knowingly writing Linux-only code in a Linux-only path. If you build something like this, the default assumption for a finding in a well-used package should be "this is intentional and I do not understand the context yet," not "I found a bug."

The survey audited the tool, not the ecosystem. I set out to measure npm and measured myself. Every one of those six bugs was invisible until real code walked into it, and none would have been caught by more unit tests, because I would have written the tests with the same wrong assumptions.

Check whether you are looking in the right file at all. I spent days tuning the precision of rules that read JavaScript, and the whole time the more common bug was one package.json away — in a file my directory walker was already opening and throwing away. No amount of refining an answer helps if the question has the wrong scope.

Calibrate against code you believe is correct. It is tempting to test a checker on things you suspect are broken, because finding something feels like success. The useful signal is the opposite: point it at code you are confident about and see what it says. Every improvement here came from a finding I did not want.