CI on Linux, macOS and Windows for a Bun project
For its first two days, Aletheia’s tests ran on Linux and macOS. A claim of “works on Windows” needs more than hope, so I added a Windows runner. It was red for three runs before it went green. This post lists what broke and what fixed it, in the order it happened.
What broke
| Symptom | Cause | Fix |
|---|---|---|
Tests failed removing temporary folders with EBUSY |
Windows keeps a SQLite file open until the handle is released | One removeDir helper, used in every test that cleans a temporary folder. It retries on Windows only, three times, and then gives up quietly |
| A TOML config could not be uninstalled | A Windows path contains backslashes, which TOML escapes, and the uninstall code compared the escaped text with the raw path | Unescape the TOML string before comparing |
Tests that run hook commands through sh -c failed |
sh -c mangles Windows paths and command lines |
Skip those tests on Windows and say so in the README |
| Python helper tests failed | The runner’s python3 was a shim |
Prefer python, and set PYTHONUTF8 |
| Three installer tests timed out at 5 seconds | A slow Windows runner | A 20-second default test timeout in bunfig.toml |
| A crash test sometimes saw exit code 1 from the writer | The writer sometimes failed to start on a loaded runner (seen once) | Retry the start up to three times and print stderr. This fix came later, on the main branch |
The honest part
The runners are ubuntu-latest, macos-latest and windows-latest, with Bun 1.3.10. Several test groups are skipped on Windows, mostly those that run hooks through sh -c, and no real AI coding tool was tried there. The README says “CI tested” for Windows, not “works”, and lists the main skipped checks. A green badge that hides skipped tests is worse than a red one.
What I would do differently
I would have added the Windows runner on day one. Every problem above is small. Together they cost about an hour of CI rounds on one afternoon, because they surfaced one at a time.
Sources
- Aletheia repository, Windows test pull request, fetched 2026-10-31
- Aletheia CI workflow, fetched 2026-10-31
