Rendered at 15:15:45 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
lzhgusapp 19 hours ago [-]
Nice writeup. I work on native macOS utilities and have dealt with file monitoring quite a bit. kqueue is reliable but the per-file descriptor requirement can get tricky when you need to watch large directory trees.
For anyone looking at this space, FSEvents is the higher-level alternative Apple provides. It watches directory-level changes without needing a file descriptor per file, which scales better for broad monitoring. But kqueue gives you more granular control, like detecting attribute changes or renames specifically, which FSEvents sometimes lumps together.
In practice I've found a combination works well: FSEvents for broad directory watching to catch that something changed, then kqueue for targeted monitoring of specific files you care about.
saagarjha 14 hours ago [-]
NSFilePresenter is an even higher level API and the one I typically recommend to people since it’s very easy to use
hrmtst93837 7 hours ago [-]
NSFilePresenter is fine for the happy path.
Once other processes, external sync tools, or multi-user access get involved, it can miss events or bury you in delegate noise from Finder and coordination machinery, which is the part people tend to skip over. If you care about low-level correctness or perf, the API gets messy fast. It's easy right up until teh moment you need guarantees.
feznyng 15 hours ago [-]
Can you treat kqueue as infallible? I've found FSEvents sometimes drops events at high volume (unless I'm misunderstanding how to use it).
steve1977 10 hours ago [-]
That's the exciting thing about macOS development, right? There's always some funny riddles because Apple doesn't care to document stuff for 3rd parties (or at all).
There are benchmarks in there, and comparisons with all the other watchers out there I could find at the time I made it (see "Comparison with Similar Projects" at the very bottom of the readme).
It's a pretty long list. Tons of watchers out there with different design philosophies and shapes of problems they solve.
There are a lot of caveats to the filesystem monitoring APIs provided by kernels. Some projects (like facebook's watchman) take that as a kind of antagonism, and decide to fight back with layers and layers of fallbacks and distrust and rescans. That projects basically only makes sense as a daemon.
Other programs and libraries try to take that complexity and tame it by being super-focused on one platform or providing a lot of configuration options.
Some provide debouncing logic. This particular feature comes up from time to time, I believe, both for practical reasons and because over-reported events from the kernel subsystems, especially for some arcane events like moving a file across mount points can trigger a flurry of hard-to-associate events for the same path.
If you want to avoid dealing with under-documented filesystem event subsystems, you can also just make your own with ebpf. Especially for security-oriented systems, you'll find that the only (nearly) perfectly accurate filesystem event subsystem you can make, is the one you make from the ground up.
somat 20 hours ago [-]
I love small personal projects like this, especially their writeups. I always have a hard time learning unless I am actually using it for something.
For a while I've been annoyed that esbuild, which is written in Go, eschews these APIs to detect changes in watch mode and instead continually polls the filesystem: https://github.com/evanw/esbuild/issues/1527#issuecomment-90.... It actually consumes quite a bit of battery, so I might fork it and apply this post's implementation!
WhyNotHugo 19 hours ago [-]
kqueue is quite portable and works across all the BSDs.
The OpenBSD documentation for it is top notch, as usual. No idea about the rest (I suspect they’ve all converged at this point).
technothrasher 18 hours ago [-]
Kqueue originated in FreeBSD, and like most all of FreeBSD, it is very well documented.
jvican 9 hours ago [-]
I personally like and use watchexec [1] to run commands on repeat.
This looks interesting and I might give it a try. I did look at FSEvents in the past but due to my needs in macOS app, I took a very niche (and not so fast) approach of running find | ls -lT | shasum spawned in a separate process to reliably tell if a folder has changed.
chasil 14 hours ago [-]
In Linux, this is done with inotify.
Path units in systemd present inotify, but their use is somewhat constrained.
The incron utility is more flexible.
justanotherunit 19 hours ago [-]
Love it. Are there any benchmarks available on how fast this is from file change, kernel event received, event dispatched? I would expect it to be really quick.
lysace 17 hours ago [-]
Would be neat if POSIX was reasonably kept up to date. Just saying.
For anyone looking at this space, FSEvents is the higher-level alternative Apple provides. It watches directory-level changes without needing a file descriptor per file, which scales better for broad monitoring. But kqueue gives you more granular control, like detecting attribute changes or renames specifically, which FSEvents sometimes lumps together.
In practice I've found a combination works well: FSEvents for broad directory watching to catch that something changed, then kqueue for targeted monitoring of specific files you care about.
Once other processes, external sync tools, or multi-user access get involved, it can miss events or bury you in delegate noise from Finder and coordination machinery, which is the part people tend to skip over. If you care about low-level correctness or perf, the API gets messy fast. It's easy right up until teh moment you need guarantees.
There are benchmarks in there, and comparisons with all the other watchers out there I could find at the time I made it (see "Comparison with Similar Projects" at the very bottom of the readme).
It's a pretty long list. Tons of watchers out there with different design philosophies and shapes of problems they solve.
There are a lot of caveats to the filesystem monitoring APIs provided by kernels. Some projects (like facebook's watchman) take that as a kind of antagonism, and decide to fight back with layers and layers of fallbacks and distrust and rescans. That projects basically only makes sense as a daemon.
Other programs and libraries try to take that complexity and tame it by being super-focused on one platform or providing a lot of configuration options.
Some provide debouncing logic. This particular feature comes up from time to time, I believe, both for practical reasons and because over-reported events from the kernel subsystems, especially for some arcane events like moving a file across mount points can trigger a flurry of hard-to-associate events for the same path.
If you want to avoid dealing with under-documented filesystem event subsystems, you can also just make your own with ebpf. Especially for security-oriented systems, you'll find that the only (nearly) perfectly accurate filesystem event subsystem you can make, is the one you make from the ground up.
For more software in this domain see also the excellent entr https://eradman.com/entrproject/
For a while I've been annoyed that esbuild, which is written in Go, eschews these APIs to detect changes in watch mode and instead continually polls the filesystem: https://github.com/evanw/esbuild/issues/1527#issuecomment-90.... It actually consumes quite a bit of battery, so I might fork it and apply this post's implementation!
The OpenBSD documentation for it is top notch, as usual. No idea about the rest (I suspect they’ve all converged at this point).
Syntax example: `watchexec -r -e py uv run pytest -xvvs file.py`
[1]: https://github.com/watchexec/watchexec
Path units in systemd present inotify, but their use is somewhat constrained.
The incron utility is more flexible.