[PATCH] fsmonitor: flush pending FSEvents before cookie wait

[PATCH] fsmonitor: flush pending FSEvents before cookie wait

[Tamir Duberstein]

56cef9cb1a (fsmonitor: use pthread_cond_timedwait for cookie wait,
2026-04-15) limits the cookie wait to one second so that a filesystem
which never delivers events cannot hang fsmonitor clients. A client that
times out receives a trivial response and scans the entire index.

FSEvents can defer delivery while it batches notifications and does not
guarantee that its queue is drained in one latency interval. A loaded
macOS system can therefore time out even though the event stream is
working.

On an Apple M4 Max (16 cores, 128 GiB RAM) running macOS 26.5.2, two
worktrees with a 1,001,178-entry index timed out 484 of 545 and 297 of
365 fsmonitor requests. One status call performed 934,519 lstat() calls
during a 47-second preload and took 52 seconds overall.

Ask FSEvents to flush pending notifications after creating the cookie
and before starting the timed wait. Use the asynchronous form because
the client handler holds main_lock, which the listener callback also
acquires. Keep the timeout and the behavior of the other backends
unchanged.

Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 builtin/fsmonitor--daemon.c          | 3 +++
 compat/fsmonitor/fsm-darwin-gcc.h    | 1 +
 compat/fsmonitor/fsm-listen-darwin.c | 5 +++++
 compat/fsmonitor/fsm-listen-linux.c  | 4 ++++
 compat/fsmonitor/fsm-listen-win32.c  | 4 ++++
 compat/fsmonitor/fsm-listen.h        | 6 ++++++
 6 files changed, 23 insertions(+)

diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index 4161dd8282..8e32b5ae5e 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -206,6 +206,9 @@ static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie(
 	close(fd);
 	unlink(cookie_pathname.buf);
 
+	/* The listener callback takes main_lock, so this must not block. */
+	fsm_listen__flush_async(state);
+
 	/*
 	 * Wait for the listener thread to observe the cookie file.
 	 * Time out after a short interval so that the client
diff --git a/compat/fsmonitor/fsm-darwin-gcc.h b/compat/fsmonitor/fsm-darwin-gcc.h
index 3496e29b3a..c209dc2f68 100644
--- a/compat/fsmonitor/fsm-darwin-gcc.h
+++ b/compat/fsmonitor/fsm-darwin-gcc.h
@@ -82,6 +82,7 @@ CFRunLoopRef CFRunLoopGetCurrent(void);
 extern CFStringRef kCFRunLoopDefaultMode;
 void FSEventStreamSetDispatchQueue(FSEventStreamRef stream, dispatch_queue_t q);
 unsigned char FSEventStreamStart(FSEventStreamRef stream);
+FSEventStreamEventId FSEventStreamFlushAsync(FSEventStreamRef stream);
 void FSEventStreamStop(FSEventStreamRef stream);
 void FSEventStreamInvalidate(FSEventStreamRef stream);
 void FSEventStreamRelease(FSEventStreamRef stream);
diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c
index 43c3a915a0..64bee248d2 100644
--- a/compat/fsmonitor/fsm-listen-darwin.c
+++ b/compat/fsmonitor/fsm-listen-darwin.c
@@ -496,6 +496,11 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
 	pthread_mutex_unlock(&data->dq_lock);
 }
 
+void fsm_listen__flush_async(struct fsmonitor_daemon_state *state)
+{
+	FSEventStreamFlushAsync(state->listen_data->stream);
+}
+
 void fsm_listen__loop(struct fsmonitor_daemon_state *state)
 {
 	struct fsm_listen_data *data;
diff --git a/compat/fsmonitor/fsm-listen-linux.c b/compat/fsmonitor/fsm-listen-linux.c
index e3dca14b62..7aae29ea22 100644
--- a/compat/fsmonitor/fsm-listen-linux.c
+++ b/compat/fsmonitor/fsm-listen-linux.c
@@ -493,6 +493,10 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
 		state->listen_data->shutdown = SHUTDOWN_STOP;
 }
 
+void fsm_listen__flush_async(struct fsmonitor_daemon_state *state UNUSED)
+{
+}
+
 /*
  * Process a single inotify event and queue for publication.
  */
diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c
index 9a6efc9bea..039d797000 100644
--- a/compat/fsmonitor/fsm-listen-win32.c
+++ b/compat/fsmonitor/fsm-listen-win32.c
@@ -290,6 +290,10 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
 	SetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]);
 }
 
+void fsm_listen__flush_async(struct fsmonitor_daemon_state *state UNUSED)
+{
+}
+
 static struct one_watch *create_watch(const char *path)
 {
 	struct one_watch *watch = NULL;
diff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h
index 41650bf897..cfeca1f4b6 100644
--- a/compat/fsmonitor/fsm-listen.h
+++ b/compat/fsmonitor/fsm-listen.h
@@ -38,6 +38,12 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state);
  */
 void fsm_listen__loop(struct fsmonitor_daemon_state *state);
 
+/*
+ * Prompt the listener to deliver queued filesystem events, if supported.
+ * This does not wait for the events to be processed.
+ */
+void fsm_listen__flush_async(struct fsmonitor_daemon_state *state);
+
 /*
  * Gently request that the fsmonitor listener thread shutdown.
  * It does not wait for it to stop.  The caller should do a JOIN

---
base-commit: 5d2e7709234afea1b6ddb25cd4f60d3d5fb3c200
change-id: 20260721-fsmonitor-darwin-cookie-flush-0f0d6e554a56

Re: [PATCH] fsmonitor: flush pending FSEvents before cookie wait

[Koji Nakamaru]

On Wed, Jul 22, 2026 at 6:05 AM Tamir Duberstein <tamird@gmail.com> wrote:
>
> 56cef9cb1a (fsmonitor: use pthread_cond_timedwait for cookie wait,
> 2026-04-15) limits the cookie wait to one second so that a filesystem
> which never delivers events cannot hang fsmonitor clients. A client that
> times out receives a trivial response and scans the entire index.
>
> FSEvents can defer delivery while it batches notifications and does not
> guarantee that its queue is drained in one latency interval. A loaded
> macOS system can therefore time out even though the event stream is
> working.
>
> On an Apple M4 Max (16 cores, 128 GiB RAM) running macOS 26.5.2, two
> worktrees with a 1,001,178-entry index timed out 484 of 545 and 297 of
> 365 fsmonitor requests. One status call performed 934,519 lstat() calls
> during a 47-second preload and took 52 seconds overall.
>
> Ask FSEvents to flush pending notifications after creating the cookie
> and before starting the timed wait. Use the asynchronous form because
> the client handler holds main_lock, which the listener callback also
> acquires. Keep the timeout and the behavior of the other backends
> unchanged.
>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---
>  builtin/fsmonitor--daemon.c          | 3 +++
>  compat/fsmonitor/fsm-darwin-gcc.h    | 1 +
>  compat/fsmonitor/fsm-listen-darwin.c | 5 +++++
>  compat/fsmonitor/fsm-listen-linux.c  | 4 ++++
>  compat/fsmonitor/fsm-listen-win32.c  | 4 ++++
>  compat/fsmonitor/fsm-listen.h        | 6 ++++++
>  6 files changed, 23 insertions(+)
>
> diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
> index 4161dd8282..8e32b5ae5e 100644
> --- a/builtin/fsmonitor--daemon.c
> +++ b/builtin/fsmonitor--daemon.c
> @@ -206,6 +206,9 @@ static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie(
>         close(fd);
>         unlink(cookie_pathname.buf);
>
> +       /* The listener callback takes main_lock, so this must not block. */
> +       fsm_listen__flush_async(state);
> +
>         /*
>          * Wait for the listener thread to observe the cookie file.
>          * Time out after a short interval so that the client
> diff --git a/compat/fsmonitor/fsm-darwin-gcc.h b/compat/fsmonitor/fsm-darwin-gcc.h
> index 3496e29b3a..c209dc2f68 100644
> --- a/compat/fsmonitor/fsm-darwin-gcc.h
> +++ b/compat/fsmonitor/fsm-darwin-gcc.h
> @@ -82,6 +82,7 @@ CFRunLoopRef CFRunLoopGetCurrent(void);
>  extern CFStringRef kCFRunLoopDefaultMode;
>  void FSEventStreamSetDispatchQueue(FSEventStreamRef stream, dispatch_queue_t q);
>  unsigned char FSEventStreamStart(FSEventStreamRef stream);
> +FSEventStreamEventId FSEventStreamFlushAsync(FSEventStreamRef stream);
>  void FSEventStreamStop(FSEventStreamRef stream);
>  void FSEventStreamInvalidate(FSEventStreamRef stream);
>  void FSEventStreamRelease(FSEventStreamRef stream);
> diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c
> index 43c3a915a0..64bee248d2 100644
> --- a/compat/fsmonitor/fsm-listen-darwin.c
> +++ b/compat/fsmonitor/fsm-listen-darwin.c
> @@ -496,6 +496,11 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
>         pthread_mutex_unlock(&data->dq_lock);
>  }
>
> +void fsm_listen__flush_async(struct fsmonitor_daemon_state *state)
> +{
> +       FSEventStreamFlushAsync(state->listen_data->stream);
> +}
> +
>  void fsm_listen__loop(struct fsmonitor_daemon_state *state)
>  {
>         struct fsm_listen_data *data;
> diff --git a/compat/fsmonitor/fsm-listen-linux.c b/compat/fsmonitor/fsm-listen-linux.c
> index e3dca14b62..7aae29ea22 100644
> --- a/compat/fsmonitor/fsm-listen-linux.c
> +++ b/compat/fsmonitor/fsm-listen-linux.c
> @@ -493,6 +493,10 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
>                 state->listen_data->shutdown = SHUTDOWN_STOP;
>  }
>
> +void fsm_listen__flush_async(struct fsmonitor_daemon_state *state UNUSED)
> +{
> +}
> +
>  /*
>   * Process a single inotify event and queue for publication.
>   */
> diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c
> index 9a6efc9bea..039d797000 100644
> --- a/compat/fsmonitor/fsm-listen-win32.c
> +++ b/compat/fsmonitor/fsm-listen-win32.c
> @@ -290,6 +290,10 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
>         SetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]);
>  }
>
> +void fsm_listen__flush_async(struct fsmonitor_daemon_state *state UNUSED)
> +{
> +}
> +
>  static struct one_watch *create_watch(const char *path)
>  {
>         struct one_watch *watch = NULL;
> diff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h
> index 41650bf897..cfeca1f4b6 100644
> --- a/compat/fsmonitor/fsm-listen.h
> +++ b/compat/fsmonitor/fsm-listen.h
> @@ -38,6 +38,12 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state);
>   */
>  void fsm_listen__loop(struct fsmonitor_daemon_state *state);
>
> +/*
> + * Prompt the listener to deliver queued filesystem events, if supported.
> + * This does not wait for the events to be processed.
> + */
> +void fsm_listen__flush_async(struct fsmonitor_daemon_state *state);
> +
>  /*
>   * Gently request that the fsmonitor listener thread shutdown.
>   * It does not wait for it to stop.  The caller should do a JOIN
>
> ---
> base-commit: 5d2e7709234afea1b6ddb25cd4f60d3d5fb3c200
> change-id: 20260721-fsmonitor-darwin-cookie-flush-0f0d6e554a56
>

This patch is carefully designed to minimize any risks. To drain events,
we could also call FSEventStreamFlushSync before acquiring main_lock in
do_handle_client(), but this patch should be sufficient if it mitigates
the issue. The commit message would be much more convincing if you also
included benchmark results showing how many timeouts were reduced.

--
Koji Nakamaru

Re: [PATCH] fsmonitor: flush pending FSEvents before cookie wait

[Junio C Hamano]

Koji Nakamaru <koji.nakamaru@gree.net> writes:

> On Wed, Jul 22, 2026 at 6:05 AM Tamir Duberstein <tamird@gmail.com> wrote:
>>
>> 56cef9cb1a (fsmonitor: use pthread_cond_timedwait for cookie wait,
>> 2026-04-15) limits the cookie wait to one second so that a filesystem
>> which never delivers events cannot hang fsmonitor clients. A client that
>> times out receives a trivial response and scans the entire index.
>>
>> FSEvents can defer delivery while it batches notifications and does not
>> guarantee that its queue is drained in one latency interval. A loaded
>> macOS system can therefore time out even though the event stream is
>> working.
>>
>> On an Apple M4 Max (16 cores, 128 GiB RAM) running macOS 26.5.2, two
>> worktrees with a 1,001,178-entry index timed out 484 of 545 and 297 of
>> 365 fsmonitor requests. One status call performed 934,519 lstat() calls
>> during a 47-second preload and took 52 seconds overall.
>>
>> Ask FSEvents to flush pending notifications after creating the cookie
>> and before starting the timed wait. Use the asynchronous form because
>> the client handler holds main_lock, which the listener callback also
>> acquires. Keep the timeout and the behavior of the other backends
>> unchanged.
>>
>> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
>> ---
>>...
> This patch is carefully designed to minimize any risks. To drain events,
> we could also call FSEventStreamFlushSync before acquiring main_lock in
> do_handle_client(), but this patch should be sufficient if it mitigates
> the issue. The commit message would be much more convincing if you also
> included benchmark results showing how many timeouts were reduced.

Thanks for a review.