[LSF/MM TOPIC ATTEND] thawing the fsnotify subsyetm

[LSF/MM TOPIC ATTEND] thawing the fsnotify subsyetm

[Amir Goldstein]

My employer has a use case of watching file system changes
over millions of directories.

It so happens that the same product (cloud sync) also runs on
Windows and on MacOS. both OS have a scalable API to monitor
file systems changes over millions of directories.

Since the product requires being notified on filename events
(e.g. create/delete/rename), the only available option on Linux is
inotify, but inotify does not scale well to millions of directories
use case and not a the right tool for the job in general.

For that purpose, I implemented fanotify super block watch [1],
to be able to:
1. report filename events
2. watch over file system root (as opposed to mount point)
3. report event information using struct file_handle,
    instead of keeping open file descriptor per event

Jan Kara has reviewed the high level design and did not
dismiss it.

I was trying to get some cleanup patches through to Al,
but changes to fsnotify that are not bug fixes are not very
popular these days.

I would like to present the suggested solution and make a
case for an API that is a super set of inotify and fanotify,
that will allow users to slowly move to the latter and eventually
to phase out the former.

I would like to discuss with the stake holders how we can
thaw the fsnotify subsystem, to the point that any improvement
can be at least considered.

[1] https://lkml.org/lkml/2016/12/20/312

Re: [LSF/MM TOPIC ATTEND] thawing the fsnotify subsyetm

[Jan Kara]

On Thu 19-01-17 16:28:09, Amir Goldstein wrote:
> My employer has a use case of watching file system changes
> over millions of directories.
> 
> It so happens that the same product (cloud sync) also runs on
> Windows and on MacOS. both OS have a scalable API to monitor
> file systems changes over millions of directories.
> 
> Since the product requires being notified on filename events
> (e.g. create/delete/rename), the only available option on Linux is
> inotify, but inotify does not scale well to millions of directories
> use case and not a the right tool for the job in general.
> 
> For that purpose, I implemented fanotify super block watch [1],
> to be able to:
> 1. report filename events
> 2. watch over file system root (as opposed to mount point)
> 3. report event information using struct file_handle,
>     instead of keeping open file descriptor per event
 
Yeah, I think this would be worth a discussion. I've got other requests for
similar functionality so Amir is not the only one with the needs. But given
how unsuccessful we have been with file change notification API in the past
I'm very conservative about its extensions and would prefer to have wider
discussion about this.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR