dnotify/inotify and vfs questions

dnotify/inotify and vfs questions

[Asser Femø]

[-- Attachment #1: Type: text/plain, Size: 1097 bytes --]

Hi,

I'm currently implementing change notification support for the linux
cifs client as part of Google's Summer of Code program.

In cifs, change notification works pretty much the same as dnotify does
in the kernel, and you cancel the notification by sending a NT_CANCEL
request. 

According to the fcntl manual you can cancel a notification by doing
fcntl(fd, F_NOTIFY, 0) (ie. sending 0 as the notification mask), but
looking in the kernel code fcntl_dirnotify() immediately calls
dnotify_flush() with neither telling the vfs module about it. Is there a
reason for this?  Otherwise I'd propose calling
filp->f_op->dir_notify(filp, 0) at some point in this scenario.

Regarding inotify, inotify_add_watch doesn't seem to pass on the request
either, which works fine for local filesystem operations as they call
fsnotify_* functions every time, but that isn't really feasible for
filesystems like cifs because we'd have to request change notification
on everything. Is there plans for implementing a mechanism to let vfs
modules get watch requests too?

cheers,
Asser


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: dnotify/inotify and vfs questions

[Jamie Lokier]

Asser Femø wrote:
> According to the fcntl manual you can cancel a notification by doing
> fcntl(fd, F_NOTIFY, 0) (ie. sending 0 as the notification mask), but
> looking in the kernel code fcntl_dirnotify() immediately calls
> dnotify_flush() with neither telling the vfs module about it. Is there a
> reason for this?  Otherwise I'd propose calling
> filp->f_op->dir_notify(filp, 0) at some point in this scenario.
> 
> Regarding inotify, inotify_add_watch doesn't seem to pass on the request
> either, which works fine for local filesystem operations as they call
> fsnotify_* functions every time, but that isn't really feasible for
> filesystems like cifs because we'd have to request change notification
> on everything. Is there plans for implementing a mechanism to let vfs
> modules get watch requests too?

On a related note:

dnotify and inotify on local filesystems appear to be synchronous, in
the following rather useful sense:

If you have previously registered for inotify/dnotify events that will
catch a change to a file, and called stat() on the file, then the
following operation:

    <receive some request>...
    stat_info = stat(file)

may be replaced in userspace code with:

    <receive some request>...
    if (any_dnotify_or_inotify_events_pending) {
        read_dnotify_or_inotify_events();
        if (any_events_related_to(file)) {
            store_in_userspace_stat_cache(file, stat(file));
        }
    }
    stat_info = lookup_userspace_stat_cache(file);

Now that's a silly way to save one system call in the fast path by itself.

But when the stat_info is a prerequisite for validating cached data --
such as the contents of a file parsed into a data structure -- it can
save a lot of system calls and logical work.

For example, an Apache-style path walk which checks for .htaccess, or
a Samba-style path walk which is checking for unsafe symbolic links,
can be reduced from say 20 system calls to zero using this method.

Pre-compiled or pre-parsed programs/scripts/templates/config-files
where all the source files used are prerequisites for invalidating a
cached compiled form, reduces from say 40 system calls to stat() all
the source files, to zero....  that's quite a saving.

It's not just reducing system calls.  The logical tests in userspace
are also skipped, if coded properly, facilitating very quick decisions
about things that depend on files which mostly don't change.
(Cascading structured cache prerequisites...mmm).

Remote dnotify/inotify doesn't _necessarily_ have this synchronous
property.  It may do in some cases, depending on the implementation
(this is subtle...).

So, it would be nice if there was a way to query this... rather than
the tedious method of testing the filesystem type and having a table
of "known local filesystem types" where it's safe to depend on this
property.  Alternatively, a way to specify at dnotify/inotify creation
type that synchronous notifications are required, and have the request
rejected if those can't be provided.

-- Jamie


-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dnotify/inotify and vfs questions

[Ian Campbell]

On Tue, 2005-08-23 at 16:23 +0100, Jamie Lokier wrote:
>     <receive some request>...
>     if (any_dnotify_or_inotify_events_pending) {
>         read_dnotify_or_inotify_events();
>         if (any_events_related_to(file)) {
>             store_in_userspace_stat_cache(file, stat(file));
>         }
>     }
>     stat_info = lookup_userspace_stat_cache(file);
> 
> Now that's a silly way to save one system call in the fast path by itself.

I'm not that familiar with inotify internals but doesn't
read_dnotify_or_inotify_events() or
any_dnotify_or_inotify_events_pending() involve a syscall?

Ian.
-- 
Ian Campbell
Current Noise: Primordial - The Coffin Ships

Today is the first day of the rest of the mess.

Re: dnotify/inotify and vfs questions

[Jamie Lokier]

Ian Campbell wrote:
> On Tue, 2005-08-23 at 16:23 +0100, Jamie Lokier wrote:
> >     <receive some request>...
> >     if (any_dnotify_or_inotify_events_pending) {
> >         read_dnotify_or_inotify_events();
> >         if (any_events_related_to(file)) {
> >             store_in_userspace_stat_cache(file, stat(file));
> >         }
> >     }
> >     stat_info = lookup_userspace_stat_cache(file);
> > 
> > Now that's a silly way to save one system call in the fast path by itself.
> 
> I'm not that familiar with inotify internals but doesn't
> read_dnotify_or_inotify_events() or
> any_dnotify_or_inotify_events_pending() involve a syscall?

The fast path is just any_dnotify_or_inotify_events_pending: there
aren't any relevant events pending in the fast path.

There's a few methods of doing this for free per individual stat cache check.

1. Signal handler.

    dnotify: Check a variable set by the signal handler.

         [ There's a small time window between dnotify sending the
         signal and the receiving thread noticing on an SMP system due
         to the IPI, during which the sending task might have another
         way to signal the recieving task that it's finished some
         operation, so this method of using dnotify to invalidate a
         stat cache only has the correct ordering properties on UP systems. ]

         This works because dnotify signals are thread-specific,
         so the checking thread will definitely have received the
         signal after the time another process modifies the file.
         
    inotify: Disappointingly inotify doesn't support SIGIO readiness :(

2. If you have to mix the test with a poll/select/epoll/rtsig fd waiting
   for some other purpose.  For example: a file/web/local server, where the
   constraint is only that each stat() to revalidate a cached response
   appears to happen any time after the beginning of receiving the
   network request is known.

    dnotify: It's free if you were using sigtimedwait anyway for I/O events,
             provided you completely read the queue, or get the signal
             priority right.

    inotify: It's free if you were using poll/select/epoll anyway for I/O
             events, provided in the case of epoll that you completely
             read the queue, or use a two-level queue.

3. Amortising the test over many stat cache checks.

   Even if you must use a system call to check for any pending events,
   for revalidating an object which depends on multiple files, only
   one call is needed for all of the stat cache checks.

   More generally (this is more flexible), you can separate the notion
   of "cache time checkpoint" from "cache validation".  It's enough to
   know that a stat result was valid any time between the checkpoint
   time, and the current time.  That's how I'm implementing the
   file/web/local server case described above in step 2.  Then the
   events only need to be checked once during that time interval, no
   matter how many complex objects are being revalidated.

   It gets slightly more efficient when you have multiple, overlapping
   checkpoint->validation time intervals due to multiple outstanding
   requests being processed concurrently.

As I explained in the previous mail, all this is absolutely pointless
to save one system call.  It's a lot of work for negligable gain.

The point is when it saves lots of calls and userspace logic together,
for things like web page templates and compiled programs, which depend
on many files which can be revalidated in a small number of operations.

-- Jamie

Re: dnotify/inotify and vfs questions

[Ian Campbell]

On Thu, 2005-08-25 at 15:57 +0100, Jamie Lokier wrote:
> Ian Campbell wrote:
> > On Tue, 2005-08-23 at 16:23 +0100, Jamie Lokier wrote:
> > >     <receive some request>...
> > >     if (any_dnotify_or_inotify_events_pending) {
> > >         read_dnotify_or_inotify_events();
> > >         if (any_events_related_to(file)) {
> > >             store_in_userspace_stat_cache(file, stat(file));
> > >         }
> > >     }
> > >     stat_info = lookup_userspace_stat_cache(file);
> > > 
> > > Now that's a silly way to save one system call in the fast path by itself.
> > 
> > I'm not that familiar with inotify internals but doesn't
> > read_dnotify_or_inotify_events() or
> > any_dnotify_or_inotify_events_pending() involve a syscall?
> 
> The fast path is just any_dnotify_or_inotify_events_pending: there
> aren't any relevant events pending in the fast path.
[snip]
> As I explained in the previous mail, all this is absolutely pointless
> to save one system call.  It's a lot of work for negligable gain.
> 
> The point is when it saves lots of calls and userspace logic together,
> for things like web page templates and compiled programs, which depend
> on many files which can be revalidated in a small number of operations.

Thanks for the explaination.

Ian.
-- 
Ian Campbell
Current Noise: Iron Maiden - Prodigal Son

Fay: The British police force used to be run by men of integrity.
Truscott: That is a mistake which has been rectified.
		-- Joe Orton, "Loot"