which ioctls matter across filesystems

which ioctls matter across filesystems

[Steve French]

Other than the obvious example of
    EXT2_IOC_GETFLAGS
and
    EXT2_IOC_SETFLAGS
which are implemented by multiple filesystems (and are necessary to 
support a few commonly used tools), are there any other ioctls which 
should be able to be sent remotely (optionally)?   For it to be worth 
extending the network protocol (in my case CIFS to servers such as 
Samba, but presumably cluster filesystems have similar interests in 
supporting all key local tools across the network), an ioctl would have 
to be
    - used by more than one local filesystem
    - not have an equivalent way to do the same thing without an ioctl

I have added the GET/SETFLAGS client support, but am not aware of any 
others which would need to be remoted.   For fcntl there are more, but 
it requires more research to figure out how to handle setlease/getlease 
and a few others with network implications without degrading 
performance.   Although I am not a fan of ioctls and fcntls, there are a 
few that are necessary to achieve 100% local semantics across the network.

The new inotify mechanism being prototyped in -mm currently is the other 
one which needs work to determine how to map it across the network.   
Since it was added for support of Samba, the corresponding client part 
(for cifs) may turn out to map to the network protocol quite well 
already, and given NFSv4 having various similarities to CIFS, it would 
be interesting if the semantics of inotify would map to NFSv4 write 
protocol.

Re: which ioctls matter across filesystems

[Trond Myklebust]

fr den 29.04.2005 Klokka 14:22 (-0500) skreiv Steve French:

> The new inotify mechanism being prototyped in -mm currently is the other 
> one which needs work to determine how to map it across the network.   
> Since it was added for support of Samba, the corresponding client part 
> (for cifs) may turn out to map to the network protocol quite well 
> already, and given NFSv4 having various similarities to CIFS, it would 
> be interesting if the semantics of inotify would map to NFSv4 write 
> protocol.

We are discussing the equivalent of dnotify as a potential candidate for
the first minor version of NFSv4, but not inotify.
The purpose of our dnotify implementation is address the needs of things
like file browsers that don't really care about synchronous notification
of changes, but that do currently cause a lot of unnecessary traffic on
the wire due to constantly polling stat() and doing readdir() updates.
The jury is still out as to whether or not the callbacks actually do
reduce on-the-wire traffic, though, so we may drop it.

What kind of real-world applications exist out there that need inotify
functionality, and what sort of requirements do they have (in particular
w.r.t. the notification mechanism)?

Cheers,
  Trond
-- 
Trond Myklebust <trond.myklebust@fys.uio.no>

Re: which ioctls matter across filesystems

[Robert Love]

On Fri, 2005-04-29 at 15:53 -0400, Trond Myklebust wrote:

> We are discussing the equivalent of dnotify as a potential candidate for
> the first minor version of NFSv4, but not inotify.
> The purpose of our dnotify implementation is address the needs of things
> like file browsers that don't really care about synchronous notification
> of changes, but that do currently cause a lot of unnecessary traffic on
> the wire due to constantly polling stat() and doing readdir() updates.
> The jury is still out as to whether or not the callbacks actually do
> reduce on-the-wire traffic, though, so we may drop it.

What about inotify makes it insufficient for your needs?

> What kind of real-world applications exist out there that need inotify
> functionality, and what sort of requirements do they have (in particular
> w.r.t. the notification mechanism)?

A few worksets:

	- Current users, such as FAM and Samba, that need simple file
	  change notification
	- Random applications that want to watch a file or two
	- The Linux desktop
	- Real-time live-updating indexing systems, such as Beagle,
	  that compete with f.e. Apple's Spotlight.

Best,

	Robert Love

Re: which ioctls matter across filesystems

[Trond Myklebust]

fr den 29.04.2005 Klokka 16:03 (-0400) skreiv Robert Love:
> On Fri, 2005-04-29 at 15:53 -0400, Trond Myklebust wrote:
> 
> > We are discussing the equivalent of dnotify as a potential candidate for
> > the first minor version of NFSv4, but not inotify.
> > The purpose of our dnotify implementation is address the needs of things
> > like file browsers that don't really care about synchronous notification
> > of changes, but that do currently cause a lot of unnecessary traffic on
> > the wire due to constantly polling stat() and doing readdir() updates.
> > The jury is still out as to whether or not the callbacks actually do
> > reduce on-the-wire traffic, though, so we may drop it.
> 
> What about inotify makes it insufficient for your needs?

Nothing, if you are just tracking changes that are made by local
processes.

For remote filesystems like CIFS and NFS, however, there may be
applications that want to monitor changes that are made directly on the
server by other clients. In the above paragraph, I'm therefore referring
to a debate about whether or not adding officially sanctioned NFSv4
protocol support for something like dnotify/inotify makes sense.

The problem is that having the server call back a bunch of clients every
time a file changes does not really scale too well. The current
dnotify-like proposal therefore specifies that notification is not
synchronous (i.e. there may be a delay of several seconds), and that the
server may want to group several notifications into a single callback.
(if interested, you can view the current proposal on
http://www.ietf.org/internet-drafts/draft-ietf-nfsv4-directory-delegation-01.txt)


> > What kind of real-world applications exist out there that need inotify
> > functionality, and what sort of requirements do they have (in particular
> > w.r.t. the notification mechanism)?
> 
> A few worksets:
> 
> 	- Current users, such as FAM and Samba, that need simple file
> 	  change notification
> 	- Random applications that want to watch a file or two
> 	- The Linux desktop
> 	- Real-time live-updating indexing systems, such as Beagle,
> 	  that compete with f.e. Apple's Spotlight.

Right, but I assume that most of the use-cases (with possible exception
of FAM) do not really involve tracking changes made by other NFS
clients. Although I imagine in particular that the real-time indexing
system workload trying to run on several clients and tracking the same
files at the same time might be a lot of fun...

I'm therefore asking whether or not there is a killer-app out there that
we need to support and that does want to track changes made by other
clients. File browsers are more or less the only thing that come to
mind.

Cheers,
  Trond
-- 
Trond Myklebust <trond.myklebust@fys.uio.no>

Re: which ioctls matter across filesystems

[Robert Love]

On Fri, 2005-04-29 at 16:42 -0400, Trond Myklebust wrote:

> The problem is that having the server call back a bunch of clients every
> time a file changes does not really scale too well. The current
> dnotify-like proposal therefore specifies that notification is not
> synchronous (i.e. there may be a delay of several seconds), and that the
> server may want to group several notifications into a single callback.

Yah, so what I am asking is why not use inotify for the user-side
component of this system?

Wouldn't the deferring and coalescing of events occur on the server
side?  So the server-side stuff would be whatever you need--your own
code using whatever protocol you wanted--but the client-side interface
would be over inotify.

Even if not, I'd be willing to make changes to inotify to accommodate
NFS's needs.

	Robert Love

Re: which ioctls matter across filesystems

[Trond Myklebust]

fr den 29.04.2005 Klokka 16:47 (-0400) skreiv Robert Love:
> On Fri, 2005-04-29 at 16:42 -0400, Trond Myklebust wrote:
> 
> > The problem is that having the server call back a bunch of clients every
> > time a file changes does not really scale too well. The current
> > dnotify-like proposal therefore specifies that notification is not
> > synchronous (i.e. there may be a delay of several seconds), and that the
> > server may want to group several notifications into a single callback.
> 
> Yah, so what I am asking is why not use inotify for the user-side
> component of this system?

> Wouldn't the deferring and coalescing of events occur on the server
> side?  So the server-side stuff would be whatever you need--your own
> code using whatever protocol you wanted--but the client-side interface
> would be over inotify.

Sure. We're not talking about inventing new user interfaces here. Just
how best to support the existing ones.

> Even if not, I'd be willing to make changes to inotify to accommodate
> NFS's needs.

I think what the IETF NFS working group rather needs right now is an
advocate that is willing to stand up and demonstrate why protocol
support for inotify-style callbacks would be a more scalable solution
than a solution based on a combination of GETATTR polling and read
delegations (essentially the same thing as CIFS' op-locks) for
directories.

The current research (see
http://www3.ietf.org/proceedings/05mar/slides/nfsv4-4/sld1.htm) which
has uses real-life on-the-wire traffic actually leans more towards the
GETATTR solution. That research was based on a set of anonymous tcpdump
traces taken at Harvard University, though, so it reflects the traffic
in a typical university environment. It may be that other use-cases
exist that favour the inotify callbacks case.
If so, now is the time to step forward and say so...

Cheers,
   Trond
-- 
Trond Myklebust <trond.myklebust@fys.uio.no>

Re: which ioctls matter across filesystems

[Steve French]

Trond Myklebust wrote:

>I think what the IETF NFS working group rather needs right now is an
>advocate that is willing to stand up and demonstrate why protocol
>support for inotify-style callbacks would be a more scalable solution
>than a solution based on a combination of GETATTR polling and read
>delegations (essentially the same thing as CIFS' op-locks) for
>directories.
>
>  
>
I agree.

>The current research (see
>http://www3.ietf.org/proceedings/05mar/slides/nfsv4-4/sld1.htm) which
>has uses real-life on-the-wire traffic actually leans more towards the
>GETATTR solution. That research was based on a set of anonymous tcpdump
>traces taken at Harvard University, though, so it reflects the traffic
>in a typical university environment. It may be that other use-cases
>exist that favour the inotify callbacks case.
>  
>
Very interesting, I had not seen that.   FYI - There are many years of 
real world experience on the current transact2 notify (it is deployed in 
some form on most clients) but I don't know whether one of the NAS 
storage companies or researchers has done a good research paper on this 
topic - although there is no lack of customer traces in SPEC and SNIA.

My gut reaction is that as
1) directory size increases (number of files per directory)
and
2) change rate goes down

(both of which could be client heuristics) the notify mechanism (on the 
directory, or parent directory) is much better, but with small 
directories and more frequent changes the getattr (Transact2 
QueryPathinfo) approach wins.   There is no one-size-fits-all that 
covers both cases.

Re: which ioctls matter across filesystems

[Trond Myklebust]

fr den 29.04.2005 Klokka 16:22 (-0500) skreiv Steve French:

> Very interesting, I had not seen that.   FYI - There are many years of 
> real world experience on the current transact2 notify (it is deployed in 
> some form on most clients) but I don't know whether one of the NAS 
> storage companies or researchers has done a good research paper on this 
> topic - although there is no lack of customer traces in SPEC and SNIA.

If you are able to release those traces to them, then CITI does have a
simulator that can be used to test the various models. It is currently
only capable of analysing NFSv3 and v4 traffic, but it might perhaps be
possible to add a CIFS interpreter onto it.

Cheers,
  Trond
-- 
Trond Myklebust <trond.myklebust@fys.uio.no>

Re: which ioctls matter across filesystems

[Steve French]

Trond Myklebust wrote:

>I'm therefore asking whether or not there is a killer-app out there that
>we need to support and that does want to track changes made by other
>clients. File browsers are more or less the only thing that come to
>mind.
>
>Cheers,
>  Trond
>  
>
I believe that the spotlight facility of MacOS, and the somewhat similar 
Longhorn feature (think Google desktop search/indexing on steroids) 
qualify as killer-apps.   I am concerned about how to do better with our 
implementations across a distributed (NFS, CIFS etc.) network.   And of 
course coalescing async notifications most efficiently is a fascinating 
and difficult area to do right - for servers at least.

Re: which ioctls matter across filesystems

[Robert Love]

On Fri, 2005-04-29 at 15:55 -0500, Steve French wrote:

> I believe that the spotlight facility of MacOS, and the somewhat similar 
> Longhorn feature (think Google desktop search/indexing on steroids) 
> qualify as killer-apps.   I am concerned about how to do better with our 
> implementations across a distributed (NFS, CIFS etc.) network.   And of 
> course coalescing async notifications most efficiently is a fascinating 
> and difficult area to do right - for servers at least.

If we had some way to efficiently coalesce events, even non-remote stuff
would drool.  Beagle (our Spotlight killer) would love it.

First thing is, the events cannot be stored in a linked list. ;-)

	Robert Love

Re: which ioctls matter across filesystems

[Steve French]

Robert Love wrote:

>On Fri, 2005-04-29 at 15:55 -0500, Steve French wrote:
>
>  
>
>>I believe that the spotlight facility of MacOS, and the somewhat similar 
>>Longhorn feature (think Google desktop search/indexing on steroids) 
>>qualify as killer-apps.   I am concerned about how to do better with our 
>>implementations across a distributed (NFS, CIFS etc.) network.   And of 
>>course coalescing async notifications most efficiently is a fascinating 
>>and difficult area to do right - for servers at least.
>>    
>>
>
>If we had some way to efficiently coalesce events, even non-remote stuff
>would drool.  Beagle (our Spotlight killer) would love it.
>
>First thing is, the events cannot be stored in a linked list. ;-)
>
>	Robert Love
>  
>
I should mention one obvious thing - there can be two wire protocols 
here (at least for the CIFS case) - an optimized case which matches -- 
exactly -- to the [better] new semantics that Linux allows and the 
interoperability case (which can be less efficient) ie
the existing Transact2 NOTIFY mechanism which is widely supported by 
current servers and currently defined in fs/cifs/cifspdu.h

I don't mind, nor do others on the Samba team, extending the wire 
protocol for CIFS where it would help that - "killer app" as long as it is
    - optional
    - persuasive benefit
    - helps the network case to better match the -- exact -- local 
semantics applications would expect.

Re: which ioctls matter across filesystems

[Steve French]

Robert Love wrote:

>>What about inotify makes it insufficient for your needs?
>>
>>    
>>
I am not sure - but it is obviously required that inotify can pass over 
CIFS (and probably NFS) since change notification is hardest for the 
user to figure out when they are on a network.   I am not sure how the 
filesystem can detect that a new watch is added to one of its inodes - 
it looks like you added an ioctl to a device but I am still reading 
through your latest patch.   I was looking for something like an inode 
operation that cifs could hook so the fs could be told when a new watch 
was added or one was changed.   In any case I need to construct 
functions somewhat similar to what is in fs/cifs/fcntl.c and need to 
finish/modify CIFSSMBNotify in fs/cifs/cifssmb.c to map the inotify 
flags to the flags available in the CIFS network protocol specifications.
The existing network protocol support for ChangeNotify is more 
straightforward than you might think and the filter flags & actions that 
I have at my disposal for implementing notify across the network are in 
fs/cifs/cifspdu.h already (search for FILE_ACTION_   and FILE_NOTIFY_ if 
you are curious) but obviously they are similar to what the other Samba 
team guys have already told you.

>>What kind of real-world applications exist out there that need inotify
>>functionality, and what sort of requirements do they have (in particular
>>w.r.t. the notification mechanism)?
>>    
>>
>
>A few worksets:
>
>	- Current users, such as FAM and Samba, that need simple file
>	  change notification
>	- Random applications that want to watch a file or two
>	- The Linux desktop
>	- Real-time live-updating indexing systems, such as Beagle,
>	  that compete with f.e. Apple's Spotlight.
>
>Best,
>
>	Robert Love
>
strongly agree, very strongly agree.

The one example we left out - distributed backup applications (those 
guys query me from time to time asking for improvements) - they want to 
know when a server file changes so they can do backup across CIFS.

Re: which ioctls matter across filesystems

[Robert Love]

On Fri, 2005-04-29 at 15:47 -0500, Steve French wrote:

> I am not sure - but it is obviously required that inotify can pass over 
> CIFS (and probably NFS) since change notification is hardest for the 
> user to figure out when they are on a network.   I am not sure how the 
> filesystem can detect that a new watch is added to one of its inodes - 
> it looks like you added an ioctl to a device but I am still reading 
> through your latest patch.   I was looking for something like an inode 
> operation that cifs could hook so the fs could be told when a new watch 
> was added or one was changed.   In any case I need to construct 
> functions somewhat similar to what is in fs/cifs/fcntl.c and need to 
> finish/modify CIFSSMBNotify in fs/cifs/cifssmb.c to map the inotify 
> flags to the flags available in the CIFS network protocol specifications.
> The existing network protocol support for ChangeNotify is more 
> straightforward than you might think and the filter flags & actions that 
> I have at my disposal for implementing notify across the network are in 
> fs/cifs/cifspdu.h already (search for FILE_ACTION_   and FILE_NOTIFY_ if 
> you are curious) but obviously they are similar to what the other Samba 
> team guys have already told you.

So a client adds a watch and the server needs to then physically add the
inotify watch?

If you have a user-space, user-space could just add an inotify watch.

But I guess you live entirely in kernel-space?  Couldn't we just export
our "add watch" interface to you?

	Robert Love

Re: which ioctls matter across filesystems

[Steve French]

Robert Love wrote:

>So a client adds a watch and the server needs to then physically add the
>inotify watch?
>
>  
>
Yes, this creates the interesting situation of two responses (one from 
the local client code, and one from the server) potentially coming as a 
client changes a file which he has a watch on.

>If you have a user-space, user-space could just add an inotify watch.
>
>But I guess you live entirely in kernel-space?  Couldn't we just export
>our "add watch" interface to you?
>
>	Robert Love
>
>  
>
Yes - add watch could be exported, I don't see a way around this since a 
filesystem has to be able to tell the server what to watch.  It does not 
really matter if that were done in kernel or not, but I would prefer it 
to be done in kernel since that would avoid having to ioctl down to the 
kernel

Re: which ioctls matter across filesystems

[Steve French]

Trond Myklebust wrote:

>What kind of real-world applications exist out there that need inotify
>functionality, and what sort of requirements do they have (in particular
>w.r.t. the notification mechanism)?
>
>Cheers,
>  Trond
>  
>
The two cases I can think of which matter (other than the case you 
mention) are:
1) KDE File manager - autoupdate of directory listings which today calls 
D_NOTIFY (a similar feature was first done IIRC in OS/2 for support of 
the workplace shell).   Obviously this is as or more important to 
support well over the network as it is in the local fs, but the client 
implications.   I don't know whether their needs (and the equivalent in 
Gnome) map better to fcntl DNOTIFY or inotify.

2) Support of Network File Servers - The Samba example has already been 
mentioned, but it is important because it would be quite common for a 
series of Samba servers to export shares that are NFS mounted to a set 
of NFS servers (or on other platforms mounted to a cluster 
filesystem).   The CIFS network protocol has long had a notify 
mechanism, and client implementations on various operating systems use 
it, so there is pressure for Samba to support it better.   The Linux 
CIFS client can issue these calls too, but it is marked experimental and 
disabled by default as more work needs to be done to clean it up.

A loosely related issue which will matter a lot in the long run are 
figuring out a way to pass get/setlease requests as the network caching 
mechanisms would otherwise not work in three tier environments (e.g. 
SMB/CIFS client -> Samba server over NFS client mounted to -> NFS 
server, or the reverse).