From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jim Lieb Subject: Re: Re: RFC: fsnotify - Add support for ignoring self initiated events Date: Wed, 11 Sep 2013 10:03:48 -0700 Message-ID: <5053985.ulIRIhdU4n@jlieb-e6410> References: <1378319462-4767-1-git-send-email-jlieb@panasas.com> <20130911142754.GB29749@fieldses.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: To: "J. Bruce Fields" Return-path: Received: from natasha.panasas.com ([67.152.220.90]:55753 "EHLO natasha.panasas.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754249Ab3IKREJ convert rfc822-to-8bit (ORCPT ); Wed, 11 Sep 2013 13:04:09 -0400 In-Reply-To: <20130911142754.GB29749@fieldses.org> Sender: linux-fsdevel-owner@vger.kernel.org List-ID: On Wednesday, September 11, 2013 10:27:54 J. Bruce Fields wrote: > What sort of cache consistency does this give you? (It's not perfect= , > because you always get the notification after the event has already > happened, right?) It is certainly not atomic. However, our (nfs-ganesha) use cases can t= olerate=20 some level of delay. 1. We populate an avl tree with the names from a readdir. If another p= rocess,=20 e.g. samba, adds/removes a file we want to know about it. Readdir alre= ady has=20 races in that the same thing can happen in a portion of the directory a= lready=20 scanned while we are still reading the last of it. 2. We use the avl tree for lookups. Same thing applies here. If the=20 directory is current, we can serve the lookup from cache. In the case = of a=20 lookup prior to event, the next step in the protocol would be to do som= ething=20 directy to the file which will discover the removal. We've always had = the=20 problem of lookups for files that don't exist yet and we cope with retr= ies or=20 ???. All we would get here is a bit more delay. The event will trigger an upcall into the cache inode layer to mark the= =20 directory as "stale". The inotify payload gives us the name as well so= we can=20 do things to the tree like add/remove the entry. In the lookup case, i= f we=20 know the directory is stale, we can go directly to the filesystem rathe= r than=20 assume the directory is consistant. In the end, yes there is an inconsistancy window but it does close. >=20 > It looks like you're using the tgid. I guess Ganesha runs a bunch of > threads all sharing the same tgid? Would a server using multiple > processes instead need a different interface? Yes, it is using the tgid which, of course, implies that the server is = NPTL=20 based (or equiv). It does not take into consideration the multiple pro= cesses=20 with shared memory case. The completely separate process case is cove= red in=20 the samba+nfs-gaanesha case. One knows what the other is doing filtere= d by the=20 ignore flag. >=20 > --b. >=20 > On Wed, Sep 04, 2013 at 11:30:59AM -0700, Jim Lieb wrote: > > Our use case is an NFS+pNFS+9P user mode server. We need to keep o= ur > > caches (dentry+inode) current with the underlying kernel. To do th= is we > > need inotify to feed filesystem events to our upcall infrastructure= =2E We > > place a watch on each directory we have cached and any events that = morph > > that directory would cause invalidates and/or updates to those entr= ies. > >=20 > > The current fsnotify subsystem does most of what we want but implic= it in > > the dnotify/inotify/fanotify interfaces is the assumption that the = watcher > > is an "innocent bystander" whose sole/main function is to draw/remo= ve > > icons > > on a window when someone else adds/removes things from a directory.= Part > > of our use case is that the ganesha.nfsd server is co-resident with= a CIFS > > server which is also exporting the same filesystem(s) and service > > management tools that modify the filesystem structure (snapshots an= d > > volume adds...). The current inotify interface will send up all eve= nts > > but if both servers are equally busy we get two bad results: > >=20 > > 1. each server gets twice the traffic it really needs (theirs and o= urs). > >=20 > > 2. there is no simple way to tell their events from ours in each ev= ent. > >=20 > > This patch set adds a new watch/mark flag (FS_IGNORE_ME) to fsnotif= y. > > Setting this flag causes the watching process's pid to be stored in= the > > mark for the inode. The flag is tested at event time and if set an= d if > > the pid of the event generating process matches the stored pid, the= event > > is ignored, saving the overhead of allocating an event, pushing it = up to > > user space only to be rejected. Being in fsnotify makes it availabl= e to > > any notification scheme built on fsnotify. > >=20 > > The IN_IGNORE_ME flag bit is added to inotify. When set, none of t= he > > other > > event flags will generate an event if the calling process generated= the > > event. Given the current way that inotify_add_watch() validates th= e flags > > argument, discovering whether the kernel supports the flag requires= an > > extra test (set the watch and generate an event...). > >=20 > > The FAN_IGNORE_ME flag bit does the same for fanotify. fanotify in > > current > > kernels will return an EINVAL error if this bit is set, making disc= overy > > easier. One performance side effect is that this flag eliminates t= he need > > and overhead for a test of my_pid =3D=3D e.pid in the event process= ing loop. > >=20 > > We chose inotify rather than the current fanotify because we need t= he > > extra > > events that fanotify cannot (currently) support. dnotify was not t= ouched > > because it is both obsolete and its api makes this extension diffic= ult. > >=20 > > Please review and comment. If it is acceptable, please ACK and mer= ge. > >=20 > > Thanks > >=20 > > Jim Lieb, NFS Ganesha project > >=20 > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-fsd= evel" > > in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html --=20 Jim Lieb Linux Systems Engineer Panasas Inc. "If ease of use was the only requirement, we would all be riding tricyc= les" - Douglas Engelbart 1925=E2=80=932013 -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel= " in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html