* Re: unlink within an open directory stream [not found] <1598378492.82.1332790088063.JavaMail.root@thunderbeast.private.linuxbox.com> @ 2012-03-26 19:29 ` Matt W. Benjamin 0 siblings, 0 replies; 19+ messages in thread From: Matt W. Benjamin @ 2012-03-26 19:29 UTC (permalink / raw) To: Trond Myklebust; +Cc: Boaz Harrosh, linux-nfs, Ganesha NFS List Hi, ----- "Trond Myklebust" <Trond.Myklebust@netapp.com> wrote: > On Mon, 2012-03-26 at 14:55 -0400, Matt W. Benjamin wrote: > > Hi, > > > > Boaz: we do not any longer send a readdir index. We do send a > cookieverf. > > Uhh... That would be so broken, that I can't imagine that is true... I think you're misreading my statement. My intended meaning had to do with Ganesha's generator for the cookies themselves. It's not relevant. > > > Fist of all, I haven't established that the issue we're actually > observing is caused > > by the Linux client sending old cookies to readdir(something). > However, if it is, > > it's in no way better to try to make cookies "more persistent." Nor > should the Linux > > client be expecting it. The assumption is simply flawed. The > protocol introduced > > cookie verifier (a LONG time ago) for a reason. > > Bullshit... The cookie verifier is unimplementable. Feel free to try > or > to show me an implementation that works. However I refuse to waste > any > more of my time trying to make that crap work, having already wasted > several months of my life doing just that. There must be something I'm missing here, but ok. > > -- > Trond Myklebust > Linux NFS client maintainer > > NetApp > Trond.Myklebust@netapp.com > www.netapp.com -- Matt Benjamin The Linux Box 206 South Fifth Ave. Suite 150 Ann Arbor, MI 48104 http://linuxbox.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 ^ permalink raw reply [flat|nested] 19+ messages in thread
[parent not found: <209745840.4.1332606289405.JavaMail.root@thunderbeast.private.linuxbox.com>]
* unlink within an open directory stream [not found] <209745840.4.1332606289405.JavaMail.root@thunderbeast.private.linuxbox.com> @ 2012-03-24 16:34 ` Matt W. Benjamin 2012-03-24 16:44 ` Myklebust, Trond 0 siblings, 1 reply; 19+ messages in thread From: Matt W. Benjamin @ 2012-03-24 16:34 UTC (permalink / raw) To: linux-nfs; +Cc: Boaz Harrosh Hi, Folks testing linux nfs + Ganesha with bonnie++ have noticed the issue described here (but I didn't see traffic on this list): http://bugs.centos.org/view.php?id=5496 https://bugzilla.redhat.com/show_bug.cgi?id=789452 Ie, an application which unlinks on an open (nfs) directory stream, and continues to read and unlink progressively, will generally not see all the entries originally in the stream. Reopening the stream ives the "correct" result. So, apparently older clients were more forgiving, as mentioned in the links. Is the current behavior one the client is intending? Thanks, Matt -- Matt Benjamin The Linux Box 206 South Fifth Ave. Suite 150 Ann Arbor, MI 48104 http://linuxbox.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: unlink within an open directory stream 2012-03-24 16:34 ` Matt W. Benjamin @ 2012-03-24 16:44 ` Myklebust, Trond 2012-03-24 16:53 ` Matt W. Benjamin 0 siblings, 1 reply; 19+ messages in thread From: Myklebust, Trond @ 2012-03-24 16:44 UTC (permalink / raw) To: Matt W. Benjamin; +Cc: linux-nfs, Boaz Harrosh T24gU2F0LCAyMDEyLTAzLTI0IGF0IDEyOjM0IC0wNDAwLCBNYXR0IFcuIEJlbmphbWluIHdyb3Rl Og0KPiBIaSwNCj4gDQo+IEZvbGtzIHRlc3RpbmcgbGludXggbmZzICsgR2FuZXNoYSB3aXRoIGJv bm5pZSsrIGhhdmUgbm90aWNlZCB0aGUgaXNzdWUgZGVzY3JpYmVkIGhlcmUgKGJ1dCBJIGRpZG4n dCBzZWUgdHJhZmZpYyBvbiB0aGlzIGxpc3QpOg0KPiANCj4gaHR0cDovL2J1Z3MuY2VudG9zLm9y Zy92aWV3LnBocD9pZD01NDk2DQo+IGh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93X2J1 Zy5jZ2k/aWQ9Nzg5NDUyDQo+IA0KPiBJZSwgYW4gYXBwbGljYXRpb24gd2hpY2ggdW5saW5rcyBv biBhbiBvcGVuIChuZnMpIGRpcmVjdG9yeSBzdHJlYW0sIGFuZCBjb250aW51ZXMgdG8gcmVhZCBh bmQgdW5saW5rIHByb2dyZXNzaXZlbHksIHdpbGwgZ2VuZXJhbGx5IG5vdCBzZWUgYWxsIHRoZSBl bnRyaWVzIG9yaWdpbmFsbHkgaW4gdGhlIHN0cmVhbS4gIFJlb3BlbmluZyB0aGUgc3RyZWFtIGl2 ZXMgdGhlICJjb3JyZWN0IiByZXN1bHQuICBTbywgYXBwYXJlbnRseSBvbGRlciBjbGllbnRzIHdl cmUgbW9yZSBmb3JnaXZpbmcsIGFzIG1lbnRpb25lZCBpbiB0aGUgbGlua3MuICBJcyB0aGUgY3Vy cmVudCBiZWhhdmlvciBvbmUgdGhlIGNsaWVudCBpcyBpbnRlbmRpbmc/DQo+IA0KPiBUaGFua3Ms DQo+IA0KPiBNYXR0DQoNCldoYXQgaXMgc28gcGFydGljdWxhciBhYm91dCB0aGUgZ2FuZXNoYSBy ZWFkZGlyIGltcGxlbWVudGF0aW9uPw0KDQotLSANClRyb25kIE15a2xlYnVzdA0KTGludXggTkZT IGNsaWVudCBtYWludGFpbmVyDQoNCk5ldEFwcA0KVHJvbmQuTXlrbGVidXN0QG5ldGFwcC5jb20N Cnd3dy5uZXRhcHAuY29tDQoNCg== ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: unlink within an open directory stream 2012-03-24 16:44 ` Myklebust, Trond @ 2012-03-24 16:53 ` Matt W. Benjamin 2012-03-24 17:12 ` Myklebust, Trond 0 siblings, 1 reply; 19+ messages in thread From: Matt W. Benjamin @ 2012-03-24 16:53 UTC (permalink / raw) To: Trond Myklebust; +Cc: linux-nfs, Boaz Harrosh Hi, I don't think anything is. Or, people originally reported the behavior against knfsd. Matt ----- "Trond Myklebust" <Trond.Myklebust@netapp.com> wrote: > On Sat, 2012-03-24 at 12:34 -0400, Matt W. Benjamin wrote: > > Hi, > > > > Folks testing linux nfs + Ganesha with bonnie++ have noticed the > issue described here (but I didn't see traffic on this list): > > > > http://bugs.centos.org/view.php?id=5496 > > https://bugzilla.redhat.com/show_bug.cgi?id=789452 > > > > Ie, an application which unlinks on an open (nfs) directory stream, > and continues to read and unlink progressively, will generally not see > all the entries originally in the stream. Reopening the stream ives > the "correct" result. So, apparently older clients were more > forgiving, as mentioned in the links. Is the current behavior one the > client is intending? > > > > Thanks, > > > > Matt > > What is so particular about the ganesha readdir implementation? > > -- > Trond Myklebust > Linux NFS client maintainer > > NetApp > Trond.Myklebust@netapp.com > www.netapp.com -- Matt Benjamin The Linux Box 206 South Fifth Ave. Suite 150 Ann Arbor, MI 48104 http://linuxbox.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: unlink within an open directory stream 2012-03-24 16:53 ` Matt W. Benjamin @ 2012-03-24 17:12 ` Myklebust, Trond 2012-03-24 17:43 ` Matt W. Benjamin 2012-03-26 18:17 ` Boaz Harrosh 0 siblings, 2 replies; 19+ messages in thread From: Myklebust, Trond @ 2012-03-24 17:12 UTC (permalink / raw) To: Matt W. Benjamin; +Cc: linux-nfs, Boaz Harrosh T24gU2F0LCAyMDEyLTAzLTI0IGF0IDEyOjUzIC0wNDAwLCBNYXR0IFcuIEJlbmphbWluIHdyb3Rl Og0KPiBIaSwNCj4gDQo+IEkgZG9uJ3QgdGhpbmsgYW55dGhpbmcgaXMuICBPciwgcGVvcGxlIG9y aWdpbmFsbHkgcmVwb3J0ZWQgdGhlIGJlaGF2aW9yIGFnYWluc3Qga25mc2QuDQo+IA0KPiBNYXR0 DQoNClRoZXJlIGlzIGEga25vd24gaXNzdWUgd2l0aCBleHQyLzMvNCBnZW5lcmF0aW5nIG5vbi11 bmlxdWUgcmVhZGRpcg0KY29va2llcy4gSXQgcmFyZWx5IGhpdHMgeW91IHdoZW4geW91IGFyZSBj cmVhdGluZyBzbWFsbCBkaXJlY3RvcmllcywgYnV0DQppdCBmcmVxdWVudGx5IGhpdHMgeW91IHdp dGggbGFyZ2VyIG9uZXMuIEEgZml4IGlzIHVuZGVyd2F5IHRoYXQgc2hvdWxkDQpzaWduaWZpY2Fu dGx5IHJlZHVjZSB0aGUgZnJlcXVlbmN5IG9mIGNvb2tpZSBjb2xsaXNpb25zLg0KDQpSZWNlbnQg TkZTIGNsaWVudHMgd2lsbCBhY3R1YWxseSBkZXRlY3QgdGhlIHByZXNlbmNlIG9mIHRob3NlIGNv b2tpZQ0KbG9vcHMsIGFuZCBsb2cgdGhlbSBpbiB0aGUga2VybmVsIHN5c2xvZy4gVGhhdCB3b3Vs ZCB0aGVyZWZvcmUgYmUgdGhlDQpmaXJzdCB0aGluZyB0aGF0IEknZCBjaGVjayBpZiBjb25mcm9u dGVkIHdpdGggdGhpcyBraW5kIG9mIHByb2JsZW0uDQoNCkNoZWVycw0KICBUcm9uZA0KDQo+IC0t LS0tICJUcm9uZCBNeWtsZWJ1c3QiIDxUcm9uZC5NeWtsZWJ1c3RAbmV0YXBwLmNvbT4gd3JvdGU6 DQo+IA0KPiA+IE9uIFNhdCwgMjAxMi0wMy0yNCBhdCAxMjozNCAtMDQwMCwgTWF0dCBXLiBCZW5q YW1pbiB3cm90ZToNCj4gPiA+IEhpLA0KPiA+ID4gDQo+ID4gPiBGb2xrcyB0ZXN0aW5nIGxpbnV4 IG5mcyArIEdhbmVzaGEgd2l0aCBib25uaWUrKyBoYXZlIG5vdGljZWQgdGhlDQo+ID4gaXNzdWUg ZGVzY3JpYmVkIGhlcmUgKGJ1dCBJIGRpZG4ndCBzZWUgdHJhZmZpYyBvbiB0aGlzIGxpc3QpOg0K PiA+ID4gDQo+ID4gPiBodHRwOi8vYnVncy5jZW50b3Mub3JnL3ZpZXcucGhwP2lkPTU0OTYNCj4g PiA+IGh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93X2J1Zy5jZ2k/aWQ9Nzg5NDUyDQo+ ID4gPiANCj4gPiA+IEllLCBhbiBhcHBsaWNhdGlvbiB3aGljaCB1bmxpbmtzIG9uIGFuIG9wZW4g KG5mcykgZGlyZWN0b3J5IHN0cmVhbSwNCj4gPiBhbmQgY29udGludWVzIHRvIHJlYWQgYW5kIHVu bGluayBwcm9ncmVzc2l2ZWx5LCB3aWxsIGdlbmVyYWxseSBub3Qgc2VlDQo+ID4gYWxsIHRoZSBl bnRyaWVzIG9yaWdpbmFsbHkgaW4gdGhlIHN0cmVhbS4gIFJlb3BlbmluZyB0aGUgc3RyZWFtIGl2 ZXMNCj4gPiB0aGUgImNvcnJlY3QiIHJlc3VsdC4gIFNvLCBhcHBhcmVudGx5IG9sZGVyIGNsaWVu dHMgd2VyZSBtb3JlDQo+ID4gZm9yZ2l2aW5nLCBhcyBtZW50aW9uZWQgaW4gdGhlIGxpbmtzLiAg SXMgdGhlIGN1cnJlbnQgYmVoYXZpb3Igb25lIHRoZQ0KPiA+IGNsaWVudCBpcyBpbnRlbmRpbmc/ DQo+ID4gPiANCj4gPiA+IFRoYW5rcywNCj4gPiA+IA0KPiA+ID4gTWF0dA0KPiA+IA0KPiA+IFdo YXQgaXMgc28gcGFydGljdWxhciBhYm91dCB0aGUgZ2FuZXNoYSByZWFkZGlyIGltcGxlbWVudGF0 aW9uPw0KPiA+IA0KPiA+IC0tIA0KPiA+IFRyb25kIE15a2xlYnVzdA0KPiA+IExpbnV4IE5GUyBj bGllbnQgbWFpbnRhaW5lcg0KPiA+IA0KPiA+IE5ldEFwcA0KPiA+IFRyb25kLk15a2xlYnVzdEBu ZXRhcHAuY29tDQo+ID4gd3d3Lm5ldGFwcC5jb20NCj4gDQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0 DQpMaW51eCBORlMgY2xpZW50IG1haW50YWluZXINCg0KTmV0QXBwDQpUcm9uZC5NeWtsZWJ1c3RA bmV0YXBwLmNvbQ0Kd3d3Lm5ldGFwcC5jb20NCg0K ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: unlink within an open directory stream 2012-03-24 17:12 ` Myklebust, Trond @ 2012-03-24 17:43 ` Matt W. Benjamin 2012-03-26 18:17 ` Boaz Harrosh 1 sibling, 0 replies; 19+ messages in thread From: Matt W. Benjamin @ 2012-03-24 17:43 UTC (permalink / raw) To: Trond Myklebust; +Cc: linux-nfs, Boaz Harrosh Hi, Thanks. I'm pretty sure that's not happening (haven't seen any such message from Linux 3.3.0-rc7, for example), but I'll retest. Matt ----- "Trond Myklebust" <Trond.Myklebust@netapp.com> wrote: -- Matt Benjamin The Linux Box 206 South Fifth Ave. Suite 150 Ann Arbor, MI 48104 http://linuxbox.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: unlink within an open directory stream 2012-03-24 17:12 ` Myklebust, Trond 2012-03-24 17:43 ` Matt W. Benjamin @ 2012-03-26 18:17 ` Boaz Harrosh 2012-03-26 18:25 ` Myklebust, Trond ` (2 more replies) 1 sibling, 3 replies; 19+ messages in thread From: Boaz Harrosh @ 2012-03-26 18:17 UTC (permalink / raw) To: Myklebust, Trond; +Cc: Matt W. Benjamin, linux-nfs On 03/24/2012 10:12 AM, Myklebust, Trond wrote: > On Sat, 2012-03-24 at 12:53 -0400, Matt W. Benjamin wrote: >> Hi, >> >> I don't think anything is. Or, people originally reported the behavior against knfsd. >> >> Matt > > There is a known issue with ext2/3/4 generating non-unique readdir > cookies. It rarely hits you when you are creating small directories, but > it frequently hits you with larger ones. A fix is underway that should > significantly reduce the frequency of cookie collisions. > > Recent NFS clients will actually detect the presence of those cookie > loops, and log them in the kernel syslog. That would therefore be the > first thing that I'd check if confronted with this kind of problem. > > Cheers > Trond > Trond please look on the bug report links below. It's not the "cookie collisions" case. It's the new (post RHEL 6.0 Kernel) NFS need for opendir after an unlink. Now the POSIX man page *does* say that applications must re-opendir after unlink, but there are some applications who did not read the manual, and since it works with local filesystems and old nfs, (What Kernel RHEL 6.0 is based on?) they never noticed the bug and never fixed it. Could we easily support the broken application by being bug compatible to old NFS versions? .i.e Don't require re-opendir after unlink of a file. There are more examples in the bug reports below but basically bonnie++ does the following: DIR *d = opendir("."); dirent *file_ent; while((file_ent = readdir(d)) != NULL) { unlink( file_ent->d_name)) } closedir(d); where it actually needs to do: DIR *d = opendir("."); dirent *file_ent; while((file_ent = readdir(d)) != NULL) { unlink( file_ent->d_name)) closedir(d); d = opendir("."); } closedir(d); But again case one used to work with old NFS. And it looks like it is not Server dependent. We saw this both with Ganesha as well as knfsd <snip> >> http://bugs.centos.org/view.php?id=5496 >> https://bugzilla.redhat.com/show_bug.cgi?id=789452 Thanks Boaz ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: unlink within an open directory stream 2012-03-26 18:17 ` Boaz Harrosh @ 2012-03-26 18:25 ` Myklebust, Trond 2012-03-26 18:43 ` Boaz Harrosh 2012-03-30 20:17 ` Jeff Layton 2012-04-04 15:35 ` Jeff Layton 2 siblings, 1 reply; 19+ messages in thread From: Myklebust, Trond @ 2012-03-26 18:25 UTC (permalink / raw) To: Boaz Harrosh; +Cc: Matt W. Benjamin, linux-nfs T24gTW9uLCAyMDEyLTAzLTI2IGF0IDExOjE3IC0wNzAwLCBCb2F6IEhhcnJvc2ggd3JvdGU6DQo+ IE9uIDAzLzI0LzIwMTIgMTA6MTIgQU0sIE15a2xlYnVzdCwgVHJvbmQgd3JvdGU6DQo+IA0KPiA+ IE9uIFNhdCwgMjAxMi0wMy0yNCBhdCAxMjo1MyAtMDQwMCwgTWF0dCBXLiBCZW5qYW1pbiB3cm90 ZToNCj4gPj4gSGksDQo+ID4+DQo+ID4+IEkgZG9uJ3QgdGhpbmsgYW55dGhpbmcgaXMuICBPciwg cGVvcGxlIG9yaWdpbmFsbHkgcmVwb3J0ZWQgdGhlIGJlaGF2aW9yIGFnYWluc3Qga25mc2QuDQo+ ID4+DQo+ID4+IE1hdHQNCj4gPiANCj4gPiBUaGVyZSBpcyBhIGtub3duIGlzc3VlIHdpdGggZXh0 Mi8zLzQgZ2VuZXJhdGluZyBub24tdW5pcXVlIHJlYWRkaXINCj4gPiBjb29raWVzLiBJdCByYXJl bHkgaGl0cyB5b3Ugd2hlbiB5b3UgYXJlIGNyZWF0aW5nIHNtYWxsIGRpcmVjdG9yaWVzLCBidXQN Cj4gPiBpdCBmcmVxdWVudGx5IGhpdHMgeW91IHdpdGggbGFyZ2VyIG9uZXMuIEEgZml4IGlzIHVu ZGVyd2F5IHRoYXQgc2hvdWxkDQo+ID4gc2lnbmlmaWNhbnRseSByZWR1Y2UgdGhlIGZyZXF1ZW5j eSBvZiBjb29raWUgY29sbGlzaW9ucy4NCj4gPiANCj4gPiBSZWNlbnQgTkZTIGNsaWVudHMgd2ls bCBhY3R1YWxseSBkZXRlY3QgdGhlIHByZXNlbmNlIG9mIHRob3NlIGNvb2tpZQ0KPiA+IGxvb3Bz LCBhbmQgbG9nIHRoZW0gaW4gdGhlIGtlcm5lbCBzeXNsb2cuIFRoYXQgd291bGQgdGhlcmVmb3Jl IGJlIHRoZQ0KPiA+IGZpcnN0IHRoaW5nIHRoYXQgSSdkIGNoZWNrIGlmIGNvbmZyb250ZWQgd2l0 aCB0aGlzIGtpbmQgb2YgcHJvYmxlbS4NCj4gPiANCj4gPiBDaGVlcnMNCj4gPiAgIFRyb25kDQo+ ID4gDQo+IA0KPiANCj4gVHJvbmQgcGxlYXNlIGxvb2sgb24gdGhlIGJ1ZyByZXBvcnQgbGlua3Mg YmVsb3cuIEl0J3Mgbm90IHRoZSAiY29va2llIGNvbGxpc2lvbnMiIGNhc2UuDQo+IA0KPiBJdCdz IHRoZSBuZXcgKHBvc3QgUkhFTCA2LjAgS2VybmVsKSBORlMgbmVlZCBmb3Igb3BlbmRpciBhZnRl ciBhbiB1bmxpbmsuDQoNCldoYXQ/DQoNCj4gTm93IHRoZSBQT1NJWCBtYW4gcGFnZSAqZG9lcyog c2F5IHRoYXQgYXBwbGljYXRpb25zIG11c3QgcmUtb3BlbmRpciBhZnRlcg0KPiB1bmxpbmssIGJ1 dCB0aGVyZSBhcmUgc29tZSBhcHBsaWNhdGlvbnMgd2hvIGRpZCBub3QgcmVhZCB0aGUgbWFudWFs LCBhbmQgc2luY2UNCj4gaXQgd29ya3Mgd2l0aCBsb2NhbCBmaWxlc3lzdGVtcyBhbmQgb2xkIG5m cywgKFdoYXQgS2VybmVsIFJIRUwgNi4wIGlzIGJhc2VkIG9uPykNCj4gdGhleSBuZXZlciBub3Rp Y2VkIHRoZSBidWcgYW5kIG5ldmVyIGZpeGVkIGl0Lg0KPiANCj4gQ291bGQgd2UgZWFzaWx5IHN1 cHBvcnQgdGhlIGJyb2tlbiBhcHBsaWNhdGlvbiBieSBiZWluZyBidWcgY29tcGF0aWJsZSB0bw0K PiBvbGQgTkZTIHZlcnNpb25zPw0KPiAuaS5lIERvbid0IHJlcXVpcmUgcmUtb3BlbmRpciBhZnRl ciB1bmxpbmsgb2YgYSBmaWxlLg0KPiANCj4gVGhlcmUgYXJlIG1vcmUgZXhhbXBsZXMgaW4gdGhl IGJ1ZyByZXBvcnRzIGJlbG93IGJ1dCBiYXNpY2FsbHkgYm9ubmllKysNCj4gZG9lcyB0aGUgZm9s bG93aW5nOg0KPiAJRElSICpkID0gb3BlbmRpcigiLiIpOw0KPiAJZGlyZW50ICpmaWxlX2VudDsN Cj4gCXdoaWxlKChmaWxlX2VudCA9IHJlYWRkaXIoZCkpICE9IE5VTEwpIHsNCj4gCQl1bmxpbmso IGZpbGVfZW50LT5kX25hbWUpKQ0KPiAJfQ0KPiAJY2xvc2VkaXIoZCk7DQo+IA0KPiB3aGVyZSBp dCBhY3R1YWxseSBuZWVkcyB0byBkbzoNCj4gDQo+IAlESVIgKmQgPSBvcGVuZGlyKCIuIik7DQo+ IAlkaXJlbnQgKmZpbGVfZW50Ow0KPiAJd2hpbGUoKGZpbGVfZW50ID0gcmVhZGRpcihkKSkgIT0g TlVMTCkgew0KPiAJCXVubGluayggZmlsZV9lbnQtPmRfbmFtZSkpDQo+IA0KPiAJCWNsb3NlZGly KGQpOw0KPiAJCWQgPSBvcGVuZGlyKCIuIik7DQo+IAl9DQo+IAljbG9zZWRpcihkKTsNCj4gDQo+ IEJ1dCBhZ2FpbiBjYXNlIG9uZSB1c2VkIHRvIHdvcmsgd2l0aCBvbGQgTkZTLiBBbmQgaXQgbG9v a3MgbGlrZQ0KPiBpdCBpcyBub3QgU2VydmVyIGRlcGVuZGVudC4gV2Ugc2F3IHRoaXMgYm90aCB3 aXRoIEdhbmVzaGEgYXMgd2VsbA0KPiBhcyBrbmZzZA0KDQpOby4NCg0KSWYgdGhlIHNlcnZlciBz dXBwb3J0cyBwZXJtYW5lbnQgcmVhZGRpciBjb29raWVzLCB0aGVuIHRoZXJlIGlzIG5vIG5lZWQN CmZvciBvcGVuZGlyIGFmdGVyIHVubGluay4NCg0KSWYgdGhlIHNlcnZlciBkb2VzIG5vdCBoYXZl IHBlcm1hbmVudCByZWFkZGlyIGNvb2tpZXMsIHRoZW4gb3VyIGNsaWVudA0KaGFzIG5ldmVyIHN1 cHBvcnRlZCBpdCBhbnl3YXkgKG5vciB3aWxsIGl0IGV2ZXIgZG8gc28pLiBUaGF0IHdob2xlDQon Y29va2lldmVyZicgUkVBRERJUiBidWxsc2hpdCBoYXMgbmV2ZXIgcHJvdmlkZWQgYSB3b3JrYWJs ZSBtb2RlbCBmb3IgYQ0KUE9TSVggY2xpZW50Li4uDQoNCg0KLS0gDQpUcm9uZCBNeWtsZWJ1c3QN CkxpbnV4IE5GUyBjbGllbnQgbWFpbnRhaW5lcg0KDQpOZXRBcHANClRyb25kLk15a2xlYnVzdEBu ZXRhcHAuY29tDQp3d3cubmV0YXBwLmNvbQ0KDQo= ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: unlink within an open directory stream 2012-03-26 18:25 ` Myklebust, Trond @ 2012-03-26 18:43 ` Boaz Harrosh 2012-03-26 18:55 ` Matt W. Benjamin 0 siblings, 1 reply; 19+ messages in thread From: Boaz Harrosh @ 2012-03-26 18:43 UTC (permalink / raw) To: Myklebust, Trond; +Cc: Matt W. Benjamin, linux-nfs, Ganesha NFS List On 03/26/2012 11:25 AM, Myklebust, Trond wrote: > On Mon, 2012-03-26 at 11:17 -0700, Boaz Harrosh wrote: >> On 03/24/2012 10:12 AM, Myklebust, Trond wrote: >> >> >> It's the new (post RHEL 6.0 Kernel) NFS need for opendir after an unlink. > > What? > The links below report on regression in RHEL 6.2 which have a new NFS implementation where 6.0 used to be fine. (Or it's what I understood from the reporters) <snip> > > No. > > If the server supports permanent readdir cookies, then there is no need > for opendir after unlink. > > If the server does not have permanent readdir cookies, then our client > has never supported it anyway (nor will it ever do so). That whole > 'cookieverf' READDIR bullshit has never provided a workable model for a > POSIX client... Thanks Trond for the explanation. Forgive my slowness. So you are saying that with knfsd it's actually filesystem dependent and maybe the reporters did not compare apples-to-apples and the difference is in the behind file system. Matt I'm sure Trond is right with regard to Ganesha, because we send a readdir index and a cookieverf, which will change after unlink. Since the application does not call opendir again the client will send the old cookies, which is now a different file or might get out-of-bounds for Ganesha. Let me think about it a bit. I'm sure we can send a Better 64bit cookie, which will be more persistent, even across unlinks. > > Thanks Boaz ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: unlink within an open directory stream 2012-03-26 18:43 ` Boaz Harrosh @ 2012-03-26 18:55 ` Matt W. Benjamin 2012-03-26 19:01 ` Myklebust, Trond ` (2 more replies) 0 siblings, 3 replies; 19+ messages in thread From: Matt W. Benjamin @ 2012-03-26 18:55 UTC (permalink / raw) To: Boaz Harrosh; +Cc: linux-nfs, Ganesha NFS List, Trond Myklebust Hi, Boaz: we do not any longer send a readdir index. We do send a cookieverf. Fist of all, I haven't established that the issue we're actually observing is caused by the Linux client sending old cookies to readdir(something). However, if it is, it's in no way better to try to make cookies "more persistent." Nor should the Linux client be expecting it. The assumption is simply flawed. The protocol introduced cookie verifier (a LONG time ago) for a reason. Matt ----- "Boaz Harrosh" <bharrosh@panasas.com> wrote: > On 03/26/2012 11:25 AM, Myklebust, Trond wrote: > > > On Mon, 2012-03-26 at 11:17 -0700, Boaz Harrosh wrote: > >> On 03/24/2012 10:12 AM, Myklebust, Trond wrote: > >> > >> > >> It's the new (post RHEL 6.0 Kernel) NFS need for opendir after an > unlink. > > > > What? > > > > > The links below report on regression in RHEL 6.2 which have a new NFS > implementation where 6.0 used to be fine. > (Or it's what I understood from the reporters) > > <snip> > > > > > > No. > > > > If the server supports permanent readdir cookies, then there is no > need > > for opendir after unlink. > > > > If the server does not have permanent readdir cookies, then our > client > > has never supported it anyway (nor will it ever do so). That whole > > 'cookieverf' READDIR bullshit has never provided a workable model > for a > > POSIX client... > > > Thanks Trond for the explanation. Forgive my slowness. So you are > saying that with knfsd it's actually filesystem dependent and maybe > the reporters did not compare apples-to-apples and the difference is > in the behind file system. > > Matt I'm sure Trond is right with regard to Ganesha, because we > send a readdir index and a cookieverf, which will change after > unlink. Since the application does not call opendir again the > client will send the old cookies, which is now a different file > or might get out-of-bounds for Ganesha. > > Let me think about it a bit. I'm sure we can send a Better > 64bit cookie, which will be more persistent, even across unlinks. > > > > > > > > Thanks > Boaz -- Matt Benjamin The Linux Box 206 South Fifth Ave. Suite 150 Ann Arbor, MI 48104 http://linuxbox.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: unlink within an open directory stream 2012-03-26 18:55 ` Matt W. Benjamin @ 2012-03-26 19:01 ` Myklebust, Trond 2012-03-26 19:07 ` Boaz Harrosh 2012-03-30 15:24 ` Peter Staubach 2 siblings, 0 replies; 19+ messages in thread From: Myklebust, Trond @ 2012-03-26 19:01 UTC (permalink / raw) To: Matt W. Benjamin; +Cc: Boaz Harrosh, linux-nfs, Ganesha NFS List T24gTW9uLCAyMDEyLTAzLTI2IGF0IDE0OjU1IC0wNDAwLCBNYXR0IFcuIEJlbmphbWluIHdyb3Rl Og0KPiBIaSwNCj4gDQo+IEJvYXo6ICB3ZSBkbyBub3QgYW55IGxvbmdlciBzZW5kIGEgcmVhZGRp ciBpbmRleC4gIFdlIGRvIHNlbmQgYSBjb29raWV2ZXJmLg0KDQpVaGguLi4gVGhhdCB3b3VsZCBi ZSBzbyBicm9rZW4sIHRoYXQgSSBjYW4ndCBpbWFnaW5lIHRoYXQgaXMgdHJ1ZS4uLg0KDQo+IEZp c3Qgb2YgYWxsLCBJIGhhdmVuJ3QgZXN0YWJsaXNoZWQgdGhhdCB0aGUgaXNzdWUgd2UncmUgYWN0 dWFsbHkgb2JzZXJ2aW5nIGlzIGNhdXNlZA0KPiBieSB0aGUgTGludXggY2xpZW50IHNlbmRpbmcg b2xkIGNvb2tpZXMgdG8gcmVhZGRpcihzb21ldGhpbmcpLiAgSG93ZXZlciwgaWYgaXQgaXMsDQo+ IGl0J3MgaW4gbm8gd2F5IGJldHRlciB0byB0cnkgdG8gbWFrZSBjb29raWVzICJtb3JlIHBlcnNp c3RlbnQuIiAgTm9yIHNob3VsZCB0aGUgTGludXgNCj4gY2xpZW50IGJlIGV4cGVjdGluZyBpdC4g IFRoZSBhc3N1bXB0aW9uIGlzIHNpbXBseSBmbGF3ZWQuICBUaGUgcHJvdG9jb2wgaW50cm9kdWNl ZA0KPiBjb29raWUgdmVyaWZpZXIgKGEgTE9ORyB0aW1lIGFnbykgZm9yIGEgcmVhc29uLg0KDQpC dWxsc2hpdC4uLiBUaGUgY29va2llIHZlcmlmaWVyIGlzIHVuaW1wbGVtZW50YWJsZS4gRmVlbCBm cmVlIHRvIHRyeSBvcg0KdG8gc2hvdyBtZSBhbiBpbXBsZW1lbnRhdGlvbiB0aGF0IHdvcmtzLiBI b3dldmVyIEkgcmVmdXNlIHRvIHdhc3RlIGFueQ0KbW9yZSBvZiBteSB0aW1lIHRyeWluZyB0byBt YWtlIHRoYXQgY3JhcCB3b3JrLCBoYXZpbmcgYWxyZWFkeSB3YXN0ZWQNCnNldmVyYWwgbW9udGhz IG9mIG15IGxpZmUgZG9pbmcganVzdCB0aGF0Lg0KDQotLSANClRyb25kIE15a2xlYnVzdA0KTGlu dXggTkZTIGNsaWVudCBtYWludGFpbmVyDQoNCk5ldEFwcA0KVHJvbmQuTXlrbGVidXN0QG5ldGFw cC5jb20NCnd3dy5uZXRhcHAuY29tDQoNCg== ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: unlink within an open directory stream 2012-03-26 18:55 ` Matt W. Benjamin 2012-03-26 19:01 ` Myklebust, Trond @ 2012-03-26 19:07 ` Boaz Harrosh 2012-03-30 15:24 ` Peter Staubach 2 siblings, 0 replies; 19+ messages in thread From: Boaz Harrosh @ 2012-03-26 19:07 UTC (permalink / raw) To: Matt W. Benjamin; +Cc: linux-nfs, Ganesha NFS List, Trond Myklebust On 03/26/2012 11:55 AM, Matt W. Benjamin wrote: > Hi, > > Boaz: we do not any longer send a readdir index. We do send a cookieverf. > > Fist of all, I haven't established that the issue we're actually observing is caused > by the Linux client sending old cookies to readdir(something). However, if it is, > it's in no way better to try to make cookies "more persistent." Nor should the Linux > client be expecting it. The assumption is simply flawed. The protocol introduced > cookie verifier (a LONG time ago) for a reason. > > Matt Please don't top-post in a Linux-mailing lists. Us Linux guys are not used to it and get confused. Regarding the cookie-verifier it looks like the Linux client never supported that. So even if it will start to, from here on, we are in trouble with old clients. Since we are in the business of being bug backward compatible then fixing the client will not help. Because you must agree the easiest is to fix bonnie++ but that's a privilege we do not have. I think we should imitate the behavior of knfsd+exfs3 I'm sure it's easy for Ganesha to fix it. Lets talk about it in this week call. I have ideas that will not change current code, just a small enhancement. Thanks Boaz > ----- "Boaz Harrosh" <bharrosh@panasas.com> wrote: > >> On 03/26/2012 11:25 AM, Myklebust, Trond wrote: >> >>> On Mon, 2012-03-26 at 11:17 -0700, Boaz Harrosh wrote: >>>> On 03/24/2012 10:12 AM, Myklebust, Trond wrote: >>>> >>>> >>>> It's the new (post RHEL 6.0 Kernel) NFS need for opendir after an >> unlink. >>> >>> What? >>> >> >> >> The links below report on regression in RHEL 6.2 which have a new NFS >> implementation where 6.0 used to be fine. >> (Or it's what I understood from the reporters) >> >> <snip> >> >>> >> >>> No. >>> >>> If the server supports permanent readdir cookies, then there is no >> need >>> for opendir after unlink. >>> >>> If the server does not have permanent readdir cookies, then our >> client >>> has never supported it anyway (nor will it ever do so). That whole >>> 'cookieverf' READDIR bullshit has never provided a workable model >> for a >>> POSIX client... >> >> >> Thanks Trond for the explanation. Forgive my slowness. So you are >> saying that with knfsd it's actually filesystem dependent and maybe >> the reporters did not compare apples-to-apples and the difference is >> in the behind file system. >> >> Matt I'm sure Trond is right with regard to Ganesha, because we >> send a readdir index and a cookieverf, which will change after >> unlink. Since the application does not call opendir again the >> client will send the old cookies, which is now a different file >> or might get out-of-bounds for Ganesha. >> >> Let me think about it a bit. I'm sure we can send a Better >> 64bit cookie, which will be more persistent, even across unlinks. >> >>> >>> >> >> >> Thanks >> Boaz > ^ permalink raw reply [flat|nested] 19+ messages in thread
* RE: unlink within an open directory stream 2012-03-26 18:55 ` Matt W. Benjamin 2012-03-26 19:01 ` Myklebust, Trond 2012-03-26 19:07 ` Boaz Harrosh @ 2012-03-30 15:24 ` Peter Staubach 2012-03-30 15:46 ` Myklebust, Trond 2 siblings, 1 reply; 19+ messages in thread From: Peter Staubach @ 2012-03-30 15:24 UTC (permalink / raw) To: Matt W. Benjamin, Boaz Harrosh Cc: linux-nfs, Ganesha NFS List, Trond Myklebust WWVzLCBobW1tLCB1bW1tLi4uDQoNCkEgbG9uZyB0aW1lLCBJIHRob3VnaHQgdGhhdCB0aGUgcmVh ZGRpciBjb29raWUgdmVyaWZpZXIgc291bmRlZCBsaWtlIGEgZ29vZCBpZGVhLiAgQWZ0ZXIgYWxs LCBpZiBkaXJlY3RvcmllcyBjYW4gYmUgY29tcGFjdGVkLCBpdCB3b3VsZCBiZSBuaWNlIHRvIGJl IGFibGUgdG8gdGVsbCB3aGVuIHRoZSBjb29raWUgaGFzIGJlY29tZSBpbnZhbGlkLCByaWdodD8g IFNvLCBJIGFkZGVkIGl0IHRvIHRoZSBORlN2MyBwcm90b2NvbC4NCg0KSXQgd2FzIG9ubHkgYWZ0 ZXJ3YXJkcywgaW4gYXR0ZW1wdGluZyB0byBpbXBsZW1lbnQgaXQsIHRoYXQgdGhlIHJlYWwgdHJ1 dGggd2FzIGRpc2NvdmVyZWQuICBUaGlzIGlzIHRoYXQgdGhlIGludGVyZmFjZXMgYXJlIG5vdCBz dWZmaWNpZW50IHRvIGJlIGFibGUgdG8gY29udmV5IHRoaXMgaW5mb3JtYXRpb24gc3VmZmljaWVu dGx5IGFuZCB0byBiZSBhYmxlIHRvIHJlY292ZXIgZnJvbSB0aGUgc2l0dWF0aW9uLiAgQ3VycmVu dGx5LCByZWNvdmVyeSByZXF1aXJlIGFjdGlvbiBvbiB0aGUgcGFydCBvZiB0aGUgYXBwbGljYXRp b24uDQoNCkluIHJldHJvc3BlY3QsIEkgd291bGRuJ3QgYWRkIHRoZSBjb29raWUgdmVyaWZpZXIg aWYgSSB3YXMgZG9pbmcgaXQgYWdhaW4uDQoNClRoYW5rcyBmb3IgdGhlIHN1cHBvcnQsIHRob3Vn aC4gIDotKQ0KDQpJIGRvIGRpc2FncmVlIHdpdGggVHJvbmQgdGhvdWdoLCBpbiB0aGF0IGFwcGxp Y2F0aW9ucyBtdXN0IGJlIGNvZGVkIHRvIGJlIGFibGUgdG8gaGFuZGxlIGRpcmVjdG9yaWVzIHdo ZXJlIGNvb2tpZXMgY2FuIGJlY29tZSBpbnZhbGlkYXRlZC4gIFRoYXQncyB3aHkgcm0gYW5kIHN1 Y2ggaGF2ZSB0byBiZSBjb2RlZCB0byBrZWVwIHRyYWNrIG9mIHRoZWlyIGxvY2F0aW9uIGluIHRo ZSBkaXJlY3RvcnkgYnkgbW9yZSB0aGFuIGp1c3QgZF9vZmYuDQoNCgkJcHMNCg0KDQotLS0tLU9y aWdpbmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogbGludXgtbmZzLW93bmVyQHZnZXIua2VybmVsLm9y ZyBbbWFpbHRvOmxpbnV4LW5mcy1vd25lckB2Z2VyLmtlcm5lbC5vcmddIE9uIEJlaGFsZiBPZiBN YXR0IFcuIEJlbmphbWluDQpTZW50OiBNb25kYXksIE1hcmNoIDI2LCAyMDEyIDI6NTUgUE0NClRv OiBCb2F6IEhhcnJvc2gNCkNjOiBsaW51eC1uZnM7IEdhbmVzaGEgTkZTIExpc3Q7IFRyb25kIE15 a2xlYnVzdA0KU3ViamVjdDogUmU6IHVubGluayB3aXRoaW4gYW4gb3BlbiBkaXJlY3Rvcnkgc3Ry ZWFtDQoNCkhpLA0KDQpCb2F6OiAgd2UgZG8gbm90IGFueSBsb25nZXIgc2VuZCBhIHJlYWRkaXIg aW5kZXguICBXZSBkbyBzZW5kIGEgY29va2lldmVyZi4NCg0KRmlzdCBvZiBhbGwsIEkgaGF2ZW4n dCBlc3RhYmxpc2hlZCB0aGF0IHRoZSBpc3N1ZSB3ZSdyZSBhY3R1YWxseSBvYnNlcnZpbmcgaXMg Y2F1c2VkIGJ5IHRoZSBMaW51eCBjbGllbnQgc2VuZGluZyBvbGQgY29va2llcyB0byByZWFkZGly KHNvbWV0aGluZykuICBIb3dldmVyLCBpZiBpdCBpcywgaXQncyBpbiBubyB3YXkgYmV0dGVyIHRv IHRyeSB0byBtYWtlIGNvb2tpZXMgIm1vcmUgcGVyc2lzdGVudC4iICBOb3Igc2hvdWxkIHRoZSBM aW51eCBjbGllbnQgYmUgZXhwZWN0aW5nIGl0LiAgVGhlIGFzc3VtcHRpb24gaXMgc2ltcGx5IGZs YXdlZC4gIFRoZSBwcm90b2NvbCBpbnRyb2R1Y2VkIGNvb2tpZSB2ZXJpZmllciAoYSBMT05HIHRp bWUgYWdvKSBmb3IgYSByZWFzb24uDQoNCk1hdHQNCi0tLS0tICJCb2F6IEhhcnJvc2giIDxiaGFy cm9zaEBwYW5hc2FzLmNvbT4gd3JvdGU6DQoNCj4gT24gMDMvMjYvMjAxMiAxMToyNSBBTSwgTXlr bGVidXN0LCBUcm9uZCB3cm90ZToNCj4gDQo+ID4gT24gTW9uLCAyMDEyLTAzLTI2IGF0IDExOjE3 IC0wNzAwLCBCb2F6IEhhcnJvc2ggd3JvdGU6DQo+ID4+IE9uIDAzLzI0LzIwMTIgMTA6MTIgQU0s IE15a2xlYnVzdCwgVHJvbmQgd3JvdGU6DQo+ID4+DQo+ID4+DQo+ID4+IEl0J3MgdGhlIG5ldyAo cG9zdCBSSEVMIDYuMCBLZXJuZWwpIE5GUyBuZWVkIGZvciBvcGVuZGlyIGFmdGVyIGFuDQo+IHVu bGluay4NCj4gPiANCj4gPiBXaGF0Pw0KPiA+IA0KPiANCj4gDQo+IFRoZSBsaW5rcyBiZWxvdyBy ZXBvcnQgb24gcmVncmVzc2lvbiBpbiBSSEVMIDYuMiB3aGljaCBoYXZlIGEgbmV3IE5GUyANCj4g aW1wbGVtZW50YXRpb24gd2hlcmUgIDYuMCB1c2VkIHRvIGJlIGZpbmUuDQo+IChPciBpdCdzIHdo YXQgSSB1bmRlcnN0b29kIGZyb20gdGhlIHJlcG9ydGVycykNCj4gDQo+IDxzbmlwPg0KPiANCj4g PiANCj4gDQo+ID4gTm8uDQo+ID4gDQo+ID4gSWYgdGhlIHNlcnZlciBzdXBwb3J0cyBwZXJtYW5l bnQgcmVhZGRpciBjb29raWVzLCB0aGVuIHRoZXJlIGlzIG5vDQo+IG5lZWQNCj4gPiBmb3Igb3Bl bmRpciBhZnRlciB1bmxpbmsuDQo+ID4gDQo+ID4gSWYgdGhlIHNlcnZlciBkb2VzIG5vdCBoYXZl IHBlcm1hbmVudCByZWFkZGlyIGNvb2tpZXMsIHRoZW4gb3VyDQo+IGNsaWVudA0KPiA+IGhhcyBu ZXZlciBzdXBwb3J0ZWQgaXQgYW55d2F5IChub3Igd2lsbCBpdCBldmVyIGRvIHNvKS4gVGhhdCB3 aG9sZSANCj4gPiAnY29va2lldmVyZicgUkVBRERJUiBidWxsc2hpdCBoYXMgbmV2ZXIgcHJvdmlk ZWQgYSB3b3JrYWJsZSBtb2RlbA0KPiBmb3IgYQ0KPiA+IFBPU0lYIGNsaWVudC4uLg0KPiANCj4g DQo+IFRoYW5rcyBUcm9uZCBmb3IgdGhlIGV4cGxhbmF0aW9uLiBGb3JnaXZlIG15IHNsb3duZXNz LiBTbyB5b3UgYXJlIA0KPiBzYXlpbmcgdGhhdCB3aXRoIGtuZnNkIGl0J3MgYWN0dWFsbHkgZmls ZXN5c3RlbSBkZXBlbmRlbnQgYW5kIG1heWJlIA0KPiB0aGUgcmVwb3J0ZXJzIGRpZCBub3QgY29t cGFyZSBhcHBsZXMtdG8tYXBwbGVzIGFuZCB0aGUgZGlmZmVyZW5jZSBpcyANCj4gaW4gdGhlIGJl aGluZCBmaWxlIHN5c3RlbS4NCj4gDQo+IE1hdHQgSSdtIHN1cmUgVHJvbmQgaXMgcmlnaHQgd2l0 aCByZWdhcmQgdG8gR2FuZXNoYSwgYmVjYXVzZSB3ZSBzZW5kIGEgDQo+IHJlYWRkaXIgaW5kZXgg YW5kIGEgY29va2lldmVyZiwgd2hpY2ggd2lsbCBjaGFuZ2UgYWZ0ZXIgdW5saW5rLiBTaW5jZSAN Cj4gdGhlIGFwcGxpY2F0aW9uIGRvZXMgbm90IGNhbGwgb3BlbmRpciBhZ2FpbiB0aGUgY2xpZW50 IHdpbGwgc2VuZCB0aGUgDQo+IG9sZCBjb29raWVzLCB3aGljaCBpcyBub3cgYSBkaWZmZXJlbnQg ZmlsZSBvciBtaWdodCBnZXQgb3V0LW9mLWJvdW5kcyANCj4gZm9yIEdhbmVzaGEuDQo+IA0KPiBM ZXQgbWUgdGhpbmsgYWJvdXQgaXQgYSBiaXQuIEknbSBzdXJlIHdlIGNhbiBzZW5kIGEgQmV0dGVy IDY0Yml0IA0KPiBjb29raWUsIHdoaWNoIHdpbGwgYmUgbW9yZSBwZXJzaXN0ZW50LCBldmVuIGFj cm9zcyB1bmxpbmtzLg0KPiANCj4gPiANCj4gPiANCj4gDQo+IA0KPiBUaGFua3MNCj4gQm9heg0K DQotLQ0KTWF0dCBCZW5qYW1pbg0KVGhlIExpbnV4IEJveA0KMjA2IFNvdXRoIEZpZnRoIEF2ZS4g U3VpdGUgMTUwDQpBbm4gQXJib3IsIE1JICA0ODEwNA0KDQpodHRwOi8vbGludXhib3guY29tDQoN CnRlbC4gNzM0LTc2MS00Njg5DQpmYXguIDczNC03NjktODkzOA0KY2VsLiA3MzQtMjE2LTUzMDkN Ci0tDQpUbyB1bnN1YnNjcmliZSBmcm9tIHRoaXMgbGlzdDogc2VuZCB0aGUgbGluZSAidW5zdWJz Y3JpYmUgbGludXgtbmZzIiBpbiB0aGUgYm9keSBvZiBhIG1lc3NhZ2UgdG8gbWFqb3Jkb21vQHZn ZXIua2VybmVsLm9yZyBNb3JlIG1ham9yZG9tbyBpbmZvIGF0ICBodHRwOi8vdmdlci5rZXJuZWwu b3JnL21ham9yZG9tby1pbmZvLmh0bWwNCg== ^ permalink raw reply [flat|nested] 19+ messages in thread
* RE: unlink within an open directory stream 2012-03-30 15:24 ` Peter Staubach @ 2012-03-30 15:46 ` Myklebust, Trond 2012-03-30 16:46 ` Peter Staubach 2012-03-30 17:10 ` Matt W. Benjamin 0 siblings, 2 replies; 19+ messages in thread From: Myklebust, Trond @ 2012-03-30 15:46 UTC (permalink / raw) To: Peter Staubach Cc: Matt W. Benjamin, Boaz Harrosh, linux-nfs, Ganesha NFS List T24gRnJpLCAyMDEyLTAzLTMwIGF0IDExOjI0IC0wNDAwLCBQZXRlciBTdGF1YmFjaCB3cm90ZToN Cj4gWWVzLCBobW1tLCB1bW1tLi4uDQo+IA0KPiBBIGxvbmcgdGltZSwgSSB0aG91Z2h0IHRoYXQg dGhlIHJlYWRkaXIgY29va2llIHZlcmlmaWVyIHNvdW5kZWQgbGlrZSBhIGdvb2QgaWRlYS4gIEFm dGVyIGFsbCwgaWYgZGlyZWN0b3JpZXMgY2FuIGJlIGNvbXBhY3RlZCwgaXQgd291bGQgYmUgbmlj ZSB0byBiZSBhYmxlIHRvIHRlbGwgd2hlbiB0aGUgY29va2llIGhhcyBiZWNvbWUgaW52YWxpZCwg cmlnaHQ/ICBTbywgSSBhZGRlZCBpdCB0byB0aGUgTkZTdjMgcHJvdG9jb2wuDQo+IA0KPiBJdCB3 YXMgb25seSBhZnRlcndhcmRzLCBpbiBhdHRlbXB0aW5nIHRvIGltcGxlbWVudCBpdCwgdGhhdCB0 aGUgcmVhbCB0cnV0aCB3YXMgZGlzY292ZXJlZC4gIFRoaXMgaXMgdGhhdCB0aGUgaW50ZXJmYWNl cyBhcmUgbm90IHN1ZmZpY2llbnQgdG8gYmUgYWJsZSB0byBjb252ZXkgdGhpcyBpbmZvcm1hdGlv biBzdWZmaWNpZW50bHkgYW5kIHRvIGJlIGFibGUgdG8gcmVjb3ZlciBmcm9tIHRoZSBzaXR1YXRp b24uICBDdXJyZW50bHksIHJlY292ZXJ5IHJlcXVpcmUgYWN0aW9uIG9uIHRoZSBwYXJ0IG9mIHRo ZSBhcHBsaWNhdGlvbi4NCj4gDQo+IEluIHJldHJvc3BlY3QsIEkgd291bGRuJ3QgYWRkIHRoZSBj b29raWUgdmVyaWZpZXIgaWYgSSB3YXMgZG9pbmcgaXQgYWdhaW4uDQo+IA0KPiBUaGFua3MgZm9y IHRoZSBzdXBwb3J0LCB0aG91Z2guICA6LSkNCj4gDQo+IEkgZG8gZGlzYWdyZWUgd2l0aCBUcm9u ZCB0aG91Z2gsIGluIHRoYXQgYXBwbGljYXRpb25zIG11c3QgYmUgY29kZWQgdG8gYmUgYWJsZSB0 byBoYW5kbGUgZGlyZWN0b3JpZXMgd2hlcmUgY29va2llcyBjYW4gYmVjb21lIGludmFsaWRhdGVk LiAgVGhhdCdzIHdoeSBybSBhbmQgc3VjaCBoYXZlIHRvIGJlIGNvZGVkIHRvIGtlZXAgdHJhY2sg b2YgdGhlaXIgbG9jYXRpb24gaW4gdGhlIGRpcmVjdG9yeSBieSBtb3JlIHRoYW4ganVzdCBkX29m Zi4NCg0KVGhlIHByb2JsZW0gd2l0aCB0aGF0IGlzIHRoYXQgdGhlcmUgaXMgbm8gc3RhbmRhcmQg Zm9yIGhvdyBhcHBsaWNhdGlvbnMNCnNob3VsZCBwcm9jZWVkIHRvIGRvIHRoaXMsIGFuZCBzbyB0 aGVyZSBhcmUgbm8gZ3VpZGVsaW5lcyBmb3IgaG93IHRoZQ0KY2xpZW50IHNob3VsZCBiZWhhdmUu DQoNCldvcnNlLCBpZiB5b3UndmUgbG9va2VkIGF0IHRoZSBnbGliYyByZWFkZGlyIGxpYnJhcnkg Y29kZSwgeW91J2xsIGhhdmUNCm5vdGljZWQgdGhhdCBpdCBfZGVwZW5kc18gb24gYmVpbmcgYWJs ZSB0byB1c2UgYSBzdHJpY3RseQ0KUE9TSVgtY29uZm9ybWluZyB2ZXJzaW9uIG9mIHRlbGxkaXIo KS9zZWVrZGlyKCkuIElmIGl0IGNhbid0IHJld2luZCB0byBhDQpwcmV2aW91c2x5IHNhdmVkIG9m ZnNldCwgdGhlbiBpdCBjYW4gZW5kIHVwIHNraXBwaW5nIGVudHJpZXMuDQoNClNvIG5vLCBJIGNh bid0IHJlbHkgb24gYXBwbGljYXRpb25zIGVpdGhlci4uLg0KDQo+IAkJcHMNCj4gDQo+IA0KPiAt LS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBsaW51eC1uZnMtb3duZXJAdmdlci5r ZXJuZWwub3JnIFttYWlsdG86bGludXgtbmZzLW93bmVyQHZnZXIua2VybmVsLm9yZ10gT24gQmVo YWxmIE9mIE1hdHQgVy4gQmVuamFtaW4NCj4gU2VudDogTW9uZGF5LCBNYXJjaCAyNiwgMjAxMiAy OjU1IFBNDQo+IFRvOiBCb2F6IEhhcnJvc2gNCj4gQ2M6IGxpbnV4LW5mczsgR2FuZXNoYSBORlMg TGlzdDsgVHJvbmQgTXlrbGVidXN0DQo+IFN1YmplY3Q6IFJlOiB1bmxpbmsgd2l0aGluIGFuIG9w ZW4gZGlyZWN0b3J5IHN0cmVhbQ0KPiANCj4gSGksDQo+IA0KPiBCb2F6OiAgd2UgZG8gbm90IGFu eSBsb25nZXIgc2VuZCBhIHJlYWRkaXIgaW5kZXguICBXZSBkbyBzZW5kIGEgY29va2lldmVyZi4N Cj4gDQo+IEZpc3Qgb2YgYWxsLCBJIGhhdmVuJ3QgZXN0YWJsaXNoZWQgdGhhdCB0aGUgaXNzdWUg d2UncmUgYWN0dWFsbHkgb2JzZXJ2aW5nIGlzIGNhdXNlZCBieSB0aGUgTGludXggY2xpZW50IHNl bmRpbmcgb2xkIGNvb2tpZXMgdG8gcmVhZGRpcihzb21ldGhpbmcpLiAgSG93ZXZlciwgaWYgaXQg aXMsIGl0J3MgaW4gbm8gd2F5IGJldHRlciB0byB0cnkgdG8gbWFrZSBjb29raWVzICJtb3JlIHBl cnNpc3RlbnQuIiAgTm9yIHNob3VsZCB0aGUgTGludXggY2xpZW50IGJlIGV4cGVjdGluZyBpdC4g IFRoZSBhc3N1bXB0aW9uIGlzIHNpbXBseSBmbGF3ZWQuICBUaGUgcHJvdG9jb2wgaW50cm9kdWNl ZCBjb29raWUgdmVyaWZpZXIgKGEgTE9ORyB0aW1lIGFnbykgZm9yIGEgcmVhc29uLg0KPiANCj4g TWF0dA0KPiAtLS0tLSAiQm9heiBIYXJyb3NoIiA8YmhhcnJvc2hAcGFuYXNhcy5jb20+IHdyb3Rl Og0KPiANCj4gPiBPbiAwMy8yNi8yMDEyIDExOjI1IEFNLCBNeWtsZWJ1c3QsIFRyb25kIHdyb3Rl Og0KPiA+IA0KPiA+ID4gT24gTW9uLCAyMDEyLTAzLTI2IGF0IDExOjE3IC0wNzAwLCBCb2F6IEhh cnJvc2ggd3JvdGU6DQo+ID4gPj4gT24gMDMvMjQvMjAxMiAxMDoxMiBBTSwgTXlrbGVidXN0LCBU cm9uZCB3cm90ZToNCj4gPiA+Pg0KPiA+ID4+DQo+ID4gPj4gSXQncyB0aGUgbmV3IChwb3N0IFJI RUwgNi4wIEtlcm5lbCkgTkZTIG5lZWQgZm9yIG9wZW5kaXIgYWZ0ZXIgYW4NCj4gPiB1bmxpbmsu DQo+ID4gPiANCj4gPiA+IFdoYXQ/DQo+ID4gPiANCj4gPiANCj4gPiANCj4gPiBUaGUgbGlua3Mg YmVsb3cgcmVwb3J0IG9uIHJlZ3Jlc3Npb24gaW4gUkhFTCA2LjIgd2hpY2ggaGF2ZSBhIG5ldyBO RlMgDQo+ID4gaW1wbGVtZW50YXRpb24gd2hlcmUgIDYuMCB1c2VkIHRvIGJlIGZpbmUuDQo+ID4g KE9yIGl0J3Mgd2hhdCBJIHVuZGVyc3Rvb2QgZnJvbSB0aGUgcmVwb3J0ZXJzKQ0KPiA+IA0KPiA+ IDxzbmlwPg0KPiA+IA0KPiA+ID4gDQo+ID4gDQo+ID4gPiBOby4NCj4gPiA+IA0KPiA+ID4gSWYg dGhlIHNlcnZlciBzdXBwb3J0cyBwZXJtYW5lbnQgcmVhZGRpciBjb29raWVzLCB0aGVuIHRoZXJl IGlzIG5vDQo+ID4gbmVlZA0KPiA+ID4gZm9yIG9wZW5kaXIgYWZ0ZXIgdW5saW5rLg0KPiA+ID4g DQo+ID4gPiBJZiB0aGUgc2VydmVyIGRvZXMgbm90IGhhdmUgcGVybWFuZW50IHJlYWRkaXIgY29v a2llcywgdGhlbiBvdXINCj4gPiBjbGllbnQNCj4gPiA+IGhhcyBuZXZlciBzdXBwb3J0ZWQgaXQg YW55d2F5IChub3Igd2lsbCBpdCBldmVyIGRvIHNvKS4gVGhhdCB3aG9sZSANCj4gPiA+ICdjb29r aWV2ZXJmJyBSRUFERElSIGJ1bGxzaGl0IGhhcyBuZXZlciBwcm92aWRlZCBhIHdvcmthYmxlIG1v ZGVsDQo+ID4gZm9yIGENCj4gPiA+IFBPU0lYIGNsaWVudC4uLg0KPiA+IA0KPiA+IA0KPiA+IFRo YW5rcyBUcm9uZCBmb3IgdGhlIGV4cGxhbmF0aW9uLiBGb3JnaXZlIG15IHNsb3duZXNzLiBTbyB5 b3UgYXJlIA0KPiA+IHNheWluZyB0aGF0IHdpdGgga25mc2QgaXQncyBhY3R1YWxseSBmaWxlc3lz dGVtIGRlcGVuZGVudCBhbmQgbWF5YmUgDQo+ID4gdGhlIHJlcG9ydGVycyBkaWQgbm90IGNvbXBh cmUgYXBwbGVzLXRvLWFwcGxlcyBhbmQgdGhlIGRpZmZlcmVuY2UgaXMgDQo+ID4gaW4gdGhlIGJl aGluZCBmaWxlIHN5c3RlbS4NCj4gPiANCj4gPiBNYXR0IEknbSBzdXJlIFRyb25kIGlzIHJpZ2h0 IHdpdGggcmVnYXJkIHRvIEdhbmVzaGEsIGJlY2F1c2Ugd2Ugc2VuZCBhIA0KPiA+IHJlYWRkaXIg aW5kZXggYW5kIGEgY29va2lldmVyZiwgd2hpY2ggd2lsbCBjaGFuZ2UgYWZ0ZXIgdW5saW5rLiBT aW5jZSANCj4gPiB0aGUgYXBwbGljYXRpb24gZG9lcyBub3QgY2FsbCBvcGVuZGlyIGFnYWluIHRo ZSBjbGllbnQgd2lsbCBzZW5kIHRoZSANCj4gPiBvbGQgY29va2llcywgd2hpY2ggaXMgbm93IGEg ZGlmZmVyZW50IGZpbGUgb3IgbWlnaHQgZ2V0IG91dC1vZi1ib3VuZHMgDQo+ID4gZm9yIEdhbmVz aGEuDQo+ID4gDQo+ID4gTGV0IG1lIHRoaW5rIGFib3V0IGl0IGEgYml0LiBJJ20gc3VyZSB3ZSBj YW4gc2VuZCBhIEJldHRlciA2NGJpdCANCj4gPiBjb29raWUsIHdoaWNoIHdpbGwgYmUgbW9yZSBw ZXJzaXN0ZW50LCBldmVuIGFjcm9zcyB1bmxpbmtzLg0KPiA+IA0KPiA+ID4gDQo+ID4gPiANCj4g PiANCj4gPiANCj4gPiBUaGFua3MNCj4gPiBCb2F6DQo+IA0KPiAtLQ0KPiBNYXR0IEJlbmphbWlu DQo+IFRoZSBMaW51eCBCb3gNCj4gMjA2IFNvdXRoIEZpZnRoIEF2ZS4gU3VpdGUgMTUwDQo+IEFu biBBcmJvciwgTUkgIDQ4MTA0DQo+IA0KPiBodHRwOi8vbGludXhib3guY29tDQo+IA0KPiB0ZWwu IDczNC03NjEtNDY4OQ0KPiBmYXguIDczNC03NjktODkzOA0KPiBjZWwuIDczNC0yMTYtNTMwOQ0K PiAtLQ0KPiBUbyB1bnN1YnNjcmliZSBmcm9tIHRoaXMgbGlzdDogc2VuZCB0aGUgbGluZSAidW5z dWJzY3JpYmUgbGludXgtbmZzIiBpbiB0aGUgYm9keSBvZiBhIG1lc3NhZ2UgdG8gbWFqb3Jkb21v QHZnZXIua2VybmVsLm9yZyBNb3JlIG1ham9yZG9tbyBpbmZvIGF0ICBodHRwOi8vdmdlci5rZXJu ZWwub3JnL21ham9yZG9tby1pbmZvLmh0bWwNCg0KLS0gDQpUcm9uZCBNeWtsZWJ1c3QNCkxpbnV4 IE5GUyBjbGllbnQgbWFpbnRhaW5lcg0KDQpOZXRBcHANClRyb25kLk15a2xlYnVzdEBuZXRhcHAu Y29tDQp3d3cubmV0YXBwLmNvbQ0KDQo= ^ permalink raw reply [flat|nested] 19+ messages in thread
* RE: unlink within an open directory stream 2012-03-30 15:46 ` Myklebust, Trond @ 2012-03-30 16:46 ` Peter Staubach 2012-03-30 17:10 ` Matt W. Benjamin 1 sibling, 0 replies; 19+ messages in thread From: Peter Staubach @ 2012-03-30 16:46 UTC (permalink / raw) To: Myklebust, Trond Cc: Matt W. Benjamin, Boaz Harrosh, linux-nfs, Ganesha NFS List UHJlY2lzZWx5LCBUcm9uZCEgIDotKQ0KDQpUaGF0J3Mgd2h5IHRoZSBjb29raWUgdmVyaWZpZXIg dHVybmVkIG91dCB0byBiZSBhIHVudGVuYWJsZSBpZGVhLg0KDQpUaGUgYXBwbGljYXRpb24gaXRz ZWxmLCBub3QganVzdCB0aGUgYXBwbGljYXRpb24gbGV2ZWwsIG11c3QgcGFydGljaXBhdGUgaW4g dGhlIHJlY292ZXJ5LiAgSWYgdGhlIHJtIGNvbW1hbmQgd2FudHMgdG8gZW5zdXJlIHRoYXQgaXQg cmVtb3ZlcyBldmVyeSBlbnRyeSBpbiBhIGRpcmVjdG9yeSBiZWZvcmUgYXR0ZW1wdGluZyB0byBy ZW1vdmUgdGhlIGRpcmVjdG9yeSBpdHNlbGYsIHRoZW4gaXQgbXVzdCBhcnJhbmdlIHRvIGRvIHNv IGFuZCBjYW4ndCByZWFsbHkgYXNzdW1lIHRoYXQgZGlyZWN0b3J5IGNvb2tpZXMgY2FuIG5ldmVy IGJlY29tZSBpbnZhbGlkLiAgUGVyaGFwcyBpdCBjb3VsZCBvbiBhIHB1cmVseSBQT1NJWCBzeXN0 ZW0sIEkgZHVubm8sIGJ1dCBJIGFtIG5vdCBzdXJlIHRoYXQgYSBwdXJlIFBPU0lYIHN5c3RlbSBl eGlzdHMgYW55d2hlcmUuDQoNCgkJcHMNCg0KDQotLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0K RnJvbTogTXlrbGVidXN0LCBUcm9uZCBbbWFpbHRvOlRyb25kLk15a2xlYnVzdEBuZXRhcHAuY29t XSANClNlbnQ6IEZyaWRheSwgTWFyY2ggMzAsIDIwMTIgMTE6NDcgQU0NClRvOiBQZXRlciBTdGF1 YmFjaA0KQ2M6IE1hdHQgVy4gQmVuamFtaW47IEJvYXogSGFycm9zaDsgbGludXgtbmZzOyBHYW5l c2hhIE5GUyBMaXN0DQpTdWJqZWN0OiBSRTogdW5saW5rIHdpdGhpbiBhbiBvcGVuIGRpcmVjdG9y eSBzdHJlYW0NCg0KT24gRnJpLCAyMDEyLTAzLTMwIGF0IDExOjI0IC0wNDAwLCBQZXRlciBTdGF1 YmFjaCB3cm90ZToNCj4gWWVzLCBobW1tLCB1bW1tLi4uDQo+IA0KPiBBIGxvbmcgdGltZSwgSSB0 aG91Z2h0IHRoYXQgdGhlIHJlYWRkaXIgY29va2llIHZlcmlmaWVyIHNvdW5kZWQgbGlrZSBhIGdv b2QgaWRlYS4gIEFmdGVyIGFsbCwgaWYgZGlyZWN0b3JpZXMgY2FuIGJlIGNvbXBhY3RlZCwgaXQg d291bGQgYmUgbmljZSB0byBiZSBhYmxlIHRvIHRlbGwgd2hlbiB0aGUgY29va2llIGhhcyBiZWNv bWUgaW52YWxpZCwgcmlnaHQ/ICBTbywgSSBhZGRlZCBpdCB0byB0aGUgTkZTdjMgcHJvdG9jb2wu DQo+IA0KPiBJdCB3YXMgb25seSBhZnRlcndhcmRzLCBpbiBhdHRlbXB0aW5nIHRvIGltcGxlbWVu dCBpdCwgdGhhdCB0aGUgcmVhbCB0cnV0aCB3YXMgZGlzY292ZXJlZC4gIFRoaXMgaXMgdGhhdCB0 aGUgaW50ZXJmYWNlcyBhcmUgbm90IHN1ZmZpY2llbnQgdG8gYmUgYWJsZSB0byBjb252ZXkgdGhp cyBpbmZvcm1hdGlvbiBzdWZmaWNpZW50bHkgYW5kIHRvIGJlIGFibGUgdG8gcmVjb3ZlciBmcm9t IHRoZSBzaXR1YXRpb24uICBDdXJyZW50bHksIHJlY292ZXJ5IHJlcXVpcmUgYWN0aW9uIG9uIHRo ZSBwYXJ0IG9mIHRoZSBhcHBsaWNhdGlvbi4NCj4gDQo+IEluIHJldHJvc3BlY3QsIEkgd291bGRu J3QgYWRkIHRoZSBjb29raWUgdmVyaWZpZXIgaWYgSSB3YXMgZG9pbmcgaXQgYWdhaW4uDQo+IA0K PiBUaGFua3MgZm9yIHRoZSBzdXBwb3J0LCB0aG91Z2guICA6LSkNCj4gDQo+IEkgZG8gZGlzYWdy ZWUgd2l0aCBUcm9uZCB0aG91Z2gsIGluIHRoYXQgYXBwbGljYXRpb25zIG11c3QgYmUgY29kZWQg dG8gYmUgYWJsZSB0byBoYW5kbGUgZGlyZWN0b3JpZXMgd2hlcmUgY29va2llcyBjYW4gYmVjb21l IGludmFsaWRhdGVkLiAgVGhhdCdzIHdoeSBybSBhbmQgc3VjaCBoYXZlIHRvIGJlIGNvZGVkIHRv IGtlZXAgdHJhY2sgb2YgdGhlaXIgbG9jYXRpb24gaW4gdGhlIGRpcmVjdG9yeSBieSBtb3JlIHRo YW4ganVzdCBkX29mZi4NCg0KDQpUaGUgcHJvYmxlbSB3aXRoIHRoYXQgaXMgdGhhdCB0aGVyZSBp cyBubyBzdGFuZGFyZCBmb3IgaG93IGFwcGxpY2F0aW9ucyBzaG91bGQgcHJvY2VlZCB0byBkbyB0 aGlzLCBhbmQgc28gdGhlcmUgYXJlIG5vIGd1aWRlbGluZXMgZm9yIGhvdyB0aGUgY2xpZW50IHNo b3VsZCBiZWhhdmUuDQoNCldvcnNlLCBpZiB5b3UndmUgbG9va2VkIGF0IHRoZSBnbGliYyByZWFk ZGlyIGxpYnJhcnkgY29kZSwgeW91J2xsIGhhdmUgbm90aWNlZCB0aGF0IGl0IF9kZXBlbmRzXyBv biBiZWluZyBhYmxlIHRvIHVzZSBhIHN0cmljdGx5IFBPU0lYLWNvbmZvcm1pbmcgdmVyc2lvbiBv ZiB0ZWxsZGlyKCkvc2Vla2RpcigpLiBJZiBpdCBjYW4ndCByZXdpbmQgdG8gYSBwcmV2aW91c2x5 IHNhdmVkIG9mZnNldCwgdGhlbiBpdCBjYW4gZW5kIHVwIHNraXBwaW5nIGVudHJpZXMuDQoNClNv IG5vLCBJIGNhbid0IHJlbHkgb24gYXBwbGljYXRpb25zIGVpdGhlci4uLg0KDQo+IAkJcHMNCj4g DQo+IA0KPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBsaW51eC1uZnMtb3du ZXJAdmdlci5rZXJuZWwub3JnIA0KPiBbbWFpbHRvOmxpbnV4LW5mcy1vd25lckB2Z2VyLmtlcm5l bC5vcmddIE9uIEJlaGFsZiBPZiBNYXR0IFcuIEJlbmphbWluDQo+IFNlbnQ6IE1vbmRheSwgTWFy Y2ggMjYsIDIwMTIgMjo1NSBQTQ0KPiBUbzogQm9heiBIYXJyb3NoDQo+IENjOiBsaW51eC1uZnM7 IEdhbmVzaGEgTkZTIExpc3Q7IFRyb25kIE15a2xlYnVzdA0KPiBTdWJqZWN0OiBSZTogdW5saW5r IHdpdGhpbiBhbiBvcGVuIGRpcmVjdG9yeSBzdHJlYW0NCj4gDQo+IEhpLA0KPiANCj4gQm9hejog IHdlIGRvIG5vdCBhbnkgbG9uZ2VyIHNlbmQgYSByZWFkZGlyIGluZGV4LiAgV2UgZG8gc2VuZCBh IGNvb2tpZXZlcmYuDQo+IA0KPiBGaXN0IG9mIGFsbCwgSSBoYXZlbid0IGVzdGFibGlzaGVkIHRo YXQgdGhlIGlzc3VlIHdlJ3JlIGFjdHVhbGx5IG9ic2VydmluZyBpcyBjYXVzZWQgYnkgdGhlIExp bnV4IGNsaWVudCBzZW5kaW5nIG9sZCBjb29raWVzIHRvIHJlYWRkaXIoc29tZXRoaW5nKS4gIEhv d2V2ZXIsIGlmIGl0IGlzLCBpdCdzIGluIG5vIHdheSBiZXR0ZXIgdG8gdHJ5IHRvIG1ha2UgY29v a2llcyAibW9yZSBwZXJzaXN0ZW50LiIgIE5vciBzaG91bGQgdGhlIExpbnV4IGNsaWVudCBiZSBl eHBlY3RpbmcgaXQuICBUaGUgYXNzdW1wdGlvbiBpcyBzaW1wbHkgZmxhd2VkLiAgVGhlIHByb3Rv Y29sIGludHJvZHVjZWQgY29va2llIHZlcmlmaWVyIChhIExPTkcgdGltZSBhZ28pIGZvciBhIHJl YXNvbi4NCj4gDQo+IE1hdHQNCj4gLS0tLS0gIkJvYXogSGFycm9zaCIgPGJoYXJyb3NoQHBhbmFz YXMuY29tPiB3cm90ZToNCj4gDQo+ID4gT24gMDMvMjYvMjAxMiAxMToyNSBBTSwgTXlrbGVidXN0 LCBUcm9uZCB3cm90ZToNCj4gPiANCj4gPiA+IE9uIE1vbiwgMjAxMi0wMy0yNiBhdCAxMToxNyAt MDcwMCwgQm9heiBIYXJyb3NoIHdyb3RlOg0KPiA+ID4+IE9uIDAzLzI0LzIwMTIgMTA6MTIgQU0s IE15a2xlYnVzdCwgVHJvbmQgd3JvdGU6DQo+ID4gPj4NCj4gPiA+Pg0KPiA+ID4+IEl0J3MgdGhl IG5ldyAocG9zdCBSSEVMIDYuMCBLZXJuZWwpIE5GUyBuZWVkIGZvciBvcGVuZGlyIGFmdGVyIGFu DQo+ID4gdW5saW5rLg0KPiA+ID4gDQo+ID4gPiBXaGF0Pw0KPiA+ID4gDQo+ID4gDQo+ID4gDQo+ ID4gVGhlIGxpbmtzIGJlbG93IHJlcG9ydCBvbiByZWdyZXNzaW9uIGluIFJIRUwgNi4yIHdoaWNo IGhhdmUgYSBuZXcgDQo+ID4gTkZTIGltcGxlbWVudGF0aW9uIHdoZXJlICA2LjAgdXNlZCB0byBi ZSBmaW5lLg0KPiA+IChPciBpdCdzIHdoYXQgSSB1bmRlcnN0b29kIGZyb20gdGhlIHJlcG9ydGVy cykNCj4gPiANCj4gPiA8c25pcD4NCj4gPiANCj4gPiA+IA0KPiA+IA0KPiA+ID4gTm8uDQo+ID4g PiANCj4gPiA+IElmIHRoZSBzZXJ2ZXIgc3VwcG9ydHMgcGVybWFuZW50IHJlYWRkaXIgY29va2ll cywgdGhlbiB0aGVyZSBpcyBubw0KPiA+IG5lZWQNCj4gPiA+IGZvciBvcGVuZGlyIGFmdGVyIHVu bGluay4NCj4gPiA+IA0KPiA+ID4gSWYgdGhlIHNlcnZlciBkb2VzIG5vdCBoYXZlIHBlcm1hbmVu dCByZWFkZGlyIGNvb2tpZXMsIHRoZW4gb3VyDQo+ID4gY2xpZW50DQo+ID4gPiBoYXMgbmV2ZXIg c3VwcG9ydGVkIGl0IGFueXdheSAobm9yIHdpbGwgaXQgZXZlciBkbyBzbykuIFRoYXQgd2hvbGUg DQo+ID4gPiAnY29va2lldmVyZicgUkVBRERJUiBidWxsc2hpdCBoYXMgbmV2ZXIgcHJvdmlkZWQg YSB3b3JrYWJsZSBtb2RlbA0KPiA+IGZvciBhDQo+ID4gPiBQT1NJWCBjbGllbnQuLi4NCj4gPiAN Cj4gPiANCj4gPiBUaGFua3MgVHJvbmQgZm9yIHRoZSBleHBsYW5hdGlvbi4gRm9yZ2l2ZSBteSBz bG93bmVzcy4gU28geW91IGFyZSANCj4gPiBzYXlpbmcgdGhhdCB3aXRoIGtuZnNkIGl0J3MgYWN0 dWFsbHkgZmlsZXN5c3RlbSBkZXBlbmRlbnQgYW5kIG1heWJlIA0KPiA+IHRoZSByZXBvcnRlcnMg ZGlkIG5vdCBjb21wYXJlIGFwcGxlcy10by1hcHBsZXMgYW5kIHRoZSBkaWZmZXJlbmNlIGlzIA0K PiA+IGluIHRoZSBiZWhpbmQgZmlsZSBzeXN0ZW0uDQo+ID4gDQo+ID4gTWF0dCBJJ20gc3VyZSBU cm9uZCBpcyByaWdodCB3aXRoIHJlZ2FyZCB0byBHYW5lc2hhLCBiZWNhdXNlIHdlIHNlbmQgDQo+ ID4gYSByZWFkZGlyIGluZGV4IGFuZCBhIGNvb2tpZXZlcmYsIHdoaWNoIHdpbGwgY2hhbmdlIGFm dGVyIHVubGluay4gDQo+ID4gU2luY2UgdGhlIGFwcGxpY2F0aW9uIGRvZXMgbm90IGNhbGwgb3Bl bmRpciBhZ2FpbiB0aGUgY2xpZW50IHdpbGwgDQo+ID4gc2VuZCB0aGUgb2xkIGNvb2tpZXMsIHdo aWNoIGlzIG5vdyBhIGRpZmZlcmVudCBmaWxlIG9yIG1pZ2h0IGdldCANCj4gPiBvdXQtb2YtYm91 bmRzIGZvciBHYW5lc2hhLg0KPiA+IA0KPiA+IExldCBtZSB0aGluayBhYm91dCBpdCBhIGJpdC4g SSdtIHN1cmUgd2UgY2FuIHNlbmQgYSBCZXR0ZXIgNjRiaXQgDQo+ID4gY29va2llLCB3aGljaCB3 aWxsIGJlIG1vcmUgcGVyc2lzdGVudCwgZXZlbiBhY3Jvc3MgdW5saW5rcy4NCj4gPiANCj4gPiA+ IA0KPiA+ID4gDQo+ID4gDQo+ID4gDQo+ID4gVGhhbmtzDQo+ID4gQm9heg0KPiANCj4gLS0NCj4g TWF0dCBCZW5qYW1pbg0KPiBUaGUgTGludXggQm94DQo+IDIwNiBTb3V0aCBGaWZ0aCBBdmUuIFN1 aXRlIDE1MA0KPiBBbm4gQXJib3IsIE1JICA0ODEwNA0KPiANCj4gaHR0cDovL2xpbnV4Ym94LmNv bQ0KPiANCj4gdGVsLiA3MzQtNzYxLTQ2ODkNCj4gZmF4LiA3MzQtNzY5LTg5MzgNCj4gY2VsLiA3 MzQtMjE2LTUzMDkNCj4gLS0NCj4gVG8gdW5zdWJzY3JpYmUgZnJvbSB0aGlzIGxpc3Q6IHNlbmQg dGhlIGxpbmUgInVuc3Vic2NyaWJlIGxpbnV4LW5mcyIgDQo+IGluIHRoZSBib2R5IG9mIGEgbWVz c2FnZSB0byBtYWpvcmRvbW9Admdlci5rZXJuZWwub3JnIE1vcmUgbWFqb3Jkb21vIA0KPiBpbmZv IGF0ICBodHRwOi8vdmdlci5rZXJuZWwub3JnL21ham9yZG9tby1pbmZvLmh0bWwNCg0KLS0NClRy b25kIE15a2xlYnVzdA0KTGludXggTkZTIGNsaWVudCBtYWludGFpbmVyDQoNCk5ldEFwcA0KVHJv bmQuTXlrbGVidXN0QG5ldGFwcC5jb20NCnd3dy5uZXRhcHAuY29tDQoNCg== ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: unlink within an open directory stream 2012-03-30 15:46 ` Myklebust, Trond 2012-03-30 16:46 ` Peter Staubach @ 2012-03-30 17:10 ` Matt W. Benjamin 1 sibling, 0 replies; 19+ messages in thread From: Matt W. Benjamin @ 2012-03-30 17:10 UTC (permalink / raw) To: Trond Myklebust; +Cc: Boaz Harrosh, linux-nfs, Ganesha NFS List, Peter Staubach Hi, Thanks for providing all this reasoning and additional background. I guess I've always constructed an imaginary intention of the protocol feature that's different from the actual one. Yet the feature nevertheless seemed useful. There seem to be two parts to my construction: 1. negatively, it never occurred to me that a client implementation would assume that it could protect an application from invalidation, unless it was able to perform, and cache a complete traversal--this would be one with a consistent verifier for all component readdirs 2. at the same time, it seems a conforming server implementation can be made which would often support a client in doing this, presuming it was able to use the cookie verifier to select a prior version of the directory; not only do some filesystems now appear able to do this, this technique is used by the dcache nfs server, though iiuc, dcache doesn't make any promises that it will have a specific component cached at every version, etc Matt ----- "Trond Myklebust" <Trond.Myklebust@netapp.com> wrote: > On Fri, 2012-03-30 at 11:24 -0400, Peter Staubach wrote: > > Yes, hmmm, ummm... > > > > A long time, I thought that the readdir cookie verifier sounded like > a good idea. After all, if directories can be compacted, it would be > nice to be able to tell when the cookie has become invalid, right? > So, I added it to the NFSv3 protocol. > > > > It was only afterwards, in attempting to implement it, that the real > truth was discovered. This is that the interfaces are not sufficient > to be able to convey this information sufficiently and to be able to > recover from the situation. Currently, recovery require action on the > part of the application. > > > > In retrospect, I wouldn't add the cookie verifier if I was doing it > again. > > > > Thanks for the support, though. :-) > > > > I do disagree with Trond though, in that applications must be coded > to be able to handle directories where cookies can become invalidated. > That's why rm and such have to be coded to keep track of their > location in the directory by more than just d_off. > > The problem with that is that there is no standard for how > applications > should proceed to do this, and so there are no guidelines for how the > client should behave. > > Worse, if you've looked at the glibc readdir library code, you'll > have > noticed that it _depends_ on being able to use a strictly > POSIX-conforming version of telldir()/seekdir(). If it can't rewind to > a > previously saved offset, then it can end up skipping entries. > > So no, I can't rely on applications either... > > > ps > > > > > > -----Original Message----- > > From: linux-nfs-owner@vger.kernel.org > [mailto:linux-nfs-owner@vger.kernel.org] On Behalf Of Matt W. > Benjamin > > Sent: Monday, March 26, 2012 2:55 PM > > To: Boaz Harrosh > > Cc: linux-nfs; Ganesha NFS List; Trond Myklebust > > Subject: Re: unlink within an open directory stream > > > > Hi, > > > > Boaz: we do not any longer send a readdir index. We do send a > cookieverf. > > > > Fist of all, I haven't established that the issue we're actually > observing is caused by the Linux client sending old cookies to > readdir(something). However, if it is, it's in no way better to try > to make cookies "more persistent." Nor should the Linux client be > expecting it. The assumption is simply flawed. The protocol > introduced cookie verifier (a LONG time ago) for a reason. > > > > Matt > > ----- "Boaz Harrosh" <bharrosh@panasas.com> wrote: > > > > > On 03/26/2012 11:25 AM, Myklebust, Trond wrote: > > > > > > > On Mon, 2012-03-26 at 11:17 -0700, Boaz Harrosh wrote: > > > >> On 03/24/2012 10:12 AM, Myklebust, Trond wrote: > > > >> > > > >> > > > >> It's the new (post RHEL 6.0 Kernel) NFS need for opendir after > an > > > unlink. > > > > > > > > What? > > > > > > > > > > > > > The links below report on regression in RHEL 6.2 which have a new > NFS > > > implementation where 6.0 used to be fine. > > > (Or it's what I understood from the reporters) > > > > > > <snip> > > > > > > > > > > > > > > No. > > > > > > > > If the server supports permanent readdir cookies, then there is > no > > > need > > > > for opendir after unlink. > > > > > > > > If the server does not have permanent readdir cookies, then our > > > client > > > > has never supported it anyway (nor will it ever do so). That > whole > > > > 'cookieverf' READDIR bullshit has never provided a workable > model > > > for a > > > > POSIX client... > > > > > > > > > Thanks Trond for the explanation. Forgive my slowness. So you are > > > > saying that with knfsd it's actually filesystem dependent and > maybe > > > the reporters did not compare apples-to-apples and the difference > is > > > in the behind file system. > > > > > > Matt I'm sure Trond is right with regard to Ganesha, because we > send a > > > readdir index and a cookieverf, which will change after unlink. > Since > > > the application does not call opendir again the client will send > the > > > old cookies, which is now a different file or might get > out-of-bounds > > > for Ganesha. > > > > > > Let me think about it a bit. I'm sure we can send a Better 64bit > > > cookie, which will be more persistent, even across unlinks. > > > > > > > > > > > > > > > > > > > > Thanks > > > Boaz > > > > -- > > Matt Benjamin > > The Linux Box > > 206 South Fifth Ave. Suite 150 > > Ann Arbor, MI 48104 > > > > http://linuxbox.com > > > > tel. 734-761-4689 > > fax. 734-769-8938 > > cel. 734-216-5309 > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-nfs" > in the body of a message to majordomo@vger.kernel.org More majordomo > info at http://vger.kernel.org/majordomo-info.html > > -- > Trond Myklebust > Linux NFS client maintainer > > NetApp > Trond.Myklebust@netapp.com > www.netapp.com -- Matt Benjamin The Linux Box 206 South Fifth Ave. Suite 150 Ann Arbor, MI 48104 http://linuxbox.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: unlink within an open directory stream 2012-03-26 18:17 ` Boaz Harrosh 2012-03-26 18:25 ` Myklebust, Trond @ 2012-03-30 20:17 ` Jeff Layton 2012-04-04 15:35 ` Jeff Layton 2 siblings, 0 replies; 19+ messages in thread From: Jeff Layton @ 2012-03-30 20:17 UTC (permalink / raw) To: Boaz Harrosh; +Cc: Myklebust, Trond, Matt W. Benjamin, linux-nfs On Mon, 26 Mar 2012 11:17:18 -0700 Boaz Harrosh <bharrosh@panasas.com> wrote: > On 03/24/2012 10:12 AM, Myklebust, Trond wrote: > > > On Sat, 2012-03-24 at 12:53 -0400, Matt W. Benjamin wrote: > >> Hi, > >> > >> I don't think anything is. Or, people originally reported the behavior against knfsd. > >> > >> Matt > > > > There is a known issue with ext2/3/4 generating non-unique readdir > > cookies. It rarely hits you when you are creating small directories, but > > it frequently hits you with larger ones. A fix is underway that should > > significantly reduce the frequency of cookie collisions. > > > > Recent NFS clients will actually detect the presence of those cookie > > loops, and log them in the kernel syslog. That would therefore be the > > first thing that I'd check if confronted with this kind of problem. > > > > Cheers > > Trond > > > > > Trond please look on the bug report links below. It's not the "cookie collisions" case. > > It's the new (post RHEL 6.0 Kernel) NFS need for opendir after an unlink. > Now the POSIX man page *does* say that applications must re-opendir after > unlink, but there are some applications who did not read the manual, and since > it works with local filesystems and old nfs, (What Kernel RHEL 6.0 is based on?) > they never noticed the bug and never fixed it. > The RHEL6 kernel is 2.6.32 based, but we have (as always) done some fairly extensive backports from upstream. One of the things that was backported was the Bryan's work to clean up readdir and to eliminate the readdirplus limit. With that change, you get fewer entries per READDIRPLUS call, so you make more READDIRPLUS calls to the server in order to traverse an entire directory. Since you're unlinking as you go, then I suspect that you're just more likely to see this problem crop up, but I suspect Trond is correct and this is preexisting bug. One way to confirm this might be to have them mount with -o nordirplus. Does that make the problem go away? -- Jeff Layton <jlayton@redhat.com> ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: unlink within an open directory stream 2012-03-26 18:17 ` Boaz Harrosh 2012-03-26 18:25 ` Myklebust, Trond 2012-03-30 20:17 ` Jeff Layton @ 2012-04-04 15:35 ` Jeff Layton 2012-04-05 8:45 ` Benny Halevy 2 siblings, 1 reply; 19+ messages in thread From: Jeff Layton @ 2012-04-04 15:35 UTC (permalink / raw) To: Boaz Harrosh; +Cc: Myklebust, Trond, Matt W. Benjamin, linux-nfs On Mon, 26 Mar 2012 11:17:18 -0700 Boaz Harrosh <bharrosh@panasas.com> wrote: > On 03/24/2012 10:12 AM, Myklebust, Trond wrote: > > > On Sat, 2012-03-24 at 12:53 -0400, Matt W. Benjamin wrote: > >> Hi, > >> > >> I don't think anything is. Or, people originally reported the behavior against knfsd. > >> > >> Matt > > > > There is a known issue with ext2/3/4 generating non-unique readdir > > cookies. It rarely hits you when you are creating small directories, but > > it frequently hits you with larger ones. A fix is underway that should > > significantly reduce the frequency of cookie collisions. > > > > Recent NFS clients will actually detect the presence of those cookie > > loops, and log them in the kernel syslog. That would therefore be the > > first thing that I'd check if confronted with this kind of problem. > > > > Cheers > > Trond > > > > > Trond please look on the bug report links below. It's not the "cookie collisions" case. > > It's the new (post RHEL 6.0 Kernel) NFS need for opendir after an unlink. > Now the POSIX man page *does* say that applications must re-opendir after > unlink, but there are some applications who did not read the manual, and since > it works with local filesystems and old nfs, (What Kernel RHEL 6.0 is based on?) > they never noticed the bug and never fixed it. > ^^^^^ Can you tell me which manpage says this? I'd like to be able to point application developers at it if possible... > Could we easily support the broken application by being bug compatible to > old NFS versions? > .i.e Don't require re-opendir after unlink of a file. > > There are more examples in the bug reports below but basically bonnie++ > does the following: > DIR *d = opendir("."); > dirent *file_ent; > while((file_ent = readdir(d)) != NULL) { > unlink( file_ent->d_name)) > } > closedir(d); > > where it actually needs to do: > > DIR *d = opendir("."); > dirent *file_ent; > while((file_ent = readdir(d)) != NULL) { > unlink( file_ent->d_name)) > > closedir(d); > d = opendir("."); > } > closedir(d); > > But again case one used to work with old NFS. And it looks like > it is not Server dependent. We saw this both with Ganesha as well > as knfsd > > <snip> > Again, my suspicion is that the change that triggered this is the switch to use READDIRPLUS on larger directories. Before that, we'd use READDIR on larger ones and wouldn't need to make as many RPCs to fetch directory contents. More continuation READDIRPLUS calls means that you have more opportunity to hit problems with cookies. What might be an interesting test is to see whether this is still reproducible on newer clients when you mount with '-o nordirplus'. Cheers, -- Jeff Layton <jlayton@redhat.com> ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: unlink within an open directory stream 2012-04-04 15:35 ` Jeff Layton @ 2012-04-05 8:45 ` Benny Halevy 0 siblings, 0 replies; 19+ messages in thread From: Benny Halevy @ 2012-04-05 8:45 UTC (permalink / raw) To: Jeff Layton; +Cc: Boaz Harrosh, Myklebust, Trond, Matt W. Benjamin, linux-nfs On 2012-04-04 18:35, Jeff Layton wrote: > On Mon, 26 Mar 2012 11:17:18 -0700 > Boaz Harrosh <bharrosh@panasas.com> wrote: > >> On 03/24/2012 10:12 AM, Myklebust, Trond wrote: >> >>> On Sat, 2012-03-24 at 12:53 -0400, Matt W. Benjamin wrote: >>>> Hi, >>>> >>>> I don't think anything is. Or, people originally reported the behavior against knfsd. >>>> >>>> Matt >>> >>> There is a known issue with ext2/3/4 generating non-unique readdir >>> cookies. It rarely hits you when you are creating small directories, but >>> it frequently hits you with larger ones. A fix is underway that should >>> significantly reduce the frequency of cookie collisions. >>> >>> Recent NFS clients will actually detect the presence of those cookie >>> loops, and log them in the kernel syslog. That would therefore be the >>> first thing that I'd check if confronted with this kind of problem. >>> >>> Cheers >>> Trond >>> >> >> >> Trond please look on the bug report links below. It's not the "cookie collisions" case. >> >> It's the new (post RHEL 6.0 Kernel) NFS need for opendir after an unlink. >> Now the POSIX man page *does* say that applications must re-opendir after >> unlink, but there are some applications who did not read the manual, and since >> it works with local filesystems and old nfs, (What Kernel RHEL 6.0 is based on?) >> they never noticed the bug and never fixed it. >> > > ^^^^^ > Can you tell me which manpage says this? I'd like to be able to point > application developers at it if possible... > Hmm, http://pubs.opengroup.org/onlinepubs/9699919799/functions/readdir.html says: files may be removed from a directory or added to a directory asynchronously to the operation of readdir(). ... If a file is removed from or added to the directory after the most recent call to opendir() or rewinddir(), whether a subsequent call to readdir() returns an entry for that file is unspecified. ... If a file is removed from or added to the directory after the most recent call to opendir() or rewinddir(), whether a subsequent call to readdir_r() returns an entry for that file is unspecified. Benny >> Could we easily support the broken application by being bug compatible to >> old NFS versions? >> .i.e Don't require re-opendir after unlink of a file. >> >> There are more examples in the bug reports below but basically bonnie++ >> does the following: >> DIR *d = opendir("."); >> dirent *file_ent; >> while((file_ent = readdir(d)) != NULL) { >> unlink( file_ent->d_name)) >> } >> closedir(d); >> >> where it actually needs to do: >> >> DIR *d = opendir("."); >> dirent *file_ent; >> while((file_ent = readdir(d)) != NULL) { >> unlink( file_ent->d_name)) >> >> closedir(d); >> d = opendir("."); >> } >> closedir(d); >> >> But again case one used to work with old NFS. And it looks like >> it is not Server dependent. We saw this both with Ganesha as well >> as knfsd >> >> <snip> >> > > Again, my suspicion is that the change that triggered this is the > switch to use READDIRPLUS on larger directories. Before that, we'd use > READDIR on larger ones and wouldn't need to make as many RPCs to fetch > directory contents. More continuation READDIRPLUS calls means that you > have more opportunity to hit problems with cookies. > > What might be an interesting test is to see whether this is still > reproducible on newer clients when you mount with '-o nordirplus'. > > Cheers, ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2012-04-05 8:45 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1598378492.82.1332790088063.JavaMail.root@thunderbeast.private.linuxbox.com>
2012-03-26 19:29 ` unlink within an open directory stream Matt W. Benjamin
[not found] <209745840.4.1332606289405.JavaMail.root@thunderbeast.private.linuxbox.com>
2012-03-24 16:34 ` Matt W. Benjamin
2012-03-24 16:44 ` Myklebust, Trond
2012-03-24 16:53 ` Matt W. Benjamin
2012-03-24 17:12 ` Myklebust, Trond
2012-03-24 17:43 ` Matt W. Benjamin
2012-03-26 18:17 ` Boaz Harrosh
2012-03-26 18:25 ` Myklebust, Trond
2012-03-26 18:43 ` Boaz Harrosh
2012-03-26 18:55 ` Matt W. Benjamin
2012-03-26 19:01 ` Myklebust, Trond
2012-03-26 19:07 ` Boaz Harrosh
2012-03-30 15:24 ` Peter Staubach
2012-03-30 15:46 ` Myklebust, Trond
2012-03-30 16:46 ` Peter Staubach
2012-03-30 17:10 ` Matt W. Benjamin
2012-03-30 20:17 ` Jeff Layton
2012-04-04 15:35 ` Jeff Layton
2012-04-05 8:45 ` Benny Halevy
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).