* [3.2.5] Stale NFS file handle issue on subdirectory of NFSv3 mount @ 2012-02-29 1:06 Simon Kirby 2012-02-29 1:11 ` Myklebust, Trond 0 siblings, 1 reply; 5+ messages in thread From: Simon Kirby @ 2012-02-29 1:06 UTC (permalink / raw) To: Trond Myklebust; +Cc: linux-nfs Hi, Since upgrading from 2.6.39-ish to 3.1-ish, and on 3.2.5, we are seeing a lot of occurrences of Stale NFS file handle errors when accessing a mount whose NFSv3 source is a subdirectory of another mount point. For example, in this case: # mount | grep /shared 10.10.1.1:/storage/vg1/shared on /shared type nfs (rw,hard,intr,tcp,timeo=300,retrans=2,vers=3,addr=10.10.1.1) 10.10.1.1:/storage/vg1/shared/fp on /usr/local/fp type nfs (rw,hard,intr,tcp,timeo=300,retrans=2,vers=3,addr=10.10.1.1) When the issue occurs, the /shared mount point is fine as is /shared/fp, but "df" or "ls" or anything on /usr/local/fp will ESTALE. This somehow corrected itself while I was trying to gather information this time, but usually the d_ino returned by getdents() on the parent directory shows a different inode number than for /shared/fp. When this happens, I am unable to umount -f or umount -l /usr/local/fp (ESTALE), but I can actually umount /shared; umount /usr/local/fp; and mount -a, which seems to "fix" it. is this acting similar to a bind mount internally now and revalidation or something is breaking in this case? This is happening fairly often, so I will try to collect more info again next time. Simon- ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [3.2.5] Stale NFS file handle issue on subdirectory of NFSv3 mount 2012-02-29 1:06 [3.2.5] Stale NFS file handle issue on subdirectory of NFSv3 mount Simon Kirby @ 2012-02-29 1:11 ` Myklebust, Trond 2012-02-29 19:59 ` Simon Kirby 0 siblings, 1 reply; 5+ messages in thread From: Myklebust, Trond @ 2012-02-29 1:11 UTC (permalink / raw) To: Simon Kirby; +Cc: linux-nfs@vger.kernel.org T24gVHVlLCAyMDEyLTAyLTI4IGF0IDE3OjA2IC0wODAwLCBTaW1vbiBLaXJieSB3cm90ZToNCj4g SGksDQo+IA0KPiBTaW5jZSB1cGdyYWRpbmcgZnJvbSAyLjYuMzktaXNoIHRvIDMuMS1pc2gsIGFu ZCBvbiAzLjIuNSwgd2UgYXJlIHNlZWluZyBhDQo+IGxvdCBvZiBvY2N1cnJlbmNlcyBvZiBTdGFs ZSBORlMgZmlsZSBoYW5kbGUgZXJyb3JzIHdoZW4gYWNjZXNzaW5nIGEgbW91bnQNCj4gd2hvc2Ug TkZTdjMgc291cmNlIGlzIGEgc3ViZGlyZWN0b3J5IG9mIGFub3RoZXIgbW91bnQgcG9pbnQuIEZv ciBleGFtcGxlLA0KPiBpbiB0aGlzIGNhc2U6DQo+IA0KPiAjIG1vdW50IHwgZ3JlcCAvc2hhcmVk DQo+IDEwLjEwLjEuMTovc3RvcmFnZS92ZzEvc2hhcmVkIG9uIC9zaGFyZWQgdHlwZSBuZnMgKHJ3 LGhhcmQsaW50cix0Y3AsdGltZW89MzAwLHJldHJhbnM9Mix2ZXJzPTMsYWRkcj0xMC4xMC4xLjEp DQo+IDEwLjEwLjEuMTovc3RvcmFnZS92ZzEvc2hhcmVkL2ZwIG9uIC91c3IvbG9jYWwvZnAgdHlw ZSBuZnMgKHJ3LGhhcmQsaW50cix0Y3AsdGltZW89MzAwLHJldHJhbnM9Mix2ZXJzPTMsYWRkcj0x MC4xMC4xLjEpDQo+IA0KPiBXaGVuIHRoZSBpc3N1ZSBvY2N1cnMsIHRoZSAvc2hhcmVkIG1vdW50 IHBvaW50IGlzIGZpbmUgYXMgaXMgL3NoYXJlZC9mcCwNCj4gYnV0ICJkZiIgb3IgImxzIiBvciBh bnl0aGluZyBvbiAvdXNyL2xvY2FsL2ZwIHdpbGwgRVNUQUxFLiBUaGlzIHNvbWVob3cNCj4gY29y cmVjdGVkIGl0c2VsZiB3aGlsZSBJIHdhcyB0cnlpbmcgdG8gZ2F0aGVyIGluZm9ybWF0aW9uIHRo aXMgdGltZSwgYnV0DQo+IHVzdWFsbHkgdGhlIGRfaW5vIHJldHVybmVkIGJ5IGdldGRlbnRzKCkg b24gdGhlIHBhcmVudCBkaXJlY3Rvcnkgc2hvd3MgYQ0KPiBkaWZmZXJlbnQgaW5vZGUgbnVtYmVy IHRoYW4gZm9yIC9zaGFyZWQvZnAuDQo+IA0KPiBXaGVuIHRoaXMgaGFwcGVucywgSSBhbSB1bmFi bGUgdG8gdW1vdW50IC1mIG9yIHVtb3VudCAtbCAvdXNyL2xvY2FsL2ZwDQo+IChFU1RBTEUpLCBi dXQgSSBjYW4gYWN0dWFsbHkgdW1vdW50IC9zaGFyZWQ7IHVtb3VudCAvdXNyL2xvY2FsL2ZwOyBh bmQNCj4gbW91bnQgLWEsIHdoaWNoIHNlZW1zIHRvICJmaXgiIGl0Lg0KPiANCj4gaXMgdGhpcyBh Y3Rpbmcgc2ltaWxhciB0byBhIGJpbmQgbW91bnQgaW50ZXJuYWxseSBub3cgYW5kIHJldmFsaWRh dGlvbiBvcg0KPiBzb21ldGhpbmcgaXMgYnJlYWtpbmcgaW4gdGhpcyBjYXNlPyBUaGlzIGlzIGhh cHBlbmluZyBmYWlybHkgb2Z0ZW4sIHNvIEkNCj4gd2lsbCB0cnkgdG8gY29sbGVjdCBtb3JlIGlu Zm8gYWdhaW4gbmV4dCB0aW1lLg0KDQpFU1RBTEUgaXMgYSBzZXJ2ZXIgc2lkZSBlcnJvciwgbm90 IGEgY2xpZW50IHNpZGUgZXJyb3IuIFdoYXQgc2VydmVyIGFyZQ0KeW91IHVzaW5nIGhlcmUsIGFu ZCB3aGF0IGRvIHRoZSBleHBvcnQgb3B0aW9ucyBsb29rIGxpa2U/DQoNCi0tIA0KVHJvbmQgTXlr bGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50YWluZXINCg0KTmV0QXBwDQpUcm9uZC5NeWts ZWJ1c3RAbmV0YXBwLmNvbQ0Kd3d3Lm5ldGFwcC5jb20NCg0K ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [3.2.5] Stale NFS file handle issue on subdirectory of NFSv3 mount 2012-02-29 1:11 ` Myklebust, Trond @ 2012-02-29 19:59 ` Simon Kirby 2012-02-29 20:14 ` J. Bruce Fields 0 siblings, 1 reply; 5+ messages in thread From: Simon Kirby @ 2012-02-29 19:59 UTC (permalink / raw) To: Myklebust, Trond; +Cc: linux-nfs@vger.kernel.org On Wed, Feb 29, 2012 at 01:11:31AM +0000, Myklebust, Trond wrote: > On Tue, 2012-02-28 at 17:06 -0800, Simon Kirby wrote: > > Hi, > > > > Since upgrading from 2.6.39-ish to 3.1-ish, and on 3.2.5, we are seeing a > > lot of occurrences of Stale NFS file handle errors when accessing a mount > > whose NFSv3 source is a subdirectory of another mount point. For example, > > in this case: > > > > # mount | grep /shared > > 10.10.1.1:/storage/vg1/shared on /shared type nfs (rw,hard,intr,tcp,timeo=300,retrans=2,vers=3,addr=10.10.1.1) > > 10.10.1.1:/storage/vg1/shared/fp on /usr/local/fp type nfs (rw,hard,intr,tcp,timeo=300,retrans=2,vers=3,addr=10.10.1.1) > > > > When the issue occurs, the /shared mount point is fine as is /shared/fp, > > but "df" or "ls" or anything on /usr/local/fp will ESTALE. This somehow > > corrected itself while I was trying to gather information this time, but > > usually the d_ino returned by getdents() on the parent directory shows a > > different inode number than for /shared/fp. > > > > When this happens, I am unable to umount -f or umount -l /usr/local/fp > > (ESTALE), but I can actually umount /shared; umount /usr/local/fp; and > > mount -a, which seems to "fix" it. > > > > is this acting similar to a bind mount internally now and revalidation or > > something is breaking in this case? This is happening fairly often, so I > > will try to collect more info again next time. > > ESTALE is a server side error, not a client side error. What server are > you using here, and what do the export options look like? An older 2.6.33 host running DRBD HA knfsd bits. We had problems with the XFS inode reclaim changes causing crashes on newer kernels (actually on 2.6.33, too, but not so much on this node with only locally-attached disks), so these kernels haven't been upgraded for some time. It's likely time to try again. The export is: /storage/vg1 /10.10.1.0/24(rw,sync,no_root_squash,no_subtree_check,fsid=1) I just found it weird to see the ESTALE when accessing /usr/local/fp, while /shared/fp works fine at the same time, even though they're the same path on the server. The plan was to upgrade this pair first anyway, so we can do that if the problem is likely coming from the server side. Simon- ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [3.2.5] Stale NFS file handle issue on subdirectory of NFSv3 mount 2012-02-29 19:59 ` Simon Kirby @ 2012-02-29 20:14 ` J. Bruce Fields 2012-03-01 22:45 ` Simon Kirby 0 siblings, 1 reply; 5+ messages in thread From: J. Bruce Fields @ 2012-02-29 20:14 UTC (permalink / raw) To: Simon Kirby; +Cc: Myklebust, Trond, linux-nfs@vger.kernel.org On Wed, Feb 29, 2012 at 11:59:16AM -0800, Simon Kirby wrote: > On Wed, Feb 29, 2012 at 01:11:31AM +0000, Myklebust, Trond wrote: > > > On Tue, 2012-02-28 at 17:06 -0800, Simon Kirby wrote: > > > Hi, > > > > > > Since upgrading from 2.6.39-ish to 3.1-ish, and on 3.2.5, we are seeing a > > > lot of occurrences of Stale NFS file handle errors when accessing a mount > > > whose NFSv3 source is a subdirectory of another mount point. For example, > > > in this case: > > > > > > # mount | grep /shared > > > 10.10.1.1:/storage/vg1/shared on /shared type nfs (rw,hard,intr,tcp,timeo=300,retrans=2,vers=3,addr=10.10.1.1) > > > 10.10.1.1:/storage/vg1/shared/fp on /usr/local/fp type nfs (rw,hard,intr,tcp,timeo=300,retrans=2,vers=3,addr=10.10.1.1) > > > > > > When the issue occurs, the /shared mount point is fine as is /shared/fp, > > > but "df" or "ls" or anything on /usr/local/fp will ESTALE. This somehow > > > corrected itself while I was trying to gather information this time, but > > > usually the d_ino returned by getdents() on the parent directory shows a > > > different inode number than for /shared/fp. > > > > > > When this happens, I am unable to umount -f or umount -l /usr/local/fp > > > (ESTALE), but I can actually umount /shared; umount /usr/local/fp; and > > > mount -a, which seems to "fix" it. > > > > > > is this acting similar to a bind mount internally now and revalidation or > > > something is breaking in this case? This is happening fairly often, so I > > > will try to collect more info again next time. > > > > ESTALE is a server side error, not a client side error. What server are > > you using here, and what do the export options look like? > > An older 2.6.33 host running DRBD HA knfsd bits. We had problems with the > XFS inode reclaim changes causing crashes on newer kernels (actually on > 2.6.33, too, but not so much on this node with only locally-attached > disks), so these kernels haven't been upgraded for some time. It's likely > time to try again. The export is: > > /storage/vg1 /10.10.1.0/24(rw,sync,no_root_squash,no_subtree_check,fsid=1) > > I just found it weird to see the ESTALE when accessing /usr/local/fp, > while /shared/fp works fine at the same time, even though they're the > same path on the server. Is /storage/vg1/shared/fp on that server something that can ever be removed? (Say to be replaced by something else?) For normal directories that's not a problem, the client's used to dealing with the fact that directories may come and go. For a directory that you've told the client to *mount*, that's dirty trick--it's really expecting that directory to be there as long as it's mounted.... --b. > > The plan was to upgrade this pair first anyway, so we can do that if > the problem is likely coming from the server side. > > Simon- > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [3.2.5] Stale NFS file handle issue on subdirectory of NFSv3 mount 2012-02-29 20:14 ` J. Bruce Fields @ 2012-03-01 22:45 ` Simon Kirby 0 siblings, 0 replies; 5+ messages in thread From: Simon Kirby @ 2012-03-01 22:45 UTC (permalink / raw) To: J. Bruce Fields; +Cc: Myklebust, Trond, linux-nfs@vger.kernel.org On Wed, Feb 29, 2012 at 03:14:01PM -0500, J. Bruce Fields wrote: > On Wed, Feb 29, 2012 at 11:59:16AM -0800, Simon Kirby wrote: > > On Wed, Feb 29, 2012 at 01:11:31AM +0000, Myklebust, Trond wrote: > > > > > On Tue, 2012-02-28 at 17:06 -0800, Simon Kirby wrote: > > > > Hi, > > > > > > > > Since upgrading from 2.6.39-ish to 3.1-ish, and on 3.2.5, we are seeing a > > > > lot of occurrences of Stale NFS file handle errors when accessing a mount > > > > whose NFSv3 source is a subdirectory of another mount point. For example, > > > > in this case: > > > > > > > > # mount | grep /shared > > > > 10.10.1.1:/storage/vg1/shared on /shared type nfs (rw,hard,intr,tcp,timeo=300,retrans=2,vers=3,addr=10.10.1.1) > > > > 10.10.1.1:/storage/vg1/shared/fp on /usr/local/fp type nfs (rw,hard,intr,tcp,timeo=300,retrans=2,vers=3,addr=10.10.1.1) > > > > > > > > When the issue occurs, the /shared mount point is fine as is /shared/fp, > > > > but "df" or "ls" or anything on /usr/local/fp will ESTALE. This somehow > > > > corrected itself while I was trying to gather information this time, but > > > > usually the d_ino returned by getdents() on the parent directory shows a > > > > different inode number than for /shared/fp. > > > > > > > > When this happens, I am unable to umount -f or umount -l /usr/local/fp > > > > (ESTALE), but I can actually umount /shared; umount /usr/local/fp; and > > > > mount -a, which seems to "fix" it. > > > > > > > > is this acting similar to a bind mount internally now and revalidation or > > > > something is breaking in this case? This is happening fairly often, so I > > > > will try to collect more info again next time. > > > > > > ESTALE is a server side error, not a client side error. What server are > > > you using here, and what do the export options look like? > > > > An older 2.6.33 host running DRBD HA knfsd bits. We had problems with the > > XFS inode reclaim changes causing crashes on newer kernels (actually on > > 2.6.33, too, but not so much on this node with only locally-attached > > disks), so these kernels haven't been upgraded for some time. It's likely > > time to try again. The export is: > > > > /storage/vg1 /10.10.1.0/24(rw,sync,no_root_squash,no_subtree_check,fsid=1) > > > > I just found it weird to see the ESTALE when accessing /usr/local/fp, > > while /shared/fp works fine at the same time, even though they're the > > same path on the server. > > Is /storage/vg1/shared/fp on that server something that can ever be > removed? (Say to be replaced by something else?) > > For normal directories that's not a problem, the client's used to > dealing with the fact that directories may come and go. > > For a directory that you've told the client to *mount*, that's dirty > trick--it's really expecting that directory to be there as long as it's > mounted.... Sure, but nope, the directory hasn't moved or been deleted since it was created. It only grows. Simon- ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2012-03-01 22:45 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-02-29 1:06 [3.2.5] Stale NFS file handle issue on subdirectory of NFSv3 mount Simon Kirby 2012-02-29 1:11 ` Myklebust, Trond 2012-02-29 19:59 ` Simon Kirby 2012-02-29 20:14 ` J. Bruce Fields 2012-03-01 22:45 ` Simon Kirby
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).