Spurious permission denied

Spurious permission denied

[Christoph Bartoschek]

[-- Attachment #1: Type: text/plain, Size: 885 bytes --]

Hi,

we still have the problem that some users get permission denied for 
directories they normally can access. The problem only affects a single user 
at a time and automatically goes away after about 30 minutes.

It seems to be a server problem because this happens on all client machines 
at the same time.

Here is how it looks like:

[bartosch@r1106i12 ~]$ ls /afs/eda/prod
ls: cannot open directory /afs/eda/prod: Permission denied
[bartosch@r1106i12 ~]$ id
uid=348(bartosch) gid=200(vls) 
groups=200(vlsi),100(users),201(zab),900(ibm),901(iba),902(ama),1001(eda),2030(asi),2057(ecl),41999(tra)
[bartosch@r1106i12 ~]$ 

I have attached the network traffic that was exchanged between the NFS 
server and the client for the ls command captured by wireshark.

Accessing the directory /afs/eda works as expected.

Has anyone an idea what could be wrong?

Thanks,
Christoph Bartoschek

[-- Attachment #2: network_traffic.dump --]
[-- Type: application/vnd.tcpdump.pcap, Size: 518 bytes --]

Re: Spurious permission denied

[Myklebust, Trond]

T24gVHVlLCAyMDEyLTA2LTE5IGF0IDExOjU4ICswMjAwLCBDaHJpc3RvcGggQmFydG9zY2hlayB3
cm90ZToNCj4gSGksDQo+IA0KPiB3ZSBzdGlsbCBoYXZlIHRoZSBwcm9ibGVtIHRoYXQgc29tZSB1
c2VycyBnZXQgcGVybWlzc2lvbiBkZW5pZWQgZm9yIA0KPiBkaXJlY3RvcmllcyB0aGV5IG5vcm1h
bGx5IGNhbiBhY2Nlc3MuIFRoZSBwcm9ibGVtIG9ubHkgYWZmZWN0cyBhIHNpbmdsZSB1c2VyIA0K
PiBhdCBhIHRpbWUgYW5kIGF1dG9tYXRpY2FsbHkgZ29lcyBhd2F5IGFmdGVyIGFib3V0IDMwIG1p
bnV0ZXMuDQo+IA0KPiBJdCBzZWVtcyB0byBiZSBhIHNlcnZlciBwcm9ibGVtIGJlY2F1c2UgdGhp
cyBoYXBwZW5zIG9uIGFsbCBjbGllbnQgbWFjaGluZXMgDQo+IGF0IHRoZSBzYW1lIHRpbWUuDQo+
IA0KPiBIZXJlIGlzIGhvdyBpdCBsb29rcyBsaWtlOg0KPiANCj4gW2JhcnRvc2NoQHIxMTA2aTEy
IH5dJCBscyAvYWZzL2VkYS9wcm9kDQo+IGxzOiBjYW5ub3Qgb3BlbiBkaXJlY3RvcnkgL2Fmcy9l
ZGEvcHJvZDogUGVybWlzc2lvbiBkZW5pZWQNCj4gW2JhcnRvc2NoQHIxMTA2aTEyIH5dJCBpZA0K
PiB1aWQ9MzQ4KGJhcnRvc2NoKSBnaWQ9MjAwKHZscykgDQo+IGdyb3Vwcz0yMDAodmxzaSksMTAw
KHVzZXJzKSwyMDEoemFiKSw5MDAoaWJtKSw5MDEoaWJhKSw5MDIoYW1hKSwxMDAxKGVkYSksMjAz
MChhc2kpLDIwNTcoZWNsKSw0MTk5OSh0cmEpDQo+IFtiYXJ0b3NjaEByMTEwNmkxMiB+XSQgDQo+
IA0KPiBJIGhhdmUgYXR0YWNoZWQgdGhlIG5ldHdvcmsgdHJhZmZpYyB0aGF0IHdhcyBleGNoYW5n
ZWQgYmV0d2VlbiB0aGUgTkZTIA0KPiBzZXJ2ZXIgYW5kIHRoZSBjbGllbnQgZm9yIHRoZSBscyBj
b21tYW5kIGNhcHR1cmVkIGJ5IHdpcmVzaGFyay4NCj4gDQo+IEFjY2Vzc2luZyB0aGUgZGlyZWN0
b3J5IC9hZnMvZWRhIHdvcmtzIGFzIGV4cGVjdGVkLg0KPiANCj4gSGFzIGFueW9uZSBhbiBpZGVh
IHdoYXQgY291bGQgYmUgd3Jvbmc/DQoNCk5GU3YzIGRvZXNuJ3QgdXNlIHRoZSBtb2RlIGJpdHMg
dG8gZGV0ZXJtaW5lIGFjY2VzcyByaWdodHMsIHNvIGxvb2tpbmcNCmF0IGEgR0VUQVRUUiBjYWxs
IGlzbid0IHJlYWxseSB0aGF0IGhlbHBmdWwuIFlvdSBuZWVkIHRvIGNhdGNoIHRoZQ0Kb3JpZ2lu
YWwgQUNDRVNTIGNhbGwgYW5kIHJlcGx5IGluIG9yZGVyIHRvIGZpZ3VyZSBvdXQgd2hhdCBjYXVz
ZWQgdGhlDQpwZXJtaXNzaW9uIGRlbmllZCBpc3N1ZS4NCg0KLS0gDQpUcm9uZCBNeWtsZWJ1c3QN
CkxpbnV4IE5GUyBjbGllbnQgbWFpbnRhaW5lcg0KDQpOZXRBcHANClRyb25kLk15a2xlYnVzdEBu
ZXRhcHAuY29tDQp3d3cubmV0YXBwLmNvbQ0KDQo=

Re: Spurious permission denied

[Christoph Bartoschek]

Myklebust, Trond wrote:

> 
> NFSv3 doesn't use the mode bits to determine access rights, so looking
> at a GETATTR call isn't really that helpful. You need to catch the
> original ACCESS call and reply in order to figure out what caused the
> permission denied issue.

Hmm.

I started capturing the packets before I tried the ls and stopped some 
seconds after ls.

These were the only three packets wireshark captured. 

I wait till the problem occurs again and try to get more meaningful packets.

Thanks
Christoph

Re: Spurious permission denied

[Mike Grant]

On 06/19/2012 10:58 AM, Christoph Bartoschek wrote:
> we still have the problem that some users get permission denied for
> directories they normally can access. The problem only affects a single user
> at a time and automatically goes away after about 30 minutes.

We've seen this error quite a bit too, though never managed to pin it 
down well enough to report it as it's intermittent and unpredictable.  I 
agree that it seems to be server side and we tried a few things to get 
rid of it.  I'm not sure if they worked or if it's just in remission, 
but the two things that seemed to help most were:
  a) increase the amount of memory in the server (we went from 6 to 24GB)
  b) reduce dependence on NIS by (ick) hard coding key users into the 
server's /etc/passwd file.

We tried the latter because our NIS server was struggling a lot at the 
time and I'm not sure what happens if the NFS server fails to get a NIS 
response in a timely manner or gets a failed response (i,e. it can't 
identify the user).

It may be worth you seeing if the user id resolves correctly on the 
server at the time of the failure.  It might also be interesting to know 
what happens if you restart the NFS server when you have this problem. 
I'm wondering if the 30min thing indicates a cached bad credential or 
something that expires after a bit or perhaps the NFS server thread just 
gets recycled..

It may also be worth doing a capture on the server side at the same 
time, as that might show issues with NIS or whatever you use.  There's a 
good chance it may not be visible after the problem has occurred though, 
if it's cached somewhere.

Anyway, sorry that's not directly helpful, but maybe a few things to try!

Cheers,

Mike.

<br />
<hr />
<p><font face="Arial" size="1">
Plymouth Marine Laboratory<br />
Registered Office: <br />
Prospect Place<br />
The Hoe<br />
Plymouth  PL1 3DH
</font></p>

<p><font face="Arial" size="1">Website: <a href="http://www.pml.ac.uk">www.pml.ac.uk</a>
<br />
<a href="http://www.pml.ac.uk/pdf/PML%20Annual%20Review%202011_2.pdf">Click here for the latest PML Annual Review</a>
<br />
Registered Charity No. 1091222<br />
PML is a company limited by guarantee<br />
registered in England & Wales<br />
company number 4178503</font></p>

<p><font face="Arial" size="1" color="green">Please think before you print.</font></p>

<hr />

<p><font face="Arial" size="1">This e-mail, its content and any file attachments are confidential.</font></p>

<p><font face="Arial" size="1">If you have received this e-mail in error please do not copy, disclose it to any third party or use the contents or attachments in any way. Please notify the sender by replying to this e-mail or e-mail forinfo@pml.ac.uk and then delete the email without making any copies or using it in any other way.</font></p>

<p><font face="Arial" size="1">The content of this message may contain personal views which are not the views of Plymouth Marine Laboratory unless specifically stated.</font></p>

<p><font face="Arial" size="1">You are reminded that e-mail communications are not secure and may contain viruses. Plymouth Marine Laboratory accepts no liability for any loss or damage which may be caused by viruses.</font></p>

<hr />
<br />
<br />