From: Trond Myklebust <trondmy@hammerspace.com>
To: "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
"jas@yorku.ca" <jas@yorku.ca>
Subject: Re: NFS server issue after new AD group added to AD user
Date: Sat, 13 Feb 2021 00:56:08 +0000 [thread overview]
Message-ID: <3795c23197b4f171b143821c033fa1687d3163e7.camel@hammerspace.com> (raw)
In-Reply-To: <ad4ee9a5-9285-b29e-8876-f3c7bb1bdbae@yorku.ca>
On Fri, 2021-02-12 at 19:13 -0500, Jason Keltz wrote:
> Hi.
>
> I'm having a very frustrrating NFS related problem that I am hoping
> someone might be able to shed some light on.
>
> I have several file servers, and clients all running CentOS 7.8
> (kernel
> 3.10.0-1127.18.2.el7). They are joined to an Active Directory domain
> controller (Samba).
>
> uid, gid, homedir information is in the directory. All the clients
> are
> running Winbind.
>
> I can freely ssh between all the CentOS machines, and my home
> directory
> is auto-mounted from the NFS server using NFSv4.1 and krb5. It's
> pretty
> nice!
>
> If I add myself to a new AD group (group name is cmosp18 gid is
> 100006)
> , and I wait 5 minutes (the winbind cache timeout), then I can log
> out
> of a system and back in, and I will be in the new group (that is
> "groups" command shows that I'm in the group).
>
> I can cd into /tmp, create a file, and chgrp it to the new group and
> it
> works exactly as I would expect. However, if I try to go into my
> home
> directory, create a file there, and chgrp it to the new group, the
> NFSv4
> server returns EPERM for the chgrp. From the strace of chgrp:
>
> fchownat(AT_FDCWD, "l", -1, 100006, 0) = -1 EPERM (Operation not
> permitted)
>
> If I try the same operation from the same client to the same server
> but
> using NFSv3 (also krb5), it works successfully.
>
> I'd think this means that there is an NFS4 id mapping issue, but I
> don't
> see what that is. The client seems to be able to map my user and
> group
> correctly. "groups" command on both the client and server shows I'm
> in
> the group. The server seems to map my user and group correctly.
>
> If I login to the file server as me, and create the file in my home
> directory, make it owned by the new group, then I login to the
> client, I
> can see the file (owner and group shows up perfectly fine) (even
> though
> I can't create it on the client).
>
> In addition, if I login to a client that I haven't logged into, then
> I
> can change the group of the file in my home directory to the new
> group
> as well.
>
> Here's a tcpdump between my host and the nfs server when the chgrp is
> rejected:
>
> > 15:43:50.950575 IP (tos 0x0, ttl 64, id 34491, offset 0, flags
> [DF],
> proto TCP (6), length 252)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [P.],
> cksum
> 0xc54d (incorrect -> 0xd37d), seq 912:1112, ack 729, win 8934,
> options
> [nop,nop,TS val 5954964 ecr 510044867], length 200: NFS request xid
> 2301246825 196 getattr fh 0,1/53
> > 15:43:50.951333 IP (tos 0x0, ttl 63, id 17178, offset 0, flags
> [DF],
> proto TCP (6), length 252)
> > nfs.eecs.yorku.ca.nfs > j1.eecs.yorku.ca.822: Flags [P.],
> cksum
> 0x4785 (correct), seq 729:929, ack 1112, win 1769, options
> [nop,nop,TS
> val 510052992 ecr 5954964], length 200: NFS reply xid 2301246825
> reply
> ok 196 getattr NON 4 ids 0/-1374905249 sz 9481761
> > 15:43:50.951379 IP (tos 0x0, ttl 64, id 34492, offset 0, flags
> [DF],
> proto TCP (6), length 52)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [.], cksum
> 0xc485 (incorrect -> 0x733d), seq 1112, ack 929, win 8934, options
> [nop,nop,TS val 5954965 ecr 510052992], length 0
> > 15:43:50.952260 IP (tos 0x0, ttl 64, id 34493, offset 0, flags
> [DF],
> proto TCP (6), length 252)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [P.],
> cksum
> 0xc54d (incorrect -> 0x8815), seq 1112:1312, ack 929, win 8934,
> options
> [nop,nop,TS val 5954966 ecr 510052992], length 200: NFS request xid
> 2318024041 196 getattr fh 0,1/53
> > 15:43:50.953121 IP (tos 0x0, ttl 63, id 17179, offset 0, flags
> [DF],
> proto TCP (6), length 252)
> > nfs.eecs.yorku.ca.nfs > j1.eecs.yorku.ca.822: Flags [P.],
> cksum
> 0xd848 (correct), seq 929:1129, ack 1312, win 1769, options
> [nop,nop,TS
> val 510052994 ecr 5954966], length 200: NFS reply xid 2318024041
> reply
> ok 196 getattr NON 4 ids 0/-1374905249 sz 9481761
> > 15:43:50.958636 IP (tos 0x0, ttl 64, id 34494, offset 0, flags
> [DF],
> proto TCP (6), length 264)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [P.],
> cksum
> 0xc559 (incorrect -> 0x85f1), seq 1312:1524, ack 1129, win 8934,
> options
> [nop,nop,TS val 5954972 ecr 510052994], length 212: NFS request xid
> 2334801257 208 getattr fh 0,1/53
> > 15:43:50.959401 IP (tos 0x0, ttl 63, id 17180, offset 0, flags
> [DF],
> proto TCP (6), length 252)
> > nfs.eecs.yorku.ca.nfs > j1.eecs.yorku.ca.822: Flags [P.],
> cksum
> 0x8a16 (correct), seq 1129:1329, ack 1524, win 1769, options
> [nop,nop,TS
> val 510053000 ecr 5954972], length 200: NFS reply xid 2334801257
> reply
> ok 196 getattr NON 4 ids 0/-1374905249 sz 9481761
> > 15:43:50.959794 IP (tos 0x0, ttl 64, id 34495, offset 0, flags
> [DF],
> proto TCP (6), length 252)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [P.],
> cksum
> 0xc54d (incorrect -> 0xaea5), seq 1524:1724, ack 1329, win 8934,
> options
> [nop,nop,TS val 5954973 ecr 510053000], length 200: NFS request xid
> 2351578473 196 getattr fh 0,1/53
> > 15:43:50.960530 IP (tos 0x0, ttl 63, id 17181, offset 0, flags
> [DF],
> proto TCP (6), length 252)
> > nfs.eecs.yorku.ca.nfs > j1.eecs.yorku.ca.822: Flags [P.],
> cksum
> 0xc23a (correct), seq 1329:1529, ack 1724, win 1769, options
> [nop,nop,TS
> val 510053001 ecr 5954973], length 200: NFS reply xid 2351578473
> reply
> ok 196 getattr NON 4 ids 0/-1374905249 sz 9481761
> > 15:43:50.960878 IP (tos 0x0, ttl 64, id 34496, offset 0, flags
> [DF],
> proto TCP (6), length 264)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [P.],
> cksum
> 0xc559 (incorrect -> 0x04e5), seq 1724:1936, ack 1529, win 8934,
> options
> [nop,nop,TS val 5954974 ecr 510053001], length 212: NFS request xid
> 2368355689 208 getattr fh 0,1/53
> > 15:43:50.961618 IP (tos 0x0, ttl 63, id 17182, offset 0, flags
> [DF],
> proto TCP (6), length 252)
> > nfs.eecs.yorku.ca.nfs > j1.eecs.yorku.ca.822: Flags [P.],
> cksum
> 0xcc54 (correct), seq 1529:1729, ack 1936, win 1769, options
> [nop,nop,TS
> val 510053002 ecr 5954974], length 200: NFS reply xid 2368355689
> reply
> ok 196 getattr NON 4 ids 0/-1374905249 sz 9481761
> > 15:43:50.961933 IP (tos 0x0, ttl 64, id 34497, offset 0, flags
> [DF],
> proto TCP (6), length 256)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [P.],
> cksum
> 0xc551 (incorrect -> 0x0e22), seq 1936:2140, ack 1729, win 8934,
> options
> [nop,nop,TS val 5954975 ecr 510053002], length 204: NFS request xid
> 2385132905 200 getattr fh 0,1/53
> > 15:43:50.962656 IP (tos 0x0, ttl 63, id 17183, offset 0, flags
> [DF],
> proto TCP (6), length 356)
> > nfs.eecs.yorku.ca.nfs > j1.eecs.yorku.ca.822: Flags [P.],
> cksum
> 0xb114 (correct), seq 1729:2033, ack 2140, win 1769, options
> [nop,nop,TS
> val 510053003 ecr 5954975], length 304: NFS reply xid 2385132905
> reply
> ok 300 getattr NON 3 ids 0/-1374905249 sz 9481761
> > 15:43:50.963142 IP (tos 0x0, ttl 64, id 34498, offset 0, flags
> [DF],
> proto TCP (6), length 264)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [P.],
> cksum
> 0xc559 (incorrect -> 0x80f7), seq 2140:2352, ack 2033, win 8934,
> options
> [nop,nop,TS val 5954977 ecr 510053003], length 212: NFS request xid
> 2401910121 208 getattr fh 0,1/53
> > 15:43:50.964055 IP (tos 0x0, ttl 63, id 17184, offset 0, flags
> [DF],
> proto TCP (6), length 252)
> > nfs.eecs.yorku.ca.nfs > j1.eecs.yorku.ca.822: Flags [P.],
> cksum
> 0x2d91 (correct), seq 2033:2233, ack 2352, win 1769, options
> [nop,nop,TS
> val 510053005 ecr 5954977], length 200: NFS reply xid 2401910121
> reply
> ok 196 getattr NON 4 ids 0/-1374905249 sz 9481761
> > 15:43:51.003985 IP (tos 0x0, ttl 64, id 34499, offset 0, flags
> [DF],
> proto TCP (6), length 52)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [.], cksum
> 0xc485 (incorrect -> 0x690b), seq 2352, ack 2233, win 8934, options
> [nop,nop,TS val 5955018 ecr 510053005], length 0
> > 15:43:51.027598 IP (tos 0x0, ttl 64, id 34500, offset 0, flags
> [DF],
> proto TCP (6), length 272)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [P.],
> cksum
> 0xc561 (incorrect -> 0x83a6), seq 2352:2572, ack 2233, win 8934,
> options
> [nop,nop,TS val 5955041 ecr 510053005], length 220: NFS request xid
> 2418687337 216 getattr fh 0,1/53
> > 15:43:51.027992 IP (tos 0x0, ttl 63, id 17185, offset 0, flags
> [DF],
> proto TCP (6), length 412)
> > nfs.eecs.yorku.ca.nfs > j1.eecs.yorku.ca.822: Flags [P.],
> cksum
> 0x9b87 (correct), seq 2233:2593, ack 2572, win 1769, options
> [nop,nop,TS
> val 510053069 ecr 5955041], length 360: NFS reply xid 2418687337
> reply
> ok 356 getattr NON 5 ids 0/-1374905249 sz 9481761
> > 15:43:51.028003 IP (tos 0x0, ttl 64, id 34501, offset 0, flags
> [DF],
> proto TCP (6), length 52)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [.], cksum
> 0xc485 (incorrect -> 0x666f), seq 2572, ack 2593, win 8934, options
> [nop,nop,TS val 5955042 ecr 510053069], length 0
> > 15:43:51.028359 IP (tos 0x0, ttl 64, id 34502, offset 0, flags
> [DF],
> proto TCP (6), length 320)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [P.],
> cksum
> 0xc591 (incorrect -> 0x688b), seq 2572:2840, ack 2593, win 8934,
> options
> [nop,nop,TS val 5955042 ecr 510053069], length 268: NFS request xid
> 2435464553 264 getattr fh 0,1/53
> > 15:43:51.028741 IP (tos 0x0, ttl 63, id 17186, offset 0, flags
> [DF],
> proto TCP (6), length 196)
> > nfs.eecs.yorku.ca.nfs > j1.eecs.yorku.ca.822: Flags [P.],
> cksum
> 0x7ac5 (correct), seq 2593:2737, ack 2840, win 1769, options
> [nop,nop,TS
> val 510053070 ecr 5955042], length 144: NFS reply xid 2435464553
> reply
> ok 140 getattr ERROR: Operation not permitted
> > 15:43:51.204032 IP (tos 0x0, ttl 64, id 34503, offset 0, flags
> [DF],
> proto TCP (6), length 52)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [.], cksum
> 0xc485 (incorrect -> 0x6422), seq 2840, ack 2737, win 8934, options
> [nop,nop,TS val 5955218 ecr 510053070], length 0
> > 15:44:08.039984 IP (tos 0x0, ttl 64, id 34504, offset 0, flags
> [DF],
> proto TCP (6), length 256)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [P.],
> cksum
> 0xc551 (incorrect -> 0x8c10), seq 2840:3044, ack 2737, win 8934,
> options
> [nop,nop,TS val 5972054 ecr 510053070], length 204: NFS request xid
> 2452241769 200 getattr fh 0,1/53
> > 15:44:08.040888 IP (tos 0x0, ttl 63, id 17187, offset 0, flags
> [DF],
> proto TCP (6), length 356)
> > nfs.eecs.yorku.ca.nfs > j1.eecs.yorku.ca.822: Flags [P.],
> cksum
> 0xea43 (correct), seq 2737:3041, ack 3044, win 1769, options
> [nop,nop,TS
> val 510070082 ecr 5972054], length 304: NFS reply xid 2452241769
> reply
> ok 300 getattr NON 3 ids 0/-1374905249 sz 9481761
> > 15:44:08.040945 IP (tos 0x0, ttl 64, id 34505, offset 0, flags
> [DF],
> proto TCP (6), length 52)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [.], cksum
> 0xc485 (incorrect -> 0xdded), seq 3044, ack 3041, win 8934, options
> [nop,nop,TS val 5972054 ecr 510070082], length 0
> > 15:44:11.053805 IP (tos 0x0, ttl 64, id 34506, offset 0, flags
> [DF],
> proto TCP (6), length 256)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [P.],
> cksum
> 0xc551 (incorrect -> 0x883a), seq 3044:3248, ack 3041, win 8934,
> options
> [nop,nop,TS val 5975067 ecr 510070082], length 204: NFS request xid
> 2469018985 200 getattr fh 0,1/53
> > 15:44:11.054532 IP (tos 0x0, ttl 63, id 17188, offset 0, flags
> [DF],
> proto TCP (6), length 356)
> > nfs.eecs.yorku.ca.nfs > j1.eecs.yorku.ca.822: Flags [P.],
> cksum
> 0xebd7 (correct), seq 3041:3345, ack 3248, win 1769, options
> [nop,nop,TS
> val 510073095 ecr 5975067], length 304: NFS reply xid 2469018985
> reply
> ok 300 getattr NON 3 ids 0/-1374905249 sz 9481761
> > 15:44:11.054594 IP (tos 0x0, ttl 64, id 34507, offset 0, flags
> [DF],
> proto TCP (6), length 52)
> > j1.eecs.yorku.ca.822 > nfs.eecs.yorku.ca.nfs: Flags [.], cksum
> 0xc485 (incorrect -> 0xc466), seq 3248, ack 3345, win 8934, options
> [nop,nop,TS val 5975068 ecr 510073095], length 0
>
> At 15:43:51.028741 it appears that it's the server saying no. But
> how
> can I debug why it is denying access?
>
> I restart nfs-idmap on the NFS server, and run nfsidmap -c on the
> client. Here's the rpc.idmapd log during the chgrp -- the server
> knows
> me, my uid, my gid, the group I'm trying to change a file to
> (cmosp18),
> and its GID:
>
> > Feb 12 15:51:08 nfs rpc.idmapd[13383]: libnfsidmap: using domain:
> eecs.yorku.ca
> > Feb 12 15:51:08 nfs rpc.idmapd[13383]: libnfsidmap: Realms list:
> 'EECS.YORKU.CA'
> > Feb 12 15:51:08 nfs rpc.idmapd[13383]: libnfsidmap: loaded plugin
> /lib64/libnfsidmap/nsswitch.so for method nsswitch
> > Feb 12 15:51:08 nfs rpc.idmapd: rpc.idmapd: libnfsidmap: using
> domain: eecs.yorku.ca
> > Feb 12 15:51:08 nfs rpc.idmapd: rpc.idmapd: libnfsidmap: Realms
> list:
> 'EECS.YORKU.CA'
> > Feb 12 15:51:08 nfs rpc.idmapd[13384]: Expiration time is 600
> seconds.
> > Feb 12 15:51:08 nfs rpc.idmapd: rpc.idmapd: libnfsidmap: loaded
> plugin /lib64/libnfsidmap/nsswitch.so for method nsswitch
> > Feb 12 15:51:08 nfs rpc.idmapd[13384]: Opened
> /proc/net/rpc/nfs4.nametoid/channel
> > Feb 12 15:51:08 nfs rpc.idmapd[13384]: Opened
> /proc/net/rpc/nfs4.idtoname/channel
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: nfsdcb: authbuf=gss/krb5
> authtype=user
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: nfs4_uid_to_name: calling
> nsswitch->uid_to_name
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: nfs4_uid_to_name:
> nsswitch->uid_to_name returned 0
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: nfs4_uid_to_name: final
> return
> value is 0
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: Server : (user) id "1004" -
> >
> name "jas@eecs.yorku.ca"
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: nfsdcb: authbuf=gss/krb5
> authtype=group
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: nfs4_gid_to_name: calling
> nsswitch->gid_to_name
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: nfs4_gid_to_name:
> nsswitch->gid_to_name returned 0
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: nfs4_gid_to_name: final
> return
> value is 0
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: Server : (group) id "1000"
> ->
> name "tech@eecs.yorku.ca"
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: nfsdcb: authbuf=gss/krb5
> authtype=user
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: nfs4_uid_to_name: calling
> nsswitch->uid_to_name
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: nfs4_uid_to_name:
> nsswitch->uid_to_name returned 0
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: nfs4_uid_to_name: final
> return
> value is 0
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: Server : (user) id "0" ->
> name
> "root@eecs.yorku.ca"
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: nfsdcb: authbuf=gss/krb5
> authtype=group
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: nfs4_gid_to_name: calling
> nsswitch->gid_to_name
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: nfs4_gid_to_name:
> nsswitch->gid_to_name returned 0
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: nfs4_gid_to_name: final
> return
> value is 0
> > Feb 12 15:51:10 nfs rpc.idmapd[13384]: Server : (group) id "0" ->
> name "root@eecs.yorku.ca"
> > Feb 12 15:51:11 nfs rpc.idmapd[13384]: nfsdcb: authbuf=gss/krb5
> authtype=group
> > Feb 12 15:51:11 nfs rpc.idmapd[13384]: nfs4_name_to_gid: calling
> nsswitch->name_to_gid
> > Feb 12 15:51:11 nfs rpc.idmapd[13384]: nss_name_to_gid: name
> 'cmosp18@eecs.yorku.ca' domain 'eecs.yorku.ca': resulting localname
> 'cmosp18'
> > Feb 12 15:51:11 nfs rpc.idmapd[13384]: nss_name_to_gid: name
> 'cmosp18@eecs.yorku.ca' gid 100006
> > Feb 12 15:51:11 nfs rpc.idmapd[13384]: nfs4_name_to_gid:
> nsswitch->name_to_gid returned 0
> > Feb 12 15:51:11 nfs rpc.idmapd[13384]: nfs4_name_to_gid: final
> return
> value is 0
> > Feb 12 15:51:11 nfs rpc.idmapd[13384]: Server : (group) name
> "cmosp18@eecs.yorku.ca" -> id "100006"
>
> The client knows all the same information as well. I've had the
> client
> sitting for hours already, and I still can't chgrp the file. On the
> other hand, I rebooted the client, and then, right after login, I can
> chgrp the file.
>
> How can I debug this further?
>
> I was supposed to be moving 300+ systems to the domain this coming
> week,
> but I may have to abort that move if I can't solve this problem.
>
> Thanks very much for any assistance....
>
Have you enabled the '-g' option in rpc.mountd on your server (man 8
rpc.mountd)? Chances are that you are hitting the 16 group limit in
AUTH_SYS, so you probably want to ensure that the server is mapping
your user to the correct set of groups.
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com
next prev parent reply other threads:[~2021-02-13 0:58 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-13 0:13 NFS server issue after new AD group added to AD user Jason Keltz
2021-02-13 0:56 ` Trond Myklebust [this message]
2021-02-13 2:12 ` Jason Keltz
2021-02-13 14:28 ` Trond Myklebust
2021-02-13 14:35 ` Trond Myklebust
2021-02-13 17:45 ` Jason Keltz
2021-02-13 15:47 ` Jason Keltz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3795c23197b4f171b143821c033fa1687d3163e7.camel@hammerspace.com \
--to=trondmy@hammerspace.com \
--cc=jas@yorku.ca \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox