* umount(,MNT_DETACH) for nfsv4 hangs when using sec=krb5 and network is down @ 2012-12-19 20:47 Orion Poplawski 2012-12-19 21:08 ` Myklebust, Trond 0 siblings, 1 reply; 9+ messages in thread From: Orion Poplawski @ 2012-12-19 20:47 UTC (permalink / raw) To: linux-nfs nfs4 mounts with sec=krb5 cannot be unmounted with the network down, even with umount -l because umount() with MNT_DETACH set will hang, presumably somewhere in the gss stack. A successful umount yields the following packet trace: 1 0.000000000 10.10.20.2 -> 10.10.10.1 NFS 218 V4 Call GETATTR FH:0x3b470ee7 2 0.000236000 10.10.10.1 -> 10.10.20.2 NFS 318 V4 Reply (Call In 1) GETATTR 3 0.000282000 10.10.20.2 -> 10.10.10.1 TCP 66 943 > nfs [ACK] Seq=153 Ack=253 Win=331 Len=0 TSval=3468186 TSecr=878557922 4 0.008761000 10.10.20.2 -> 10.10.10.1 TCP 66 943 > nfs [FIN, ACK] Seq=153 Ack=253 Win=331 Len=0 TSval=3468195 TSecr=878557922 5 0.008923000 10.10.10.1 -> 10.10.20.2 TCP 66 nfs > 943 [FIN, ACK] Seq=253 Ack=154 Win=683 Len=0 TSval=878557930 TSecr=3468195 6 0.008970000 10.10.20.2 -> 10.10.10.1 TCP 66 943 > nfs [ACK] Seq=154 Ack=254 Win=331 Len=0 TSval=3468195 TSecr=878557930 So my guess is that something in the gss stack is preventing the GETATTR call from succeeding as unmounting succeeds without sec=krb5. Although running rpc.gssd and rpcidmap with -vvvv does not appear to produce any output. A successful unmount produces: Dec 19 13:42:44 orca rpc.gssd[18495]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt27 Dec 19 13:42:44 orca rpc.gssd[18495]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt24 However, we need someway to be able to drop mounts after the network connection has been removed. This behavior is causing sever problems for our laptop and vpn users. Tested with: 3.6.11-3.fc18 nfs-utils-1.2.7-2.fc18 I've also filed https://bugzilla.redhat.com/show_bug.cgi?id=888942 -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder Office FAX: 303-415-9702 3380 Mitchell Lane orion@nwra.com Boulder, CO 80301 http://www.nwra.com ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: umount(,MNT_DETACH) for nfsv4 hangs when using sec=krb5 and network is down 2012-12-19 20:47 umount(,MNT_DETACH) for nfsv4 hangs when using sec=krb5 and network is down Orion Poplawski @ 2012-12-19 21:08 ` Myklebust, Trond 2012-12-19 21:50 ` Orion Poplawski 0 siblings, 1 reply; 9+ messages in thread From: Myklebust, Trond @ 2012-12-19 21:08 UTC (permalink / raw) To: Orion Poplawski; +Cc: linux-nfs@vger.kernel.org On Wed, 2012-12-19 at 20:47 +0000, Orion Poplawski wrote: > nfs4 mounts with sec=krb5 cannot be unmounted with the network down, even with > umount -l because umount() with MNT_DETACH set will hang, presumably somewhere > in the gss stack. > > A successful umount yields the following packet trace: > > 1 0.000000000 10.10.20.2 -> 10.10.10.1 NFS 218 V4 Call GETATTR FH:0x3b470ee7 > 2 0.000236000 10.10.10.1 -> 10.10.20.2 NFS 318 V4 Reply (Call In 1) GETATTR > 3 0.000282000 10.10.20.2 -> 10.10.10.1 TCP 66 943 > nfs [ACK] Seq=153 > Ack=253 Win=331 Len=0 TSval=3468186 TSecr=878557922 > 4 0.008761000 10.10.20.2 -> 10.10.10.1 TCP 66 943 > nfs [FIN, ACK] Seq=153 > Ack=253 Win=331 Len=0 TSval=3468195 TSecr=878557922 > 5 0.008923000 10.10.10.1 -> 10.10.20.2 TCP 66 nfs > 943 [FIN, ACK] Seq=253 > Ack=154 Win=683 Len=0 TSval=878557930 TSecr=3468195 > 6 0.008970000 10.10.20.2 -> 10.10.10.1 TCP 66 943 > nfs [ACK] Seq=154 > Ack=254 Win=331 Len=0 TSval=3468195 TSecr=878557930 > > So my guess is that something in the gss stack is preventing the GETATTR call > from succeeding as unmounting succeeds without sec=krb5. Although running > rpc.gssd and rpcidmap with -vvvv does not appear to produce any output. A > successful unmount produces: > > Dec 19 13:42:44 orca rpc.gssd[18495]: destroying client > /var/lib/nfs/rpc_pipefs/nfs/clnt27 > Dec 19 13:42:44 orca rpc.gssd[18495]: destroying client > /var/lib/nfs/rpc_pipefs/nfs/clnt24 > > However, we need someway to be able to drop mounts after the network connection > has been removed. This behavior is causing sever problems for our laptop and > vpn users. > > Tested with: > > 3.6.11-3.fc18 > nfs-utils-1.2.7-2.fc18 > > I've also filed https://bugzilla.redhat.com/show_bug.cgi?id=888942 No. What you need is a way to unmount _before_ you kill the network. Once the network is gone, you are in severe data loss territory, and you are entirely on your own dealing with that problem... Maybe one day we will get round to supporting offline mounts, but that's not the case today. -- Trond Myklebust Linux NFS client maintainer NetApp Trond.Myklebust@netapp.com www.netapp.com ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: umount(,MNT_DETACH) for nfsv4 hangs when using sec=krb5 and network is down 2012-12-19 21:08 ` Myklebust, Trond @ 2012-12-19 21:50 ` Orion Poplawski 2012-12-19 22:19 ` Myklebust, Trond 0 siblings, 1 reply; 9+ messages in thread From: Orion Poplawski @ 2012-12-19 21:50 UTC (permalink / raw) To: Myklebust, Trond; +Cc: linux-nfs@vger.kernel.org On 12/19/2012 02:08 PM, Myklebust, Trond wrote: > On Wed, 2012-12-19 at 20:47 퍍, Orion Poplawski wrote: >> >> However, we need someway to be able to drop mounts after the network connection >> has been removed. This behavior is causing sever problems for our laptop and >> vpn users. >> >> Tested with: >> >> 3.6.11-3.fc18 >> nfs-utils-1.2.7-2.fc18 >> >> I've also filed https://bugzilla.redhat.com/show_bug.cgi?id=888942 > > No. What you need is a way to unmount _before_ you kill the network. > Once the network is gone, you are in severe data loss territory, and you > are entirely on your own dealing with that problem... > > Maybe one day we will get round to supporting offline mounts, but that's > not the case today. > I agree (see https://bugzilla.gnome.org/show_bug.cgi?id=387832 for example), however it happens (and, because of lack of support as indicated by the bug, hard to prevent) and it seems unfortunate to then subject the user to hanging mounts (which will effectively lock up the desktop). It currently is possible for sec=sys mounts, so I thought it would be worth while making it work for sec=krb5 mounts. The same data loss issues are present for both. We have put work in the past into making umount work for offline nfs mounts (https://bugzilla.redhat.com/show_bug.cgi?id=820707). In fact that looks remarkably familiar :). [ 131.832005] umount.nfs4 D f1585bc8 0 1959 1958 0x00000080 [ 131.832005] f1585c34 00000086 0000ea8a f1585bc8 c045a297 f705f110 644b6440 0000001c [ 131.832005] f1585bd8 c0cd5080 c0cd5080 00000282 f1585c00 f7591080 f3a27110 f1585c24 [ 131.832005] 00000000 c0d2e280 00000282 00000246 f1585c00 c097a273 f1585c2c f7ee11c5 [ 131.832005] Call Trace: [ 131.832005] [<c045a297>] ? __internal_add_timer팞ﯿ䱜 [ 131.832005] [<c097a273>] ? _raw_spin_unlock_bh팝矿䱶 [ 131.832005] [<f7ee11c5>] ? rpc_wake_up_first팞맿䱵 [sunrpc] [ 131.832005] [<f7eda240>] ? rpc_show_tasks팝寓ﴱ햽 [sunrpc] [ 131.832005] [<c09794d3>] schedule팝럿䱺 [ 131.832005] [<f7ee064d>] rpc_wait_bit_killable팝鷿䱻 [sunrpc] [ 131.832005] [<c0977fc1>] __wait_on_bit팞痿䱻 [ 131.832005] [<f7ee0620>] ? __rpc_wait_for_completion_task팝䱷 [sunrpc] [ 131.832005] [<f7ee0620>] ? __rpc_wait_for_completion_task팝䱷 [sunrpc] [ 131.832005] [<c0978041>] out_of_line_wait_on_bit팞뗿䱻 [ 131.832005] [<c046c100>] ? autoremove_wake_function팞瓿䱹 [ 131.832005] [<f7ee198f>] __rpc_execute팝畿ﴱ� [sunrpc] [ 131.832005] [<c0507774>] ? mempool_alloc팞㣿䱵 [ 131.832005] [<f7ed8a50>] ? call_connect팟瓿䱽 [sunrpc] [ 131.832005] [<f7ed8a50>] ? call_connect팟瓿䱽 [sunrpc] [ 131.832005] [<c046c0a3>] ? wake_up_bit팝럿䱷 [ 131.832005] [<f7ee1ec8>] rpc_execute팞㳿䱼 [sunrpc] [ 131.832005] [<f7ed9929>] rpc_run_task팞緿䱻 [sunrpc] [ 131.832005] [<f7ed9a3c>] rpc_call_sync팝�䱺 [sunrpc] [ 131.832005] [<f8a402fc>] _nfs4_call_sync팝�䱹 [nfsv4] [ 131.832005] [<f8a403d5>] _nfs4_proc_getattr팟秿䱚 [nfsv4] [ 131.832005] [<f8a41bab>] nfs4_proc_getattr팝�䱺 [nfsv4] [ 131.832005] [<f897f891>] __nfs_revalidate_inode팟㗿䱶 [nfs] [ 131.832005] [<f897fbd2>] nfs_revalidate_inode팞뛿䱽 [nfs] [ 131.832005] [<f89793ef>] nfs_check_verifier팞䱼 [nfs] [ 131.832005] [<f897b4da>] nfs_lookup_revalidate팝魫ﴱ [nfs] [ 131.832005] [<c055f8cb>] ? follow_managed팝絯ﴱ� [ 131.832005] [<c0560000>] ? unlazy_walk팗䱵 [ 131.832005] [<f897c184>] nfs4_lookup_revalidate팝䱞 [nfs] [ 131.832005] [<c055fedc>] complete_walk팟䱜 [ 131.832005] [<c05611b3>] path_lookupat팞럿䱺 [ 131.832005] [<c05617ca>] do_path_lookup팝髿䱛 [ 131.832005] [<c0563df6>] user_path_at_empty팞㫿䱼 [ 131.832005] [<c097d440>] ? vmalloc_fault팝篫ﴱힾ [ 131.832005] [<c097d5f7>] ? do_page_fault팝寯ﴱ [ 131.832005] [<c0563e4f>] user_path_at팝忿䱷 [ 131.832005] [<c05707b1>] sys_umount팞㗿䱷 [ 131.832005] [<c04bd59c>] ? __audit_syscall_entry팖�䱶 [ 131.832005] [<c04bdac6>] ? __audit_syscall_exit팝匿ﴱ� [ 131.832005] [<c0980fdf>] sysenter_do_call팝盿䱶 I wonder if it never did get fixed for krb5 mounts then... Bah. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder Office FAX: 303-415-9702 3380 Mitchell Lane orion@nwra.com Boulder, CO 80301 http://www.nwra.com ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: umount(,MNT_DETACH) for nfsv4 hangs when using sec=krb5 and network is down 2012-12-19 21:50 ` Orion Poplawski @ 2012-12-19 22:19 ` Myklebust, Trond 2012-12-20 20:31 ` Orion Poplawski 0 siblings, 1 reply; 9+ messages in thread From: Myklebust, Trond @ 2012-12-19 22:19 UTC (permalink / raw) To: Orion Poplawski; +Cc: linux-nfs@vger.kernel.org On Wed, 2012-12-19 at 14:50 -0700, Orion Poplawski wrote: > On 12/19/2012 02:08 PM, Myklebust, Trond wrote: > > On Wed, 2012-12-19 at 20:47 퍍, Orion Poplawski wrote: > >> > >> However, we need someway to be able to drop mounts after the network connection > >> has been removed. This behavior is causing sever problems for our laptop and > >> vpn users. > >> > >> Tested with: > >> > >> 3.6.11-3.fc18 > >> nfs-utils-1.2.7-2.fc18 > >> > >> I've also filed https://bugzilla.redhat.com/show_bug.cgi?id=888942 > > > > No. What you need is a way to unmount _before_ you kill the network. > > Once the network is gone, you are in severe data loss territory, and you > > are entirely on your own dealing with that problem... > > > > Maybe one day we will get round to supporting offline mounts, but that's > > not the case today. > > > > I agree (see https://bugzilla.gnome.org/show_bug.cgi?id=387832 for example), > however it happens (and, because of lack of support as indicated by the bug, > hard to prevent) and it seems unfortunate to then subject the user to hanging > mounts (which will effectively lock up the desktop). It currently is possible > for sec=sys mounts, so I thought it would be worth while making it work for > sec=krb5 mounts. The same data loss issues are present for both. Any application which is already hanging on a file in that filesystem will continue to hang across a 'umount -l'. The only thing you are doing is preventing future attempts to access the filesystem. As I said above, this whole thing really needs to be handled as part of the suspend scripts and/or networkmanager... > We have put work in the past into making umount work for offline nfs mounts > (https://bugzilla.redhat.com/show_bug.cgi?id=820707). In fact that looks > remarkably familiar :). > > [ 131.832005] umount.nfs4 D f1585bc8 0 1959 1958 0x00000080 > [ 131.832005] f1585c34 00000086 0000ea8a f1585bc8 c045a297 f705f110 644b6440 > 0000001c > [ 131.832005] f1585bd8 c0cd5080 c0cd5080 00000282 f1585c00 f7591080 f3a27110 > f1585c24 > [ 131.832005] 00000000 c0d2e280 00000282 00000246 f1585c00 c097a273 f1585c2c > f7ee11c5 > [ 131.832005] Call Trace: > [ 131.832005] [<c045a297>] ? __internal_add_timer팞ﯿ䱜 > [ 131.832005] [<c097a273>] ? _raw_spin_unlock_bh팝矿䱶 > [ 131.832005] [<f7ee11c5>] ? rpc_wake_up_first팞맿䱵 [sunrpc] > [ 131.832005] [<f7eda240>] ? rpc_show_tasks팝寓ﴱ햽 [sunrpc] > [ 131.832005] [<c09794d3>] schedule팝럿䱺 > [ 131.832005] [<f7ee064d>] rpc_wait_bit_killable팝鷿䱻 [sunrpc] > [ 131.832005] [<c0977fc1>] __wait_on_bit팞痿䱻 > [ 131.832005] [<f7ee0620>] ? __rpc_wait_for_completion_task팝䱷 [sunrpc] > [ 131.832005] [<f7ee0620>] ? __rpc_wait_for_completion_task팝䱷 [sunrpc] > [ 131.832005] [<c0978041>] out_of_line_wait_on_bit팞뗿䱻 > [ 131.832005] [<c046c100>] ? autoremove_wake_function팞瓿䱹 > [ 131.832005] [<f7ee198f>] __rpc_execute팝畿ﴱ� [sunrpc] > [ 131.832005] [<c0507774>] ? mempool_alloc팞㣿䱵 > [ 131.832005] [<f7ed8a50>] ? call_connect팟瓿䱽 [sunrpc] > [ 131.832005] [<f7ed8a50>] ? call_connect팟瓿䱽 [sunrpc] > [ 131.832005] [<c046c0a3>] ? wake_up_bit팝럿䱷 > [ 131.832005] [<f7ee1ec8>] rpc_execute팞㳿䱼 [sunrpc] > [ 131.832005] [<f7ed9929>] rpc_run_task팞緿䱻 [sunrpc] > [ 131.832005] [<f7ed9a3c>] rpc_call_sync팝�䱺 [sunrpc] > [ 131.832005] [<f8a402fc>] _nfs4_call_sync팝�䱹 [nfsv4] > [ 131.832005] [<f8a403d5>] _nfs4_proc_getattr팟秿䱚 [nfsv4] > [ 131.832005] [<f8a41bab>] nfs4_proc_getattr팝�䱺 [nfsv4] > [ 131.832005] [<f897f891>] __nfs_revalidate_inode팟㗿䱶 [nfs] > [ 131.832005] [<f897fbd2>] nfs_revalidate_inode팞뛿䱽 [nfs] > [ 131.832005] [<f89793ef>] nfs_check_verifier팞䱼 [nfs] > [ 131.832005] [<f897b4da>] nfs_lookup_revalidate팝魫ﴱ [nfs] > [ 131.832005] [<c055f8cb>] ? follow_managed팝絯ﴱ� > [ 131.832005] [<c0560000>] ? unlazy_walk팗䱵 > [ 131.832005] [<f897c184>] nfs4_lookup_revalidate팝䱞 [nfs] > [ 131.832005] [<c055fedc>] complete_walk팟䱜 > [ 131.832005] [<c05611b3>] path_lookupat팞럿䱺 > [ 131.832005] [<c05617ca>] do_path_lookup팝髿䱛 > [ 131.832005] [<c0563df6>] user_path_at_empty팞㫿䱼 > [ 131.832005] [<c097d440>] ? vmalloc_fault팝篫ﴱힾ > [ 131.832005] [<c097d5f7>] ? do_page_fault팝寯ﴱ > [ 131.832005] [<c0563e4f>] user_path_at팝忿䱷 > [ 131.832005] [<c05707b1>] sys_umount팞㗿䱷 > [ 131.832005] [<c04bd59c>] ? __audit_syscall_entry팖�䱶 > [ 131.832005] [<c04bdac6>] ? __audit_syscall_exit팝匿ﴱ� > [ 131.832005] [<c0980fdf>] sysenter_do_call팝盿䱶 > > I wonder if it never did get fixed for krb5 mounts then... Commit eb96d5c97b0825d542e9c4ba5e0a22b519355166 (SUNRPC handle EKEYEXPIRED in call_refreshresult), which will be in 3.8-rc1 when Linus releases it, may help. -- Trond Myklebust Linux NFS client maintainer NetApp Trond.Myklebust@netapp.com www.netapp.com ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: umount(,MNT_DETACH) for nfsv4 hangs when using sec=krb5 and network is down 2012-12-19 22:19 ` Myklebust, Trond @ 2012-12-20 20:31 ` Orion Poplawski 2012-12-20 20:47 ` Myklebust, Trond 0 siblings, 1 reply; 9+ messages in thread From: Orion Poplawski @ 2012-12-20 20:31 UTC (permalink / raw) To: Myklebust, Trond; +Cc: linux-nfs@vger.kernel.org On 12/19/2012 03:19 PM, Myklebust, Trond wrote: > > Commit eb96d5c97b0825d542e9c4ba5e0a22b519355166 (SUNRPC handle > EKEYEXPIRED in call_refreshresult), which will be in 3.8-rc1 when Linus > releases it, may help. > FWIW - I cherry picked that into the latest Fedora rawhide kernel but no effect. Sounds like a nice patch though, the current hang forever behavior doesn't seem the trigger the needed "ah, need a new ticket" response. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder Office FAX: 303-415-9702 3380 Mitchell Lane orion@nwra.com Boulder, CO 80301 http://www.nwra.com ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: umount(,MNT_DETACH) for nfsv4 hangs when using sec=krb5 and network is down 2012-12-20 20:31 ` Orion Poplawski @ 2012-12-20 20:47 ` Myklebust, Trond 2012-12-20 21:52 ` Orion Poplawski 0 siblings, 1 reply; 9+ messages in thread From: Myklebust, Trond @ 2012-12-20 20:47 UTC (permalink / raw) To: Orion Poplawski; +Cc: linux-nfs@vger.kernel.org On Thu, 2012-12-20 at 13:31 -0700, Orion Poplawski wrote: > On 12/19/2012 03:19 PM, Myklebust, Trond wrote: > > > > Commit eb96d5c97b0825d542e9c4ba5e0a22b519355166 (SUNRPC handle > > EKEYEXPIRED in call_refreshresult), which will be in 3.8-rc1 when Linus > > releases it, may help. > > > > FWIW - I cherry picked that into the latest Fedora rawhide kernel but no > effect. Sounds like a nice patch though, the current hang forever behavior > doesn't seem the trigger the needed "ah, need a new ticket" response. > So does simply killing the rpc.gssd process help? -- Trond Myklebust Linux NFS client maintainer NetApp Trond.Myklebust@netapp.com www.netapp.com ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: umount(,MNT_DETACH) for nfsv4 hangs when using sec=krb5 and network is down 2012-12-20 20:47 ` Myklebust, Trond @ 2012-12-20 21:52 ` Orion Poplawski 2012-12-20 22:01 ` Myklebust, Trond 0 siblings, 1 reply; 9+ messages in thread From: Orion Poplawski @ 2012-12-20 21:52 UTC (permalink / raw) To: Myklebust, Trond; +Cc: linux-nfs@vger.kernel.org On 12/20/2012 01:47 PM, Myklebust, Trond wrote: > On Thu, 2012-12-20 at 13:31 -0700, Orion Poplawski wrote: >> On 12/19/2012 03:19 PM, Myklebust, Trond wrote: >>> >>> Commit eb96d5c97b0825d542e9c4ba5e0a22b519355166 (SUNRPC handle >>> EKEYEXPIRED in call_refreshresult), which will be in 3.8-rc1 when Linus >>> releases it, may help. >>> >> >> FWIW - I cherry picked that into the latest Fedora rawhide kernel but no >> effect. Sounds like a nice patch though, the current hang forever behavior >> doesn't seem the trigger the needed "ah, need a new ticket" response. >> > > So does simply killing the rpc.gssd process help? > Yes, if automount is already stopped (these are automounted directories). If automount is running, it still seems to hang. I think I'm going to need to spend some time talking to Ian. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder Office FAX: 303-415-9702 3380 Mitchell Lane orion@nwra.com Boulder, CO 80301 http://www.nwra.com ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: umount(,MNT_DETACH) for nfsv4 hangs when using sec=krb5 and network is down 2012-12-20 21:52 ` Orion Poplawski @ 2012-12-20 22:01 ` Myklebust, Trond 2013-02-13 17:58 ` Orion Poplawski 0 siblings, 1 reply; 9+ messages in thread From: Myklebust, Trond @ 2012-12-20 22:01 UTC (permalink / raw) To: Orion Poplawski; +Cc: linux-nfs@vger.kernel.org On Thu, 2012-12-20 at 14:52 -0700, Orion Poplawski wrote: > On 12/20/2012 01:47 PM, Myklebust, Trond wrote: > > On Thu, 2012-12-20 at 13:31 -0700, Orion Poplawski wrote: > >> On 12/19/2012 03:19 PM, Myklebust, Trond wrote: > >>> > >>> Commit eb96d5c97b0825d542e9c4ba5e0a22b519355166 (SUNRPC handle > >>> EKEYEXPIRED in call_refreshresult), which will be in 3.8-rc1 when Linus > >>> releases it, may help. > >>> > >> > >> FWIW - I cherry picked that into the latest Fedora rawhide kernel but no > >> effect. Sounds like a nice patch though, the current hang forever behavior > >> doesn't seem the trigger the needed "ah, need a new ticket" response. > >> > > > > So does simply killing the rpc.gssd process help? > > > > Yes, if automount is already stopped (these are automounted directories). If > automount is running, it still seems to hang. I think I'm going to need to > spend some time talking to Ian. > I'd suggest also taking a long hard look at rpc.gssd and making sure that it handles ENETUNREACH, ECONNREFUSED and friends correctly. I suspect right now it is just baling out of the upcall instead of completing it by propagating the error reply to the kernel. -- Trond Myklebust Linux NFS client maintainer NetApp Trond.Myklebust@netapp.com www.netapp.com ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: umount(,MNT_DETACH) for nfsv4 hangs when using sec=krb5 and network is down 2012-12-20 22:01 ` Myklebust, Trond @ 2013-02-13 17:58 ` Orion Poplawski 0 siblings, 0 replies; 9+ messages in thread From: Orion Poplawski @ 2013-02-13 17:58 UTC (permalink / raw) To: Myklebust, Trond; +Cc: linux-nfs@vger.kernel.org On 12/20/2012 03:01 PM, Myklebust, Trond wrote: > On Thu, 2012-12-20 at 14:52 -0700, Orion Poplawski wrote: >> On 12/20/2012 01:47 PM, Myklebust, Trond wrote: >>> On Thu, 2012-12-20 at 13:31 -0700, Orion Poplawski wrote: >>>> On 12/19/2012 03:19 PM, Myklebust, Trond wrote: >>>>> >>>>> Commit eb96d5c97b0825d542e9c4ba5e0a22b519355166 (SUNRPC handle >>>>> EKEYEXPIRED in call_refreshresult), which will be in 3.8-rc1 when Linus >>>>> releases it, may help. >>>>> >>>> >>>> FWIW - I cherry picked that into the latest Fedora rawhide kernel but no >>>> effect. Sounds like a nice patch though, the current hang forever behavior >>>> doesn't seem the trigger the needed "ah, need a new ticket" response. >>>> >>> >>> So does simply killing the rpc.gssd process help? >>> >> >> Yes, if automount is already stopped (these are automounted directories). If >> automount is running, it still seems to hang. I think I'm going to need to >> spend some time talking to Ian. >> > > I'd suggest also taking a long hard look at rpc.gssd and making sure > that it handles ENETUNREACH, ECONNREFUSED and friends correctly. I > suspect right now it is just baling out of the upcall instead of > completing it by propagating the error reply to the kernel. > Actually, I take that back - I'm not sure it's directly involved and killing rpc.gssd doesn't seem to be helping me now. I connected to rpc.gssd with strace, dropped the interface and tried to umount.nfs4 -l but rpc.gssd is still in poll and doesn't do anyting. kernel process trace shows: [ 2788.807017] umount.nfs4 D ffff88007cc13d40 0 3001 3000 0x00000084 [ 2788.807017] ffff8800361319a8 0000000000000082 ffff880036131fd8 0000000000013d40 [ 2788.807017] ffff880036131fd8 0000000000013d40 ffff8800773add80 ffff8800773add80 [ 2788.807017] ffff88007cfe2cb8 0000000000000082 ffffffffa0009cc0 ffff880036131a20 [ 2788.807017] Call Trace: [ 2788.807017] [<ffffffffa0009cc0>] ? __rpc_wait_for_completion_task팝䱷 [sunrpc] [ 2788.807017] [<ffffffff81634b39>] schedule팝뷿䱻 [ 2788.807017] [<ffffffffa0009cf5>] rpc_wait_bit_killable팝刺䱽 [sunrpc] [ 2788.807017] [<ffffffff816335a0>] __wait_on_bit팞듿䱽 [ 2788.807017] [<ffffffffa0001c50>] ? call_connect팟瓿䱽 [sunrpc] [ 2788.807017] [<ffffffffa0009cc0>] ? __rpc_wait_for_completion_task팝䱷 [sunrpc] [ 2788.807017] [<ffffffff81633707>] out_of_line_wait_on_bit팞ﯿ䱽 [ 2788.807017] [<ffffffff81080560>] ? autoremove_wake_function팞㓿䱸 [ 2788.807017] [<ffffffffa0001c50>] ? call_connect팟瓿䱽 [sunrpc] [ 2788.807017] [<ffffffffa0001c50>] ? call_connect팟瓿䱽 [sunrpc] [ 2788.807017] [<ffffffffa000ac7a>] __rpc_execute팝睫ﴱ� [sunrpc] [ 2788.807017] [<ffffffffa000bd65>] rpc_execute팞秿䱽 [sunrpc] [ 2788.807017] [<ffffffffa0002e60>] rpc_run_task팞䱽 [sunrpc] [ 2788.807017] [<ffffffffa0002ec3>] rpc_call_sync팞㟿䱚 [sunrpc] [ 2788.807017] [<ffffffffa01f5653>] _nfs4_call_sync팝矿䱶 [nfsv4] [ 2788.807017] [<ffffffffa01f4e50>] _nfs4_proc_getattr팖䱜 [nfsv4] [ 2788.807017] [<ffffffffa01f9b9e>] nfs4_proc_getattr팞ỿ䱻 [nfsv4] [ 2788.807017] [<ffffffffa01b67bc>] __nfs_revalidate_inode팟䱶 [nfs] [ 2788.807017] [<ffffffffa01b69a3>] nfs_revalidate_inode팞䱚 [nfs] [ 2788.807017] [<ffffffffa01afc60>] nfs_check_verifier팞瓿䱼 [nfs] [ 2788.807017] [<ffffffffa01b255b>] nfs_lookup_revalidate팝齯ﴱ [nfs] [ 2788.807017] [<ffffffffa01b2705>] nfs4_lookup_revalidate팝刺䱞 [nfs] [ 2788.807017] [<ffffffff811a18fb>] complete_walk팖�䱵 [ 2788.807017] [<ffffffff811a3310>] path_lookupat팞䱻 [ 2788.807017] [<ffffffff811a216f>] ? getname_flags팞䱵 [ 2788.807017] [<ffffffff811a3abb>] filename_lookup팝鯿䱜 [ 2788.807017] [<ffffffff811a67c4>] user_path_at_empty팞磿䱽 [ 2788.807017] [<ffffffff8117e4e6>] ? kmem_cache_free팞㫿䱵 [ 2788.807017] [<ffffffff8115d4e3>] ? remove_vma팞럿䱻 [ 2788.807017] [<ffffffff811a6811>] user_path_at팝痿䱶 [ 2788.807017] [<ffffffff811b57af>] sys_umount팝�䱷 [ 2788.807017] [<ffffffff81639f7e>] ? do_page_fault팗뿓ᵴ [ 2788.807017] [<ffffffff8163e419>] system_call_fastpath팝竿䱵 But every other process in schedule. The mount point gets "deleted": # grep mnt /proc/mounts earth:/export/home/orion /mnt\040(deleted) nfs4 rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5,clientaddr=10.10.11.101,local_lock=none,addr=10.10.10.1 0 0 but that's it. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder Office FAX: 303-415-9702 3380 Mitchell Lane orion@nwra.com Boulder, CO 80301 http://www.nwra.com ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2013-02-13 19:02 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-12-19 20:47 umount(,MNT_DETACH) for nfsv4 hangs when using sec=krb5 and network is down Orion Poplawski 2012-12-19 21:08 ` Myklebust, Trond 2012-12-19 21:50 ` Orion Poplawski 2012-12-19 22:19 ` Myklebust, Trond 2012-12-20 20:31 ` Orion Poplawski 2012-12-20 20:47 ` Myklebust, Trond 2012-12-20 21:52 ` Orion Poplawski 2012-12-20 22:01 ` Myklebust, Trond 2013-02-13 17:58 ` Orion Poplawski
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.