From mboxrd@z Thu Jan 1 00:00:00 1970 From: Lukas Hejtmanek Subject: Deadlock in NFSv4 in all kernels Date: Fri, 7 May 2010 17:39:20 +0200 Message-ID: <20100507153920.GP28167@ics.muni.cz> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-2 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: salvet-8qz54MUs51PtwjQa/ONI9g@public.gmane.org To: linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-fsdevel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Return-path: Content-Disposition: inline Sender: linux-nfs-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-Id: linux-fsdevel.vger.kernel.org Hi, I encountered the following problem. We use short expiration time for kerberos contexts created by rpc.gssd (some patches were included in ma= inline nfs-utils). In particular, we use 120secs expiration time. Now, I run application that eats 80% of available RAM. Then I run 10 pa= rallel dd processes that write data into NFS4 volume with sec=3Dkrb5. As soon as the kerberos context expires (i.e., up to 120 secs), the who= le system gets stuck in do_page_fault and succesive functions. It is becau= se there is no free memory in kernel, all free memory is used as cache for= NFS4 (due to dd traffic), kernel ask NFS to write back its pages but NFS can= not do anything as it is missing valid context. NFS contacts rpc.gssd to provi= de a renewed context, the rpc.gssd does not provide the context as it need= s some memory to scan /tmp for a ticket. I.e., it deadlocks. Longer context expiration time is no real solution as it only makes the deadlock less often.=20 Any ideas what can be done here? (Please cc me.) We could preallocate s= ome memory in rpc.gssd and use mlockall but not sure whether this proctects= also kernel malloc for things related to rpc.gssd and context creation (new = file descriptors and so on).=20 This is seen in 2.6.32 kernel but most probably this is related to all = kernel versions. rpc.gssd and all dd processes are stuck in: May 6 15:33:10 skirit20 kernel: [84087.788019] rpc.gssd D 6758d88= 1 0 26864 1 0x00000000 May 6 15:33:10 skirit20 kernel: [84087.788019] c280594c 00000086 f94c= 3d6c 6758d881 00004c5a 0f3e1c27 c1d868c0 c1c068dc May 6 15:33:10 skirit20 kernel: [84087.788019] c07d8880 c07d8880 f675= 2130 f67523d4 c1c06880 00000000 c1c06880 00000000 May 6 15:33:10 skirit20 kernel: [84087.788019] f6d7c8f0 f67523d4 f675= 2130 c1c06cd4 c1c06880 c2805960 c052101f 00000000 May 6 15:33:10 skirit20 kernel: [84087.788019] Call Trace: May 6 15:33:10 skirit20 kernel: [84087.788019] [] io_schedu= le+0x6f/0xc0 May 6 15:33:10 skirit20 kernel: [84087.788019] [] nfs_wait_= bit_uninterruptible+0x5/0x10 [nfs] May 6 15:33:10 skirit20 kernel: [84087.788019] [] __wait_on= _bit+0x47/0x70 May 6 15:33:10 skirit20 kernel: [84087.788019] [] out_of_li= ne_wait_on_bit+0xb3/0xd0 May 6 15:33:10 skirit20 kernel: [84087.788019] [] nfs_wait_= on_request+0x23/0x30 [nfs] May 6 15:33:10 skirit20 kernel: [84087.788019] [] nfs_sync_= mapping_wait+0xea/0x200 [nfs] May 6 15:33:10 skirit20 kernel: [84087.788019] [] nfs_wb_pa= ge_priority+0xae/0x170 [nfs] May 6 15:33:10 skirit20 kernel: [84087.788019] [] nfs_relea= se_page+0x5c/0x70 [nfs] May 6 15:33:10 skirit20 kernel: [84087.788019] [] try_to_re= lease_page+0x2b/0x40 May 6 15:33:10 skirit20 kernel: [84087.788019] [] shrink_pa= ge_list+0x37f/0x4b0 May 6 15:33:10 skirit20 kernel: [84087.788019] [] shrink_in= active_list+0x2ce/0x6c0 May 6 15:33:10 skirit20 kernel: [84087.788019] [] shrink_zo= ne+0x1c8/0x260 May 6 15:33:10 skirit20 kernel: [84087.788019] [] shrink_zo= nes+0x4e/0xe0 May 6 15:33:10 skirit20 kernel: [84087.788019] [] do_try_to= _free_pages+0x75/0x2e0 May 6 15:33:10 skirit20 kernel: [84087.788019] [] try_to_fr= ee_pages+0x86/0xa0 May 6 15:33:10 skirit20 kernel: [84087.788019] [] __alloc_p= ages_slowpath+0x164/0x470 May 6 15:33:10 skirit20 kernel: [84087.788019] [] __alloc_p= ages_nodemask+0x118/0x120 May 6 15:33:10 skirit20 kernel: [84087.788019] [] do_anonym= ous_page+0x100/0x240 May 6 15:33:10 skirit20 kernel: [84087.788019] [] handle_mm= _fault+0x34a/0x3d0 May 6 15:33:10 skirit20 kernel: [84087.788019] [] do_page_f= ault+0x174/0x370 May 6 15:33:10 skirit20 kernel: [84087.788019] [] error_cod= e+0x66/0x70 May 6 15:33:10 skirit20 kernel: [84087.788019] [] file_read= _actor+0x32/0xf0 May 6 15:33:10 skirit20 kernel: [84087.788019] [] do_generi= c_file_read+0x3af/0x4c0 May 6 15:33:10 skirit20 kernel: [84087.788019] [] generic_f= ile_aio_read+0xb1/0x210 May 6 15:33:10 skirit20 kernel: [84087.788019] [] do_sync_r= ead+0xd5/0x120 May 6 15:33:10 skirit20 kernel: [84087.788019] [] vfs_read+= 0x9b/0x110 May 6 15:33:10 skirit20 kernel: [84087.788019] [] sys_read+= 0x41/0x80 May 6 15:33:10 skirit20 kernel: [84087.788019] [] sysenter_= do_call+0x12/0x22 May 6 15:33:10 skirit20 kernel: [84087.788019] [] 0xffffe43= 0 --=20 Luk=E1=B9 Hejtm=E1nek -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html