From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id n753xktb213779 for ; Tue, 4 Aug 2009 22:59:46 -0500 Received: from mail.vpac.org (localhost [127.0.0.1]) by cuda.sgi.com (Spam Firewall) with ESMTP id 321623A7162 for ; Tue, 4 Aug 2009 21:00:35 -0700 (PDT) Received: from mail.vpac.org (mail.vpac.org [202.158.218.6]) by cuda.sgi.com with ESMTP id VKMYtQD6VXdYrZpH for ; Tue, 04 Aug 2009 21:00:35 -0700 (PDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.vpac.org (Postfix) with ESMTP id 676C846D010F for ; Wed, 5 Aug 2009 14:00:33 +1000 (EST) Received: from mail.vpac.org ([127.0.0.1]) by localhost (mail.vpac.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TQIfVqB5630O for ; Wed, 5 Aug 2009 14:00:29 +1000 (EST) Received: from mail.vpac.org (mail.vpac.org [202.158.218.6]) by mail.vpac.org (Postfix) with ESMTP id 9705246D010E for ; Wed, 5 Aug 2009 14:00:29 +1000 (EST) Date: Wed, 5 Aug 2009 14:00:29 +1000 (EST) From: Chris Samuel Message-ID: <1367391532.793061249444829356.JavaMail.root@mail.vpac.org> In-Reply-To: <816101050.793011249444717925.JavaMail.root@mail.vpac.org> Subject: XFS filesystem shutting down on linux 2.6.28.10 (xfs_rename) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_68988_1821615652.1249444829354" List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: xfs@oss.sgi.com ------=_Part_68988_1821615652.1249444829354 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi folks, I believe we've been hitting the same issue that Gabriel Barazer reported in 2.6.28.9 on the 22nd of July on our NFS server for our HPC Linux clusters. Here is the backtrace we got this morning: Aug 5 11:44:27 stg7 kernel: [680506.864506] Pid: 5271, comm: nfsd Not tainted 2.6.28.10-vpac-1 #1 Aug 5 11:44:27 stg7 kernel: [680506.864508] Call Trace: Aug 5 11:44:27 stg7 kernel: [680506.864541] [] xfs_rename+0x5ac/0x5af [xfs] Aug 5 11:44:27 stg7 kernel: [680506.864567] [] xfs_trans_cancel+0x56/0xee [xfs] Aug 5 11:44:27 stg7 kernel: [680506.864589] [] xfs_rename+0x5ac/0x5af [xfs] Aug 5 11:44:27 stg7 kernel: [680506.864609] [] xfs_vn_rename+0x61/0x69 [xfs] Aug 5 11:44:27 stg7 kernel: [680506.864615] [] vfs_rename+0x28a/0x404 Aug 5 11:44:27 stg7 kernel: [680506.864642] [] nfsd_rename+0x2ba/0x35f [nfsd] Aug 5 11:44:27 stg7 kernel: [680506.864654] [] nfsd3_proc_rename+0x120/0x131 [nfsd] Aug 5 11:44:27 stg7 kernel: [680506.864681] [] nfsd_dispatch+0xdd/0x1b9 [nfsd] Aug 5 11:44:27 stg7 kernel: [680506.864706] [] svc_process+0x3e6/0x70e [sunrpc] Aug 5 11:44:27 stg7 kernel: [680506.864711] [] default_wake_function+0x0/0xe Aug 5 11:44:27 stg7 kernel: [680506.864717] [] __down_read+0x15/0x99 Aug 5 11:44:27 stg7 kernel: [680506.864740] [] nfsd+0x1a0/0x26c [nfsd] Aug 5 11:44:27 stg7 kernel: [680506.864750] [] nfsd+0x0/0x26c [nfsd] Aug 5 11:44:27 stg7 kernel: [680506.864754] [] kthread+0x47/0x73 Aug 5 11:44:27 stg7 kernel: [680506.864757] [] schedule_tail+0x27/0x60 Aug 5 11:44:27 stg7 kernel: [680506.864761] [] child_rip+0xa/0x11 Aug 5 11:44:27 stg7 kernel: [680506.864764] [] kthread+0x0/0x73 Aug 5 11:44:27 stg7 kernel: [680506.864766] [] child_rip+0x0/0x11 Aug 5 11:44:27 stg7 kernel: [680506.864770] xfs_force_shutdown(md25,0x8) called from line 1165 of file fs/xfs/xfs _trans.c. Return address = 0xffffffffa032d7ac Here's another backtrace from a week ago (same kernel): Jul 28 13:27:22 stg7 kernel: [3528649.232700] Pid: 5414, comm: nfsd Not tainted 2.6.28.10-vpac-1 #1 Jul 28 13:27:22 stg7 kernel: [3528649.232702] Call Trace: Jul 28 13:27:22 stg7 kernel: [3528649.232734] [] xfs_rename+0x5ac/0x5af [xfs] Jul 28 13:27:22 stg7 kernel: [3528649.232756] [] xfs_trans_cancel+0x56/0xee [xfs] Jul 28 13:27:22 stg7 kernel: [3528649.232778] [] xfs_rename+0x5ac/0x5af [xfs] Jul 28 13:27:22 stg7 kernel: [3528649.232809] [] xfs_vn_rename+0x61/0x69 [xfs] Jul 28 13:27:22 stg7 kernel: [3528649.232814] [] vfs_rename+0x28a/0x404 Jul 28 13:27:22 stg7 kernel: [3528649.232829] [] nfsd_rename+0x2ba/0x35f [nfsd] Jul 28 13:27:22 stg7 kernel: [3528649.232855] [] nfsd3_proc_rename+0x120/0x131 [nfsd] Jul 28 13:27:22 stg7 kernel: [3528649.232879] [] nfsd_dispatch+0xdd/0x1b9 [nfsd] Jul 28 13:27:22 stg7 kernel: [3528649.232902] [] svc_process+0x3e6/0x70e [sunrpc] Jul 28 13:27:22 stg7 kernel: [3528649.232907] [] default_wake_function+0x0/0xe Jul 28 13:27:22 stg7 kernel: [3528649.232912] [] __down_read+0x15/0x99 Jul 28 13:27:22 stg7 kernel: [3528649.232922] [] nfsd+0x1a0/0x26c [nfsd] Jul 28 13:27:22 stg7 kernel: [3528649.232948] [] nfsd+0x0/0x26c [nfsd] Jul 28 13:27:22 stg7 kernel: [3528649.232952] [] kthread+0x47/0x73 Jul 28 13:27:22 stg7 kernel: [3528649.232955] [] schedule_tail+0x27/0x60 Jul 28 13:27:22 stg7 kernel: [3528649.232959] [] child_rip+0xa/0x11 Jul 28 13:27:22 stg7 kernel: [3528649.232962] [] kthread+0x0/0x73 Jul 28 13:27:22 stg7 kernel: [3528649.232964] [] child_rip+0x0/0x11 Jul 28 13:27:22 stg7 kernel: [3528649.232967] xfs_force_shutdown(md25,0x8) called from line 1165 of file fs/xfs/xf s_trans.c. Return address = 0xffffffffa03737ac This kernel is built with XFS as a kernel module so I've been able to attach the objdump output that Eric Sandeen had originally requested from Gabriel. Like Gabriel we're stuck on 2.6.28.x as the last working NFS exporting XFS kernel due to kernel bug #13375 (the radix bug), so I hope this helps! cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency ------=_Part_68988_1821615652.1249444829354 Content-Type: text/plain; name=xfs_objdump.txt Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=xfs_objdump.txt 00000000000452ed : 452fc: e8 00 00 00 00 callq 45301 45309: 74 19 je 45324 45310: 74 08 je 4531a 45315: e8 00 00 00 00 callq 4531a 45322: 75 dd jne 45301 0000000000045329 : 4538e: 74 21 je 453b1 45393: 75 1c jne 453b1 4539a: 74 15 je 453b1 453ab: 0f 87 e6 04 00 00 ja 45897 453fd: e8 00 00 00 00 callq 45402 45466: e8 00 00 00 00 callq 4546b 45474: 75 32 jne 454a8 45499: e8 00 00 00 00 callq 4549e 454aa: 74 0f je 454bb 454b6: e9 15 04 00 00 jmpq 458d0 454d6: 74 12 je 454ea 454e5: e9 e6 03 00 00 jmpq 458d0 454f9: e8 00 00 00 00 callq 454fe 45507: 74 3f je 45548 4551d: 74 29 je 45548 4552c: e8 bc fd ff ff callq 452ed 4553e: e8 00 00 00 00 callq 45543 45543: e9 4f 03 00 00 jmpq 45897 45560: e8 00 00 00 00 callq 45565 4556a: 74 1e je 4558a 45585: e8 00 00 00 00 callq 4558a 455a9: e8 00 00 00 00 callq 455ae 455b1: 74 3a je 455ed 455cb: e8 00 00 00 00 callq 455d0 455e2: 0f 85 be 00 00 00 jne 456a6 455e8: e9 9b 00 00 00 jmpq 45688 45601: e8 00 00 00 00 callq 45606 4560a: 0f 85 a0 02 00 00 jne 458b0 4563a: e8 00 00 00 00 callq 4563f 45644: 0f 84 66 02 00 00 je 458b0 4564c: 0f 85 65 02 00 00 jne 458b7 4565a: e8 00 00 00 00 callq 4565f 45664: 0f 84 bb 00 00 00 je 45725 4566d: 0f 84 b2 00 00 00 je 45725 4567e: e8 00 00 00 00 callq 45683 45683: e9 93 00 00 00 jmpq 4571b 4568b: e8 00 00 00 00 callq 45690 45692: 0f 84 13 02 00 00 je 458ab 456a0: 0f 87 05 02 00 00 ja 458ab 456d0: e8 00 00 00 00 callq 456d5 456d9: 0f 85 d8 01 00 00 jne 458b7 456e7: e8 00 00 00 00 callq 456ec 456f7: e8 00 00 00 00 callq 456fc 45700: 0f 85 b1 01 00 00 jne 458b7 45709: 74 1a je 45725 45716: e8 00 00 00 00 callq 4571b 4571f: 0f 85 92 01 00 00 jne 458b7 4572d: 74 3e je 4576d 45732: 74 39 je 4576d 4575e: e8 00 00 00 00 callq 45763 45767: 0f 85 4a 01 00 00 jne 458b7 45777: e8 00 00 00 00 callq 4577c 4577f: 74 24 je 457a5 45784: 75 05 jne 4578b 45789: 74 1a je 457a5 45796: e8 00 00 00 00 callq 4579b 4579f: 0f 85 12 01 00 00 jne 458b7 457cf: e8 00 00 00 00 callq 457d4 457d8: 0f 85 d9 00 00 00 jne 458b7 457e6: e8 00 00 00 00 callq 457eb 45802: e8 00 00 00 00 callq 45807 4580a: 74 1d je 45829 45824: e8 00 00 00 00 callq 45829 45839: 74 0c je 45847 4585f: e8 00 00 00 00 callq 45864 45868: 74 17 je 45881 4586d: e8 00 00 00 00 callq 45872 4587f: eb 4f jmp 458d0 45890: e8 00 00 00 00 callq 45895 458b5: eb 05 jmp 458bc 458c1: e8 00 00 00 00 callq 458c6 458d0: e8 00 00 00 00 callq 458d5 458d5: eb c0 jmp 45897 46738: e9 00 00 00 00 jmpq 4673d 000000000004673d : 4674f: 74 0d je 4675e 46762: 74 48 je 467ac 4676b: 75 3f jne 467ac 4678e: e8 00 00 00 00 callq 46793 467a7: e8 00 00 00 00 callq 467ac 467c5: 74 06 je 467cd 467d4: 74 13 je 467e9 467e4: e8 00 00 00 00 callq 467e9 46812: e8 00 00 00 00 callq 46817 4681a: e8 00 00 00 00 callq 4681f ------=_Part_68988_1821615652.1249444829354 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ------=_Part_68988_1821615652.1249444829354--