From: Steve Wise <swise@opengridcomputing.com>
To: "J. Bruce Fields" <bfields@fieldses.org>
Cc: Yan Burman <yanb@mellanox.com>,
linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org,
Or Gerlitz <ogerlitz@mellanox.com>,
Tom Tucker <tom@opengridcomputing.com>
Subject: Re: NFS over RDMA crashing
Date: Wed, 06 Feb 2013 16:28:19 -0600 [thread overview]
Message-ID: <5112D903.9010601@opengridcomputing.com> (raw)
In-Reply-To: <20130206222435.GL16417@fieldses.org>
On 2/6/2013 4:24 PM, J. Bruce Fields wrote:
> On Wed, Feb 06, 2013 at 05:48:15PM +0200, Yan Burman wrote:
>> When killing mount command that got stuck:
>> -------------------------------------------
>>
>> BUG: unable to handle kernel paging request at ffff880324dc7ff8
>> IP: [<ffffffffa05f3dfb>] rdma_read_xdr+0x8bb/0xd40 [svcrdma]
>> PGD 1a0c063 PUD 32f82e063 PMD 32f2fd063 PTE 8000000324dc7161
>> Oops: 0003 [#1] PREEMPT SMP
>> Modules linked in: md5 ib_ipoib xprtrdma svcrdma rdma_cm ib_cm iw_cm
>> ib_addr nfsd exportfs netconsole ip6table_filter ip6_tables
>> iptable_filter ip_tables ebtable_nat nfsv3 nfs_acl ebtables x_tables
>> nfsv4 auth_rpcgss nfs lockd autofs4 sunrpc target_core_iblock
>> target_core_file target_core_pscsi target_core_mod configfs 8021q
>> bridge stp llc ipv6 dm_mirror dm_region_hash dm_log vhost_net
>> macvtap macvlan tun uinput iTCO_wdt iTCO_vendor_support kvm_intel
>> kvm crc32c_intel microcode pcspkr joydev i2c_i801 lpc_ich mfd_core
>> ehci_pci ehci_hcd sg ioatdma ixgbe mdio mlx4_ib ib_sa ib_mad ib_core
>> mlx4_en mlx4_core igb hwmon dca ptp pps_core button dm_mod ext3 jbd
>> sd_mod ata_piix libata uhci_hcd megaraid_sas scsi_mod
>> CPU 6
>> Pid: 4744, comm: nfsd Not tainted 3.8.0-rc5+ #4 Supermicro
>> X8DTH-i/6/iF/6F/X8DTH
>> RIP: 0010:[<ffffffffa05f3dfb>] [<ffffffffa05f3dfb>]
>> rdma_read_xdr+0x8bb/0xd40 [svcrdma]
>> RSP: 0018:ffff880324c3dbf8 EFLAGS: 00010297
>> RAX: ffff880324dc8000 RBX: 0000000000000001 RCX: ffff880324dd8428
>> RDX: ffff880324dc7ff8 RSI: ffff880324dd8428 RDI: ffffffff81149618
>> RBP: ffff880324c3dd78 R08: 000060f9c0000860 R09: 0000000000000001
>> R10: ffff880324dd8000 R11: 0000000000000001 R12: ffff8806299dcb10
>> R13: 0000000000000003 R14: 0000000000000001 R15: 0000000000000010
>> FS: 0000000000000000(0000) GS:ffff88063fc00000(0000) knlGS:0000000000000000
>> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> CR2: ffff880324dc7ff8 CR3: 0000000001a0b000 CR4: 00000000000007e0
>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Process nfsd (pid: 4744, threadinfo ffff880324c3c000, task ffff880330550000)
>> Stack:
>> ffff880324c3dc78 ffff880324c3dcd8 0000000000000282 ffff880631cec000
>> ffff880324dd8000 ffff88062ed33040 0000000124c3dc48 ffff880324dd8000
>> ffff88062ed33058 ffff880630ce2b90 ffff8806299e8000 0000000000000003
>> Call Trace:
>> [<ffffffffa05f466e>] svc_rdma_recvfrom+0x3ee/0xd80 [svcrdma]
>> [<ffffffff81086540>] ? try_to_wake_up+0x2f0/0x2f0
>> [<ffffffffa045963f>] svc_recv+0x3ef/0x4b0 [sunrpc]
>> [<ffffffffa0571db0>] ? nfsd_svc+0x740/0x740 [nfsd]
>> [<ffffffffa0571e5d>] nfsd+0xad/0x130 [nfsd]
>> [<ffffffffa0571db0>] ? nfsd_svc+0x740/0x740 [nfsd]
>> [<ffffffff81071df6>] kthread+0xd6/0xe0
>> [<ffffffff81071d20>] ? __init_kthread_worker+0x70/0x70
>> [<ffffffff814b462c>] ret_from_fork+0x7c/0xb0
>> [<ffffffff81071d20>] ? __init_kthread_worker+0x70/0x70
>> Code: 63 c2 49 8d 8c c2 18 02 00 00 48 39 ce 77 e1 49 8b 82 40 0a 00
>> 00 48 39 c6 0f 84 92 f7 ff ff 90 48 8d 50 f8 49 89 92 40 0a 00 00
>> <48> c7 40 f8 00 00 00 00 49 8b 82 40 0a 00 00 49 3b 82 30 0a 00
>> RIP [<ffffffffa05f3dfb>] rdma_read_xdr+0x8bb/0xd40 [svcrdma]
>> RSP <ffff880324c3dbf8>
>> CR2: ffff880324dc7ff8
>> ---[ end trace 06d0384754e9609a ]---
>>
>>
>> It seems that commit afc59400d6c65bad66d4ad0b2daf879cbff8e23e
>> "nfsd4: cleanup: replace rq_resused count by rq_next_page pointer"
>> is responsible for the crash (it seems to be crashing in
>> net/sunrpc/xprtrdma/svc_rdma_recvfrom.c:527)
>> It may be because I have CONFIG_DEBUG_SET_MODULE_RONX and
>> CONFIG_DEBUG_RODATA enabled. I did not try to disable them yet.
>>
>> When I moved to commit 79f77bf9a4e3dd5ead006b8f17e7c4ff07d8374e I
>> was no longer getting the server crashes,
>> so the reset of my tests were done using that point (it is somewhere
>> in the middle of 3.7.0-rc2).
> OK, so this part's clearly my fault--I'll work on a patch, but the
> rdma's use of the ->rq_pages array is pretty confusing.
Maybe Tom can shed some light?
next prev parent reply other threads:[~2013-02-06 22:28 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-02-06 15:48 NFS over RDMA crashing Yan Burman
2013-02-06 15:58 ` Steve Wise
2013-02-06 17:06 ` Jeff Becker
2013-02-07 15:54 ` Yan Burman
2013-02-06 22:24 ` J. Bruce Fields
2013-02-06 22:28 ` Steve Wise [this message]
2013-02-08 5:37 ` Tom Tucker
2013-02-07 16:41 ` J. Bruce Fields
2013-02-11 15:19 ` Yan Burman
2013-02-11 18:13 ` J. Bruce Fields
2013-02-15 15:27 ` J. Bruce Fields
2013-02-18 11:44 ` Yan Burman
2014-03-07 16:59 ` Steve Wise
2014-03-07 20:41 ` Steve Wise
2014-03-08 16:39 ` Steve Wise
2014-03-08 19:20 ` Steve Wise
2014-03-08 20:13 ` Steve Wise
2014-03-12 13:33 ` Jeff Layton
2014-03-12 14:05 ` Trond Myklebust
2014-03-12 14:22 ` Tom Tucker
2014-03-12 14:28 ` Jeffrey Layton
2014-03-12 15:03 ` Trond Myklebust
2014-03-12 15:29 ` Jeffrey Layton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5112D903.9010601@opengridcomputing.com \
--to=swise@opengridcomputing.com \
--cc=bfields@fieldses.org \
--cc=linux-nfs@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=ogerlitz@mellanox.com \
--cc=tom@opengridcomputing.com \
--cc=yanb@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).