Re: NFS over RDMA crashing

Trond Myklebust
2014-03-12T14:05:24Z

On Mar 12, 2014, at 9:33, Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> On Sat, 08 Mar 2014 14:13:44 -0600
> Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org> wrote:
>
>> On 3/8/2014 1:20 PM, Steve Wise wrote:
>>>
>>>> I removed your change and started debugging original crash that
>>>> happens on top-o-tree. Seems like rq_next_pages is screwed up. It
>>>> should always be >= rq_respages, yes? I added a BUG_ON() to assert
>>>> this in rdma_read_xdr() we hit the BUG_ON(). Look
>>>>
>>>> crash> svc_rqst.rq_next_page 0xffff8800b84e6000
>>>> rq_next_page = 0xffff8800b84e6228
>>>> crash> svc_rqst.rq_respages 0xffff8800b84e6000
>>>> rq_respages = 0xffff8800b84e62a8
>>>>
>>>> Any ideas Bruce/Tom?
>>>>
>>>
>>> Guys, the patch below seems to fix the problem. Dunno if it is
>>> correct though. What do you think?
>>>
>>> diff --git a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
>>> b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
>>> index 0ce7552..6d62411 100644
>>> --- a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
>>> +++ b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
>>> @@ -90,6 +90,7 @@ static void rdma_build_arg_xdr(struct svc_rqst *rqstp,
>>> sge_no++;
>>> }
>>> rqstp->rq_respages = &rqstp->rq_pages[sge_no];
>>> + rqstp->rq_next_page = rqstp->rq_respages;
>>>
>>> /* We should never run out of SGE because the limit is defined to
>>> * support the max allowed RPC data length
>>> @@ -276,6 +277,7 @@ static int fast_reg_read_chunks(struct
>>> svcxprt_rdma *xprt,
>>>
>>> /* rq_respages points one past arg pages */
>>> rqstp->rq_respages = &rqstp->rq_arg.pages[page_no];
>>> + rqstp->rq_next_page = rqstp->rq_respages;
>>>
>>> /* Create the reply and chunk maps */
>>> offset = 0;
>>>
>>>
>>
>> While this patch avoids the crashing, it apparently isn't correct...I'm
>> getting IO errors reading files over the mount. :)
>>
>
> I hit the same oops and tested your patch and it seems to have fixed
> that particular panic, but I still see a bunch of other mem corruption
> oopses even with it. I'll look more closely at that when I get some
> time.
>
> FWIW, I can easily reproduce that by simply doing something like:
>
> $ dd if=/dev/urandom of=/file/on/nfsordma/mount bs=4k count=1
>
> I'm not sure why you're not seeing any panics with your patch in place.
> Perhaps it's due to hw differences between our test rigs.
>
> The EIO problem that you're seeing is likely the same client bug that
> Chuck recently fixed in this patch:
>
> [PATCH 2/8] SUNRPC: Fix large reads on NFS/RDMA
>
> AIUI, Trond is merging that set for 3.15, so I'd make sure your client
> has those patches when testing.
>

Nothing is in my queue yet.

_________________________________
Trond Myklebust
Linux NFS client maintainer, PrimaryData
trond.myklebust-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org