RE: [ewg] nfsrdma fails to write big file,

Vu Pham
2010-03-02T00:19:55Z

> -----Original Message-----
> From: Tom Tucker [mailto:tom-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org]
> Sent: Saturday, February 27, 2010 8:23 PM
> To: Vu Pham
> Cc: Roland Dreier; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Mahesh Siddheshwar;
> ewg-ZwoEplunGu1OwGhvXhtEPSCwEArCW2h5@public.gmane.org
> Subject: Re: [ewg] nfsrdma fails to write big file,
>
> Roland Dreier wrote:
> > > + /*
> > > + * Add room for frmr register and invalidate WRs
> > > + * Requests sometimes have two chunks, each chunk
> > > + * requires to have different frmr. The safest
> > > + * WRs required are max_send_wr * 6; however, we
> > > + * get send completions and poll fast enough, it
> > > + * is pretty safe to have max_send_wr * 4.
> > > + */
> > > + ep->rep_attr.cap.max_send_wr *= 4;
> >
> > Seems like a bad design if there is a possibility of work queue
> > overflow; if you're counting on events occurring in a particular
> order
> > or completions being handled "fast enough", then your design is
going
> to
> > fail in some high load situations, which I don't think you want.
> >
> >
>
> Vu,
>
> Would you please try the following:
>
> - Set the multiplier to 5
> - Set the number of buffer credits small as follows "echo 4 >
> /proc/sys/sunrpc/rdma_slot_table_entries"
> - Rerun your test and see if you can reproduce the problem?
>
> I did the above and was unable to reproduce, but I would like to see
if
> you can to convince ourselves that 5 is the right number.
>
>

Tom,

I did the above and can not reproduce either.

I think 5 is the right number; however, we should optimize it later.

-vu