RE: RFC: CQ pools and implicit CQ resource allocation

["Steve Wise" <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>]

> -----Original Message-----
> From: Sagi Grimberg [mailto:sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org]
> Sent: Monday, September 12, 2016 3:08 PM
> To: Steve Wise; 'Christoph Hellwig'; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: Re: RFC: CQ pools and implicit CQ resource allocation
> 
> 
> >> One other note that I wanted to raise for the folks interested in this
> >> is that with the RDMA core owning the completion queue pools, different
> >> ULPs can easily share the same completion queue (given that it uses
> >> the same poll context). For example, nvme-rdma host, iser and srp
> >> initiators can end up using the same completion queues (if running
> >> simultaneously on the same machine).
> >>
> >> Up until now, I couldn't think of anything that can introduce a problem
> >> with that but maybe someone else will...
> >
> > It would be useful to provide details on how many CQs get created and of
what
> > size for an uber iSER/NVMF/SRP initiator/host and target.
> 
> Are you talking about some debugfs layout?
>

No, just a matrix showing how the CQs scale out when shared among these three
ULPs on a machine with X cores, for example.   Just to visualize if the number
of CQs and their sizes are reduced by this new series or increased...
 
> > One concern I have is that cxgb4 CQs require contiguous memory,  So a scheme
> > like CQ pooling might cause resource problems on large core systems.
> 
> Note that the CQ allocation will never exceed the device max_cqe cap.

If this series causes, say, 2X the amount of memory needed for CQs vs the
existing private CQ approach, then that impacts how many CQs can be allocated,
due to limits on the amount of memory that can be allocated system-wide via
dma_alloc_coherent(), which is what cxgb4 uses to allocate queue memory.

So I'm just voicing the concern this design can possibly reduce the overall
number of CQs available on a given system.  It is probably not a big deal
though, but I don't have a good visualization of how much more memory this
proposed series would incur...

Stevo

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html