Re: [PATCHv12 0/3] rdmacg: IB/core: rdma controller support

[Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>]

[-- Attachment #1: Type: text/plain, Size: 6847 bytes --]

On Mon, Oct 10, 2016 at 02:05:27PM +0530, Parav Pandit wrote:
> On Mon, Oct 10, 2016 at 1:03 PM, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> > On Mon, Oct 10, 2016 at 11:59:45AM +0530, Parav Pandit wrote:
> >> Hi Leon,
> >>
> >> On Mon, Oct 10, 2016 at 10:16 AM, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> >> > On Thu, Oct 06, 2016 at 07:19:24PM +0530, Parav Pandit wrote:
> >> >> Hi Leon,
> >> >>
> >> >> On Wed, Oct 5, 2016 at 4:52 PM, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> >> >> > On Wed, Aug 31, 2016 at 02:07:24PM +0530, Parav Pandit wrote:
> >> >> >> rdmacg: IB/core: rdma controller support
> >> >> >>
> >> >> >> Overview:
> >> >> >> Currently user space applications can easily take away all the rdma
> >> >> >> device specific resources such as AH, CQ, QP, MR etc. Due to which other
> >> >> >> applications in other cgroup or kernel space ULPs may not even get chance
> >> >> >> to allocate any rdma resources. This results into service unavailibility.
> >> >> >>
> >> >> >> RDMA cgroup addresses this issue by allowing resource accounting,
> >> >> >> limit enforcement on per cgroup, per rdma device basis.
> >> >> >>
> >> >> >> RDMA uverbs layer will enforce limits on well defined RDMA verb
> >> >> >> resources without any HCA vendor device driver involvement.
> >> >> >>
> >> >> >> RDMA uverbs layer will not do limit enforcement of HCA hw vendor
> >> >> >> specific resources. Instead rdma cgroup provides set of APIs
> >> >> >> through which vendor specific drivers can do resource accounting
> >> >> >> by making use of rdma cgroup.
> >> >> >
> >> >> > Hi Parav,
> >> >> > I want to propose an extension to the RDMA cgroup which can be done as
> >> >> > follow-up patches.
> >> >> >
> >> >> > Let's add new global type, which will control whole HCA (for example in percentages). It will
> >> >> > allow natively define new objects without need to introduce them to the user.
> >> >> >
> >> >> In other cgroup such as CPU, this is done using cpu.weight API. Where
> >> >> percentage or weight is configured by the user.
> >> >> In this mode, resources taken away from other cgroup proportionately.
> >> >> It works for cpu because its mainly stateless resource unlike rdma
> >> >> resources.
> >> >> So if we want to simplify user configuration similarly,
> >> >> percentage/weight configuration can be extended.
> >> >> This way they need not be introduced to users.
> >> >> I hope your definition of "user" is actual end-user and not rdma cgroup. Right?
> >> >
> >> > Yes, "user" -> "admin".
> >> > I think that percentage is more intuitive to them and will be much easier to
> >> > explain how to use it. I always have in mind "swappiness" field and the
> >> > numerous questions on how to configure it.
> >> >
> >> >> In other words, new object should be still added as new enum value in
> >> >> rdma_cgroup.h?
> >> >
> >> > Yes, I had in mind something like IB_CGROUP_HCA, this is why it can be
> >> > done as a future work after accepting current patches.
> >> >
> >> What I meant is,
> >> today we have RDMACG_VERB_RESOURCE_QP etc,
> >> We will additionally have RDMACG_VERB_RESOURCE_INDIRECT_TBL etc in
> >> cgroup_rdma.h.
> >> So that its available for admin to override it.
> >
> > IMHO, we are talking about the same. My global HCA object will be
> > overwritten by more granular VERBS objects in case they exists.
> >
> >>
> >> >> Only than it can be overwritten by specific UVERBs type as you
> >> >> described below. I think thats what you meant as you described below.
> >> >
> >> > Exactly.
> >> >
> >> >>
> >> >> Otherwise charging/uncharging this new percentage resource can get messy.
> >> >
> >> > Agree
> >> >
> >> >>
> >> >> > This HCA share will be overwritten by specific UVERBS types which you
> >> >> > already defined.
> >> >> >
> >> >> > What do you think?
> >> >>
> >> >> So to refine your proposal from cgroup perspective, instead of adding
> >> >> new resource type in rdma_cgroup.h for percentage, I prefer to have
> >> >>
> >> >> Existing
> >> >> 1. rdma.max
> >> >> 2. rdma.current
> >> >> New,
> >> >> 3. rdma.weight
> >> >> This ABI will have similar API to say
> >> >> echo "mlx4_0 50" > rdma.weight.
> >> >> Where 50 is weight of the resources.
> >> >> For example,
> >> >> for one cgroup instance weight=sum=100% resource for a given cgroup.
> >> >> for three cgroup instances percentage=(weight/sum)% = 50/(50+50+50) = 33%.
> >> >> One cgroup gets 33% resource.
> >> >>
> >> >> Weight can be in range of 1 to 10,000 similar to cpu cgroup.
> >> >
> >> > This is exactly what I don't like, the percentage will remove from the
> >> > user the translation needs between weight and actual limitation.
> >> >
> >> > IMHO CPU used weights because everything there is in weights :).
> >> >
> >> I admit weight are not very intuitive, I was aligning to the existing
> >> other cgroup interfaces which achieves similar functionality.
> >> I will let Tejun approve the "percentage" or "ratio" new file
> >> interface as its little different than weight.
> >
> > Sure, let's close the main idea first and see if it makes sense for
> > other participants.
> >
> >>
> >> >>
> >> >> This might work if applications running in all cgroups are similar.
> >> >> But weight doesn't do justice, when there are different type of
> >> >> applications running in each cgroup. Such as few running libfabric
> >> >> based apps, few running MPI, others directly using ibverbs.
> >> >> So as you said rdma.max configuration would be required for management
> >> >> plane to override weight (percentage) for certain resources.
> >> >
> >> > Why?
> >> > The device exposes max values during initialization and if user asked
> >> > for 20% percent of HCA, he will get max*0.2.
> >>
> >> Because every application may not be equivalent of other application.
> >> For example, some require one to one QP and PD mapping.
> >> Some share single PD across multiple QPs.
> >> Some have ratio of 100 MRs per QP, as factor of memory size and operations.
> >> some servers like to have 1K MRs per QP.
> >> So if we have just weight, it will equally distributes MRs per QP in
> >> all cgroup and that either leads to unused resource per cgroup or,
> >> lesser number of cg instances.
> >> So fine tuning required for individual one, which we already have.
> >
> > I afraid that it is over complicating which can be done by curious user
> > in his user-space scripts: limit the global HCA -> read max values ->
> > overwrite with specific mapping.
> >
> >>
> >> weight or percentage helps in abstracting as starting point. So I like
> >> to add it too.
> >
> > Let's start simple
>
> Yes. I will rebase and test my patch today and see if requires resending.

It is worth to wait till -rc1. Doug didn't finish his pull requests yet.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]