clarification about misc controller and capacity vs. max

clarification about misc controller and capacity vs. max

[Accardi, Kristen C]

Hi there,
I notice in the comments for the misc controller it is stated that the
max limit can be more than actual total capacity, meaning that we can
overcommit with the resource controlled by the misc controller.
However, in the misc_cg_try_charge() code, the function will return -
EBUSY if max limit will be crossed or total usage will be more than the
capacity, which would seem to enforce total capacity as an upper limit
in addition to max and not allow for overcommit. Can you provide some
clarity on whether the resource consumption model for the misc
controller should allow for overcommit?

Thanks,
Kristen Accardi

Re: clarification about misc controller and capacity vs. max

[tj-DgEjT+Ai2ygdnm+yROfE0A]

Hello,

On Tue, Nov 01, 2022 at 04:40:22PM +0000, Accardi, Kristen C wrote:
> I notice in the comments for the misc controller it is stated that the
> max limit can be more than actual total capacity, meaning that we can
> overcommit with the resource controlled by the misc controller.
> However, in the misc_cg_try_charge() code, the function will return -
> EBUSY if max limit will be crossed or total usage will be more than the
> capacity, which would seem to enforce total capacity as an upper limit
> in addition to max and not allow for overcommit. Can you provide some
> clarity on whether the resource consumption model for the misc
> controller should allow for overcommit?

I think what it's trying to say is that the sum of first level .max's can be
higher than the total capacity. e.g. Let's say you have 5 of this resource
and a hierarchy like the following.

        R - A - A'
          + B - B'
          \ C

It's valid to have A, B, C's max set to 4, 3, 2 respectively even if they
sum up to 9 which is larger than what's available in the system, 5 - ie. the
max limits are overcommitted for the resource.

Thanks.

-- 
tejun

Re: clarification about misc controller and capacity vs. max

[Accardi, Kristen C]

On Tue, 2022-11-01 at 06:54 -1000, tj@kernel.org wrote:
> Hello,
> 
> On Tue, Nov 01, 2022 at 04:40:22PM +0000, Accardi, Kristen C wrote:
> > I notice in the comments for the misc controller it is stated that
> > the
> > max limit can be more than actual total capacity, meaning that we
> > can
> > overcommit with the resource controlled by the misc controller.
> > However, in the misc_cg_try_charge() code, the function will return
> > -
> > EBUSY if max limit will be crossed or total usage will be more than
> > the
> > capacity, which would seem to enforce total capacity as an upper
> > limit
> > in addition to max and not allow for overcommit. Can you provide
> > some
> > clarity on whether the resource consumption model for the misc
> > controller should allow for overcommit?
> 
> I think what it's trying to say is that the sum of first level .max's
> can be
> higher than the total capacity. e.g. Let's say you have 5 of this
> resource
> and a hierarchy like the following.
> 
>         R - A - A'
>           + B - B'
>           \ C
> 
> It's valid to have A, B, C's max set to 4, 3, 2 respectively even if
> they
> sum up to 9 which is larger than what's available in the system, 5 -
> ie. the
> max limits are overcommitted for the resource.
> 
> Thanks.
> 

So to be clear, if I have this:

/sys/fs/cgroup/misc.capacity
some_res 10

and this:
/sys/fs/cgroup/test

test.current will never be allowed to exceed 10.

Re: clarification about misc controller and capacity vs. max

[tj-DgEjT+Ai2ygdnm+yROfE0A]

On Tue, Nov 01, 2022 at 05:03:25PM +0000, Accardi, Kristen C wrote:
> So to be clear, if I have this:
> 
> /sys/fs/cgroup/misc.capacity
> some_res 10
> 
> and this:
> /sys/fs/cgroup/test
> 
> test.current will never be allowed to exceed 10.

Yes, but test.max can be whatever. So, the resources themselves can't be
over-committed. The max limits (ie. the promises) can be.

Thanks.

-- 
tejun

Re: clarification about misc controller and capacity vs. max

[Accardi, Kristen C]

On Tue, 2022-11-01 at 07:05 -1000, tj@kernel.org wrote:
> On Tue, Nov 01, 2022 at 05:03:25PM +0000, Accardi, Kristen C wrote:
> > So to be clear, if I have this:
> > 
> > /sys/fs/cgroup/misc.capacity
> > some_res 10
> > 
> > and this:
> > /sys/fs/cgroup/test
> > 
> > test.current will never be allowed to exceed 10.
> 
> Yes, but test.max can be whatever. So, the resources themselves can't
> be
> over-committed. The max limits (ie. the promises) can be.
> 
> Thanks.
> 

This is a bit of a deal breaker for the use of the misc controller for
SGX EPC memory - we allow overcommit of the physical EPC memory as we
have backing RAM that is used to swap. Would you be amenable to having
a flag to ignore the total capacity value and allow for overcommit of
the resource? If not I feel like we don't have a choice but to create a
new controller.

Re: clarification about misc controller and capacity vs. max

[tj-DgEjT+Ai2ygdnm+yROfE0A]

On Tue, Nov 01, 2022 at 05:11:13PM +0000, Accardi, Kristen C wrote:
> This is a bit of a deal breaker for the use of the misc controller for
> SGX EPC memory - we allow overcommit of the physical EPC memory as we
> have backing RAM that is used to swap. Would you be amenable to having
> a flag to ignore the total capacity value and allow for overcommit of
> the resource? If not I feel like we don't have a choice but to create a
> new controller.

Yeah, for sure. Maybe just introduce a special value (prolly u64 max) that
says that there's no limit. Or maybe just set it to total physical memory?

Thanks.

-- 
tejun

Re: clarification about misc controller and capacity vs. max

[Michal Koutný]

[-- Attachment #1: Type: text/plain, Size: 1070 bytes --]

Hello.

On Tue, Nov 01, 2022 at 05:03:25PM +0000, "Accardi, Kristen C" <kristen.c.accardi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> So to be clear, if I have this:
> 
> /sys/fs/cgroup/misc.capacity
> some_res 10
> 
> and this:
> /sys/fs/cgroup/test
> 
> test.current will never be allowed to exceed 10.

If the capacity was initially larger than 10 and charges inside test
succeeded and then some event caused shrinking of the resource (without
synchronously preempting distributed units), then test.misc.current can
exceed the current capacity.

See the condition in misc_cg_current_show()
    if (READ_ONCE(misc_res_capacity[i]) || usage)

What the shrinking and preemption means is up to the implementation of
the particular miscresource.

IOW sum of 1st level children's .current may overrun capacity,
semantics is not defined by the misc controller (but it'd reject new
charges in such a situation).

(That's just for completeness. I understood from the rest that you
rather want to disable capacity checking.)

HTH,
Michal

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]