Re: Cache Allocation Technology Design

Peter Zijlstra
2014-10-29T13:45:34Z

On Wed, Oct 29, 2014 at 12:48:34PM +0000, Matt Fleming wrote:
> On Wed, 29 Oct, at 09:16:40AM, Peter Zijlstra wrote:
> >
> > Ah, so one way around that is to only assign a (whats the CQE equivalent
> > of RMIDs again?) once you stick a task in.
>
> I think you're after "Class of Service" (CLOS) ID.
>
> Yeah we can do the CLOS ID assignment on-demand but what we can't do
> on-demand is the cache bitmask assignment, i.e. how we carve up the LLC.
> These need to persist irrespective of which task is running. And it's
> the cache bitmask that I'm specifically talking about not allowing
> arbitrarly deep nesting.
>
> So if I create a cgroup directory with a mask of 0x3 in the root cgroup
> directory for CAT (meow).

All we now need is a DOG to go woof :-) and they can have a party.

> Then, create two sub-directories, and split my
> 0x3 bitmask into 0x2 and 0x1, it's impossible to nest any further, i.e.
>
> /sys/fs/cgroup/cacheqe 0xffffffff
> |
> |
> meow 0x3
> / \
> / \
> sub1 sub2 0x1, 0x2
>
> Of course the pathological case is creating a cgroup directory with
> bitmask 0x1, so you can't have sub-directories because you can't split
> the cache allocation at all.
>
> Does this fly in the face of "full hierarchies"? Or is this a reasonable
> limitation?

I don't see a reason why we should not allow further children of sub1,
they'll all have to have 0x1, but that should be fine, pointless
perhaps, but perfectly consistent.

> > But basically it means you need to allow things like:
> >
> > root/virt/more/crap/hostA
> > /hostB
> > /sanityA
> > /random/other/yunk
> >
> > Now, the root will have the entire bitmask set, any child, say
> > virt/more/crap can also have them all set, and you can maybe only start
> > differentiating in the /host[AB] bits.
> >
> > Whether or not it makes sense, libvirt likes to create these pointless
> > deep hierarchies, as do a lot of other people for that matter.
>
> OK, this is something I hadn't considered; that you may *not* want to
> split the cache bitmask as you move down the hierarchy.
>
> I think that's something we could do without too much pain, though
> actually programming that from a user perspective makes my head hurt.

Right, also note that in the libvirt case, most of the intermediate
groups are empty (of tasks) and would thus not actually instantiate a
CLOS thingy.