Re: Cache Allocation Technology Design

Matt Fleming
2014-10-29T12:48:40Z

On Wed, 29 Oct, at 09:16:40AM, Peter Zijlstra wrote:
>
> Ah, so one way around that is to only assign a (whats the CQE equivalent
> of RMIDs again?) once you stick a task in.

I think you're after "Class of Service" (CLOS) ID.

Yeah we can do the CLOS ID assignment on-demand but what we can't do
on-demand is the cache bitmask assignment, i.e. how we carve up the LLC.
These need to persist irrespective of which task is running. And it's
the cache bitmask that I'm specifically talking about not allowing
arbitrarly deep nesting.

So if I create a cgroup directory with a mask of 0x3 in the root cgroup
directory for CAT (meow). Then, create two sub-directories, and split my
0x3 bitmask into 0x2 and 0x1, it's impossible to nest any further, i.e.

/sys/fs/cgroup/cacheqe 0xffffffff
|
|
meow 0x3
/ \
/ \
sub1 sub2 0x1, 0x2

Of course the pathological case is creating a cgroup directory with
bitmask 0x1, so you can't have sub-directories because you can't split
the cache allocation at all.

Does this fly in the face of "full hierarchies"? Or is this a reasonable
limitation?

> But basically it means you need to allow things like:
>
> root/virt/more/crap/hostA
> /hostB
> /sanityA
> /random/other/yunk
>
> Now, the root will have the entire bitmask set, any child, say
> virt/more/crap can also have them all set, and you can maybe only start
> differentiating in the /host[AB] bits.
>
> Whether or not it makes sense, libvirt likes to create these pointless
> deep hierarchies, as do a lot of other people for that matter.

OK, this is something I hadn't considered; that you may *not* want to
split the cache bitmask as you move down the hierarchy.

I think that's something we could do without too much pain, though
actually programming that from a user perspective makes my head hurt.

--
Matt Fleming, Intel Open Source Technology Center