Re: [PATCH RFC] ioctl based CAT interface

[Marcelo Tosatti <mtosatti@redhat.com>]

On Mon, Nov 16, 2015 at 09:59:26AM +0100, Peter Zijlstra wrote:
> On Fri, Nov 13, 2015 at 03:43:02PM -0200, Marcelo Tosatti wrote:
> > Subject: cat cgroup interface proposal (non hierarchical) was Re: [PATCH
> > V15 00/11] x86: Intel Cache
> >        	Allocation Technology Support
> > 
> > 
> > https://lkml.org/lkml/2015/11/2/700
> 
> I've really no idea what you're trying to say there. That just doesn't
> parse.

I've posted this internally at people understood and commented on it.

What part you don't understand?  

Its a proposal for a cgroups based interface. The description
starts from the main mount point of the cgroup. So you would:

#  mount -t cgroup -ointel_cat intel_cat /sys/fs/cgroup/cat/
#  cd /sys/fs/cgroup/cat/
#  ls
cat_hw_info
# mkdir cache_reservation_for_forwarding_app
# ls
cat_hw_info   cache_reservation_for_forwarding_app
# cd cache_reservation_for_forwarding_app
# ls
type	size	socketmask	enable_reservation
# ...

Yes?



cgroup CAT interface (non hierarchical):
---------------------------------------
0) Main directory files:

cat_hw_info
-----------
CAT HW information: CBM length, CDP supported, etc.
Information reported per-socket, as sockets can have
different configurations. Perhaps should be inside
sysfs.
1) Sub-directories represent cache reservations (size,type).

mkdir cache_reservation_for_forwarding_app
cd cache_reservation_for_forwarding_app
echo "80KB" > size
echo "data_and_code" > type
echo "socketmask=0xfffff" > socketmask (optional)
echo "1" > enable_reservation
echo "pid-of-forwarding-main-thread pid-of-helper-thread ..." > tasks
Files:

type
----------------
{data_and_code, only_code, only_data}. Type of
L3 CAT cache allocation to use. only_code,only_data only
supported on CDP capable processors.
size
----
size of L3 cache reservation.
rounding
--------
{round_down,round_up} whether to round up / round down
allocation size in kbytes, to cache-way size.
Default: round_up

socketmask
----------
Mask of sockets where the reservation is in effect.
A zero bit means the task will not have the L3 cache
portion that the cgroup references reserved on that socket.
Default: all sockets set.
enable
------
Allocate reservation with parameters set above.
When a reservation is enabled, it reserves L3 cache
space on any socket thats specified in "socketmask".

After cgroup has been enabled by a write of "1" to
"enable_reservation" file, only the "tasks" file can be modified.
To change the size of a cgroup reservation, recreate the directory.

tasks
-----
Contains the list of tasks which use this cache reservation.

Error reporting
---------------
Errors are reported in response to write as appropriate:
for example, write 1 > enable when there is not enough space
for "socketmask" would return -ENOSPC, etc.
Write to "enable" without size being set would return -EINVAL, etc.
Listing
-------
To list which reservations are in place, search for subdirectories
where "enabled" file has value 1.
Semantics: A task has guaranteed cache reservation on any CPU where its
scheduled in, for the lifetime of the cgroup, as long as that task is
not attached to further cgroups.

That is, a task belonging to cgroup-A can have its cache reservation
invalidated when attached to cgroup-B, (reasoning: it might be necessary
to reallocate the CBMs to respect contiguous bits in cache, a
restriction of the CAT HW interface).


-------
BLOCKER
-------
Can't use cgroups for CAT because:

"Control Groups extends the kernel as follows:

 - Each task in the system has a reference-counted pointer to a
   css_set.

 - A css_set contains a set of reference-counted pointers to
   cgroup_subsys_state objects, one for each cgroup subsystem
   registered in the system."

You need a task to be part of two cgroups at one time, 
to support the following configuration:

Task-A: 70% of cache reserved exclusively (reservation-0).
        20% of cache reserved (reservation-1).

Task-B: 20% of cache reserved (reservation-1).

Unless reservations are created separately, then added to cgroups:

mount -t cgroup ... /../catcgroup/
cd /../catcgroup/
# create reservations
cd reservations
mkdir reservation-1
echo "80K" > size
echo "socketmask" > ...
echo "1" > enable
mkdir reservation-2 
echo "160K" > size
echo "socketmask" > ...
echo "1" > enable
# attach reservation to cgroups
cd /../catcgroup/
mkdir cgroup-for-threaded-app
echo reservation-1 reservation-2 > reservations
echo "mainthread" > tasks
cd ..
mkdir cgroup-for-helper-thread
echo reservation-1 > reservations
echo "helperthread" > tasks
cd .. 
This way mainthread and helperthread can share "reservation-1".