Re: Xen-4.5 Testing - Cache Monitoring Technology

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 20/11/14 10:23, Andrew Cooper wrote:
> On 20/11/14 10:15, Chao Peng wrote:
>> On Thu, Nov 20, 2014 at 09:58:37AM +0000, Andrew Cooper wrote:
>>> Hi,
>>>
>>> I have found myself a PSR-capable server and have been having a play
>>> with Xen-4.5
>>>
>>> At a first pass, I can get some numbers out:
>>>
>>> [root@blob ~]# xl psr-cmt-attach 0
>>> [root@blob ~]# xl psr-cmt-show cache_occupancy
>>> Total RMID: 71
>>> Name                                        ID        Socket 0       
>>> Socket 1
>>> Total L3 Cache Size                                   46080 KB       
>>> 46080 KB
>>> Domain-0                                     0        45072
>>> KB            0 KB
>>> [root@blob ~]#
>>>
>>> However, I am unable to get any occupancy information for HVM guests. 
>>> Given nothing obvious in the code which would make CMT PV-guest
>>> specific, I presume this is unexpected?
>>>
>>>
>> Thank your for trying...
>> But I just tested the HVM on my box and it runs correct.
>>
>> Have you missed to run psr-cmt-attach for your HVM domain? Otherwise I
>> think there may be something wrong.
> No - I get the HVM domain (a memtest domain) listed in the output.  It
> is clear from Dom0's occupancy dropping off significantly that
> competition is underway on Socket 0 (as expected), but the domain always
> has 0 occupancy reported.
>
> I shall investigate.

On further investigation, it would appear to be a behavioural property
of memtest (5.01, 8vcpu guest, 1GB RAM).

In SMP mode with all cpus streaming data, the occupancy skyrockets. 
However, when only cpu0 is running (with all other cpus waiting until
the end of the round), the occupancy drops to 0.

On this system, it would appear that the minimum granularity is 72KB,
and 72KB is occasionally reported rather than 0.  I am guessing that
when only a single cpu is running, the occupancy is less than the
granularity, and the 0 being reported is as a result of rounding.

It is interesting to note that with dom0 only, dom0's occupancy is
almost all of the LLC, but with this single memtest domU, the
occupancies of dom0 + domU is never more than 1/4 of the LLC.  I presume
that this means that copious cache flushing is going on.


Whatever the reason, it has been interesting having a bit of a play. It
does appear that PSR and CMT JustWorked(tm) for me on Xen-4.5 which is
very nice.

~Andrew