Xen-4.5 Testing - Cache Monitoring Technology

Xen-4.5 Testing - Cache Monitoring Technology

[Andrew Cooper]

Hi,

I have found myself a PSR-capable server and have been having a play
with Xen-4.5

At a first pass, I can get some numbers out:

[root@blob ~]# xl psr-cmt-attach 0
[root@blob ~]# xl psr-cmt-show cache_occupancy
Total RMID: 71
Name                                        ID        Socket 0       
Socket 1
Total L3 Cache Size                                   46080 KB       
46080 KB
Domain-0                                     0        45072
KB            0 KB
[root@blob ~]#

However, I am unable to get any occupancy information for HVM guests. 
Given nothing obvious in the code which would make CMT PV-guest
specific, I presume this is unexpected?

~Andrew

Re: Xen-4.5 Testing - Cache Monitoring Technology

[Chao Peng]

On Thu, Nov 20, 2014 at 09:58:37AM +0000, Andrew Cooper wrote:
> Hi,
> 
> I have found myself a PSR-capable server and have been having a play
> with Xen-4.5
> 
> At a first pass, I can get some numbers out:
> 
> [root@blob ~]# xl psr-cmt-attach 0
> [root@blob ~]# xl psr-cmt-show cache_occupancy
> Total RMID: 71
> Name                                        ID        Socket 0       
> Socket 1
> Total L3 Cache Size                                   46080 KB       
> 46080 KB
> Domain-0                                     0        45072
> KB            0 KB
> [root@blob ~]#
> 
> However, I am unable to get any occupancy information for HVM guests. 
> Given nothing obvious in the code which would make CMT PV-guest
> specific, I presume this is unexpected?
> 
> 
Thank your for trying...
But I just tested the HVM on my box and it runs correct.

Have you missed to run psr-cmt-attach for your HVM domain? Otherwise I
think there may be something wrong.

The cmt itself should not be PV-specific.

Chao
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Xen-4.5 Testing - Cache Monitoring Technology

[Andrew Cooper]

On 20/11/14 10:15, Chao Peng wrote:
> On Thu, Nov 20, 2014 at 09:58:37AM +0000, Andrew Cooper wrote:
>> Hi,
>>
>> I have found myself a PSR-capable server and have been having a play
>> with Xen-4.5
>>
>> At a first pass, I can get some numbers out:
>>
>> [root@blob ~]# xl psr-cmt-attach 0
>> [root@blob ~]# xl psr-cmt-show cache_occupancy
>> Total RMID: 71
>> Name                                        ID        Socket 0       
>> Socket 1
>> Total L3 Cache Size                                   46080 KB       
>> 46080 KB
>> Domain-0                                     0        45072
>> KB            0 KB
>> [root@blob ~]#
>>
>> However, I am unable to get any occupancy information for HVM guests. 
>> Given nothing obvious in the code which would make CMT PV-guest
>> specific, I presume this is unexpected?
>>
>>
> Thank your for trying...
> But I just tested the HVM on my box and it runs correct.
>
> Have you missed to run psr-cmt-attach for your HVM domain? Otherwise I
> think there may be something wrong.

No - I get the HVM domain (a memtest domain) listed in the output.  It
is clear from Dom0's occupancy dropping off significantly that
competition is underway on Socket 0 (as expected), but the domain always
has 0 occupancy reported.

I shall investigate.

~Andrew

Re: Xen-4.5 Testing - Cache Monitoring Technology

[Andrew Cooper]

On 20/11/14 10:23, Andrew Cooper wrote:
> On 20/11/14 10:15, Chao Peng wrote:
>> On Thu, Nov 20, 2014 at 09:58:37AM +0000, Andrew Cooper wrote:
>>> Hi,
>>>
>>> I have found myself a PSR-capable server and have been having a play
>>> with Xen-4.5
>>>
>>> At a first pass, I can get some numbers out:
>>>
>>> [root@blob ~]# xl psr-cmt-attach 0
>>> [root@blob ~]# xl psr-cmt-show cache_occupancy
>>> Total RMID: 71
>>> Name                                        ID        Socket 0       
>>> Socket 1
>>> Total L3 Cache Size                                   46080 KB       
>>> 46080 KB
>>> Domain-0                                     0        45072
>>> KB            0 KB
>>> [root@blob ~]#
>>>
>>> However, I am unable to get any occupancy information for HVM guests. 
>>> Given nothing obvious in the code which would make CMT PV-guest
>>> specific, I presume this is unexpected?
>>>
>>>
>> Thank your for trying...
>> But I just tested the HVM on my box and it runs correct.
>>
>> Have you missed to run psr-cmt-attach for your HVM domain? Otherwise I
>> think there may be something wrong.
> No - I get the HVM domain (a memtest domain) listed in the output.  It
> is clear from Dom0's occupancy dropping off significantly that
> competition is underway on Socket 0 (as expected), but the domain always
> has 0 occupancy reported.
>
> I shall investigate.

On further investigation, it would appear to be a behavioural property
of memtest (5.01, 8vcpu guest, 1GB RAM).

In SMP mode with all cpus streaming data, the occupancy skyrockets. 
However, when only cpu0 is running (with all other cpus waiting until
the end of the round), the occupancy drops to 0.

On this system, it would appear that the minimum granularity is 72KB,
and 72KB is occasionally reported rather than 0.  I am guessing that
when only a single cpu is running, the occupancy is less than the
granularity, and the 0 being reported is as a result of rounding.

It is interesting to note that with dom0 only, dom0's occupancy is
almost all of the LLC, but with this single memtest domU, the
occupancies of dom0 + domU is never more than 1/4 of the LLC.  I presume
that this means that copious cache flushing is going on.


Whatever the reason, it has been interesting having a bit of a play. It
does appear that PSR and CMT JustWorked(tm) for me on Xen-4.5 which is
very nice.

~Andrew

Re: Xen-4.5 Testing - Cache Monitoring Technology

[Chao Peng]

On Thu, Nov 20, 2014 at 02:21:48PM +0000, Andrew Cooper wrote:
> On 20/11/14 10:23, Andrew Cooper wrote:
> > On 20/11/14 10:15, Chao Peng wrote:
> >> On Thu, Nov 20, 2014 at 09:58:37AM +0000, Andrew Cooper wrote:
> >>> Hi,
> >>>
> >>> I have found myself a PSR-capable server and have been having a play
> >>> with Xen-4.5
> >>>
> >>> At a first pass, I can get some numbers out:
> >>>
> >>> [root@blob ~]# xl psr-cmt-attach 0
> >>> [root@blob ~]# xl psr-cmt-show cache_occupancy
> >>> Total RMID: 71
> >>> Name                                        ID        Socket 0       
> >>> Socket 1
> >>> Total L3 Cache Size                                   46080 KB       
> >>> 46080 KB
> >>> Domain-0                                     0        45072
> >>> KB            0 KB
> >>> [root@blob ~]#
> >>>
> >>> However, I am unable to get any occupancy information for HVM guests. 
> >>> Given nothing obvious in the code which would make CMT PV-guest
> >>> specific, I presume this is unexpected?
> >>>
> >>>
> >> Thank your for trying...
> >> But I just tested the HVM on my box and it runs correct.
> >>
> >> Have you missed to run psr-cmt-attach for your HVM domain? Otherwise I
> >> think there may be something wrong.
> > No - I get the HVM domain (a memtest domain) listed in the output.  It
> > is clear from Dom0's occupancy dropping off significantly that
> > competition is underway on Socket 0 (as expected), but the domain always
> > has 0 occupancy reported.
> >
> > I shall investigate.
> 
> On further investigation, it would appear to be a behavioural property
> of memtest (5.01, 8vcpu guest, 1GB RAM).
> 
> In SMP mode with all cpus streaming data, the occupancy skyrockets. 
> However, when only cpu0 is running (with all other cpus waiting until
> the end of the round), the occupancy drops to 0.
> 
> On this system, it would appear that the minimum granularity is 72KB,
> and 72KB is occasionally reported rather than 0.  I am guessing that
> when only a single cpu is running, the occupancy is less than the
> granularity, and the 0 being reported is as a result of rounding.

This is probably the reason and the minimum granularity does exist. 
As the hardware underground just tags the cache ways with RMID. So the
granularity should be at lease the cache way size.
> 
> It is interesting to note that with dom0 only, dom0's occupancy is
> almost all of the LLC, but with this single memtest domU, the
> occupancies of dom0 + domU is never more than 1/4 of the LLC.  I presume
> that this means that copious cache flushing is going on.
> 
I saw the same result sometimes. Actually the data varies for different
types of workloads. Sometimes the data surprises me but usually it looks
reasonable. And this is the interesting thing the CMT bring to me. In
the past, we just ‘guess’ the cache utilization but now we can ‘see’ it.

Chao
> 
> Whatever the reason, it has been interesting having a bit of a play. It
> does appear that PSR and CMT JustWorked(tm) for me on Xen-4.5 which is
> very nice.
> 
> ~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel