Questing regarding KVM Guest PMU

Questing regarding KVM Guest PMU

[shashank rachamalla]

Hi

I have setup kernel 3.3 ( which has save and restore support for PMU
registers in KVM ). Also, I have started a guest machine by specifying
"--cpu host" in qemu-kvm and verified that the "arch_perfmon" cpu flag
is visible inside the guest. However, I could not succeed in counting
the number of cpu cycles for a cpu intensive program using oprofile
inside guest.

The following is a simple script used to run oprofile

#############################################
/usr/bin/opcontrol --reset
/usr/bin/opcontrol --event="default"
/usr/bin/opcontrol --setup --no-vmlinux --separate=library
/usr/bin/opcontrol --start
#This is a cpu intensive program
./a.out
/usr/bin/opcontrol --shutdown
opreport --global-percent --long-filenames --threshold 20
##############################################

Output on host

CPU: Core 2, speed 2201 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
unit mask of 0x00 (Unhalted core cycles) count 100000
CPU_CLK_UNHALT...|
 samples|      %|
------------------
   21125 43.2597 /home/shashank/workspace/seminar/code/oprofile/a.out

Output on guest

CPU: Core 2, speed 2200.19 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
unit mask of 0x00 (Unhalted core cycles) count 100000
CPU_CLK_UNHALT...|
 samples|      %|
------------------
       1 100.000 /no-vmlinux


Note that only 1 sample got collected on guest. I see the same
behavior for other workloads on guest. Also, I executed CPUID
instruction on both guest and host to check the PMU leaf details
(0x0A) and found them to be the same.  Please let me know if I am
missing anything here.  I am yet to profile the same workload on
host/guest using perf.

Regards,
Shashank

Re: Questing regarding KVM Guest PMU

[Gleb Natapov]

On Sun, Mar 18, 2012 at 03:48:53PM +0530, shashank rachamalla wrote:
> 
> CPU: Core 2, speed 2200.19 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
> unit mask of 0x00 (Unhalted core cycles) count 100000
> CPU_CLK_UNHALT...|
>  samples|      %|
> ------------------
>        1 100.000 /no-vmlinux
> 
> 
What happens if you run "perf stat -e cycles a.out"?

--
			Gleb.

Re: Questing regarding KVM Guest PMU

[shashank rachamalla]

2012/3/18 Gleb Natapov <gleb@redhat.com>:
> On Sun, Mar 18, 2012 at 03:48:53PM +0530, shashank rachamalla wrote:
>>
>> CPU: Core 2, speed 2200.19 MHz (estimated)
>> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
>> unit mask of 0x00 (Unhalted core cycles) count 100000
>> CPU_CLK_UNHALT...|
>>  samples|      %|
>> ------------------
>>        1 100.000 /no-vmlinux
>>
>>
> What happens if you run "perf stat -e cycles a.out"?

This is what i get when I run "perf stat ./a.out" ( with --cpu host
passed in qemu-kvm )

Performance counter stats for './a.out':

     954.925996  task-clock-msecs         #      0.955 CPUs
             23  context-switches         #      0.000 M/sec
              0  CPU-migrations           #      0.000 M/sec
             93  page-faults              #      0.000 M/sec
     2110635735  cycles                   #   2210.261 M/sec
      901816391  instructions             #      0.427 IPC
         247677  cache-references         #      0.259 M/sec
          15044  cache-misses             #      0.016 M/sec

    0.999846560  seconds time elapsed

This is without --cpu host  ( Note that only software events get monitored )

Performance counter stats for './a.out':

     913.826372  task-clock-msecs         #      0.990 CPUs
             13  context-switches         #      0.000 M/sec
              0  CPU-migrations           #      0.000 M/sec
             93  page-faults              #      0.000 M/sec
  <not counted>  cycles
  <not counted>  instructions
  <not counted>  cache-references
  <not counted>  cache-misses

    0.923182044  seconds time elapsed

I guess things are working fine with perf. But why not with oprofile ?

>
> --
>                        Gleb.

Re: Questing regarding KVM Guest PMU

[Avi Kivity]

On 03/18/2012 02:57 PM, shashank rachamalla wrote:
> 2012/3/18 Gleb Natapov <gleb@redhat.com>:
> > On Sun, Mar 18, 2012 at 03:48:53PM +0530, shashank rachamalla wrote:
> >>
> >> CPU: Core 2, speed 2200.19 MHz (estimated)
> >> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
> >> unit mask of 0x00 (Unhalted core cycles) count 100000
> >> CPU_CLK_UNHALT...|
> >>  samples|      %|
> >> ------------------
> >>        1 100.000 /no-vmlinux
> >>
> >>
> > What happens if you run "perf stat -e cycles a.out"?
>
> This is what i get when I run "perf stat ./a.out" ( with --cpu host
> passed in qemu-kvm )
>
> Performance counter stats for './a.out':
>
>      954.925996  task-clock-msecs         #      0.955 CPUs
>              23  context-switches         #      0.000 M/sec
>               0  CPU-migrations           #      0.000 M/sec
>              93  page-faults              #      0.000 M/sec
>      2110635735  cycles                   #   2210.261 M/sec
>       901816391  instructions             #      0.427 IPC
>          247677  cache-references         #      0.259 M/sec
>           15044  cache-misses             #      0.016 M/sec
>
>     0.999846560  seconds time elapsed
>
> This is without --cpu host  ( Note that only software events get monitored )
>
> Performance counter stats for './a.out':
>
>      913.826372  task-clock-msecs         #      0.990 CPUs
>              13  context-switches         #      0.000 M/sec
>               0  CPU-migrations           #      0.000 M/sec
>              93  page-faults              #      0.000 M/sec
>   <not counted>  cycles
>   <not counted>  instructions
>   <not counted>  cache-references
>   <not counted>  cache-misses
>
>     0.923182044  seconds time elapsed
>
> I guess things are working fine with perf. But why not with oprofile ?
>
>

What kernel are you using in the guest?

IIRC oprofile is using perf for its data source, but perhaps this wasn't
tree in older kernels.

-- 
error compiling committee.c: too many arguments to function

Re: Questing regarding KVM Guest PMU

[Gleb Natapov]

On Sun, Mar 18, 2012 at 06:27:53PM +0530, shashank rachamalla wrote:
> 2012/3/18 Gleb Natapov <gleb@redhat.com>:
> > On Sun, Mar 18, 2012 at 03:48:53PM +0530, shashank rachamalla wrote:
> >>
> >> CPU: Core 2, speed 2200.19 MHz (estimated)
> >> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
> >> unit mask of 0x00 (Unhalted core cycles) count 100000
> >> CPU_CLK_UNHALT...|
> >>  samples|      %|
> >> ------------------
> >>        1 100.000 /no-vmlinux
> >>
> >>
> > What happens if you run "perf stat -e cycles a.out"?
> 
> This is what i get when I run "perf stat ./a.out" ( with --cpu host
> passed in qemu-kvm )
> 
> Performance counter stats for './a.out':
> 
>      954.925996  task-clock-msecs         #      0.955 CPUs
>              23  context-switches         #      0.000 M/sec
>               0  CPU-migrations           #      0.000 M/sec
>              93  page-faults              #      0.000 M/sec
>      2110635735  cycles                   #   2210.261 M/sec
>       901816391  instructions             #      0.427 IPC
>          247677  cache-references         #      0.259 M/sec
>           15044  cache-misses             #      0.016 M/sec
> 
>     0.999846560  seconds time elapsed
> 
> This is without --cpu host  ( Note that only software events get monitored )
> 
> Performance counter stats for './a.out':
> 
>      913.826372  task-clock-msecs         #      0.990 CPUs
>              13  context-switches         #      0.000 M/sec
>               0  CPU-migrations           #      0.000 M/sec
>              93  page-faults              #      0.000 M/sec
>   <not counted>  cycles
>   <not counted>  instructions
>   <not counted>  cache-references
>   <not counted>  cache-misses
> 
>     0.923182044  seconds time elapsed
> 
> I guess things are working fine with perf. But why not with oprofile ?
> 
Looks like it. I never tried oprofile. Will try to reproduce your
problem and see what oprofile is doing.

--
			Gleb.

Re: Questing regarding KVM Guest PMU

[shashank rachamalla]

[-- Attachment #1: Type: text/plain, Size: 2658 bytes --]

2012/3/18 Gleb Natapov <gleb@redhat.com>:
> On Sun, Mar 18, 2012 at 06:27:53PM +0530, shashank rachamalla wrote:
>> 2012/3/18 Gleb Natapov <gleb@redhat.com>:
>> > On Sun, Mar 18, 2012 at 03:48:53PM +0530, shashank rachamalla wrote:
>> >>
>> >> CPU: Core 2, speed 2200.19 MHz (estimated)
>> >> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
>> >> unit mask of 0x00 (Unhalted core cycles) count 100000
>> >> CPU_CLK_UNHALT...|
>> >>  samples|      %|
>> >> ------------------
>> >>        1 100.000 /no-vmlinux
>> >>
>> >>
>> > What happens if you run "perf stat -e cycles a.out"?
>>
>> This is what i get when I run "perf stat ./a.out" ( with --cpu host
>> passed in qemu-kvm )
>>
>> Performance counter stats for './a.out':
>>
>>      954.925996  task-clock-msecs         #      0.955 CPUs
>>              23  context-switches         #      0.000 M/sec
>>               0  CPU-migrations           #      0.000 M/sec
>>              93  page-faults              #      0.000 M/sec
>>      2110635735  cycles                   #   2210.261 M/sec
>>       901816391  instructions             #      0.427 IPC
>>          247677  cache-references         #      0.259 M/sec
>>           15044  cache-misses             #      0.016 M/sec
>>
>>     0.999846560  seconds time elapsed
>>
>> This is without --cpu host  ( Note that only software events get monitored )
>>
>> Performance counter stats for './a.out':
>>
>>      913.826372  task-clock-msecs         #      0.990 CPUs
>>              13  context-switches         #      0.000 M/sec
>>               0  CPU-migrations           #      0.000 M/sec
>>              93  page-faults              #      0.000 M/sec
>>   <not counted>  cycles
>>   <not counted>  instructions
>>   <not counted>  cache-references
>>   <not counted>  cache-misses
>>
>>     0.923182044  seconds time elapsed
>>
>> I guess things are working fine with perf. But why not with oprofile ?
>>
> Looks like it. I never tried oprofile. Will try to reproduce your
> problem and see what oprofile is doing.

I am using ubuntu 10.04 with 2.6.32-21-generic kernel as guest and
oprofile 0.9.6.
Also, I have tried to capture kvm-events ( perf patch ) in host while
running oprofile and perf in guest.
Please see the attachment. I have run the tests in three cases for the
around 5 secs.

There are more number of MSR reads and writes in case of perf which I
think is normal. However, there are very few MSR reads and writes with
oprofile. Also, the number of NMI exceptions are too high in case of
oprofile.

>
> --
>                        Gleb.

[-- Attachment #2: report.txt --]
[-- Type: text/plain, Size: 2695 bytes --]

# Normal Case ( without any profiling in guest )
Analyze events for all VCPUs:

             VM-EXIT    Samples  Samples%     Time%         Avg time 

      IO_INSTRUCTION       1953    69.03%     2.48%     63.20us ( +-  49.77% )
         APIC_ACCESS        343    12.12%     0.07%     10.11us ( +-   4.86% )
       EXCEPTION_NMI        289    10.22%     0.03%      5.03us ( +-   2.51% )
   PENDING_INTERRUPT         81     2.86%     0.00%      1.69us ( +-   1.19% )
           CR_ACCESS         77     2.72%     0.01%      6.04us ( +-   6.86% )
                 HLT         72     2.55%    97.41%  67426.33us ( +-   9.26% )
  EXTERNAL_INTERRUPT         14     0.49%     0.01%     19.50us ( +-  22.36% )


# With perf in guest
Analyze events for all VCPUs:

             VM-EXIT    Samples  Samples%     Time%         Avg time 

       EXCEPTION_NMI       6906    26.93%     0.62%      4.16us ( +-   1.56% )
           MSR_WRITE       5894    22.99%     0.61%      4.83us ( +-   3.51% )
      IO_INSTRUCTION       3981    15.53%     2.81%     32.76us ( +-  21.95% )
            MSR_READ       2702    10.54%     0.07%      1.14us ( +-   0.38% )
         APIC_ACCESS       2370     9.24%     0.30%      5.78us ( +-   1.81% )
              INVLPG       2058     8.03%     0.10%      2.17us ( +-   4.98% )
  EXTERNAL_INTERRUPT        830     3.24%     0.35%     19.29us ( +-  25.56% )
           CR_ACCESS        418     1.63%     0.05%      5.63us ( +-   9.14% )
   PENDING_INTERRUPT        372     1.45%     0.01%      1.67us ( +-   0.47% )
                 HLT         93     0.36%    95.08%  47371.95us ( +-  10.55% )
               CPUID         18     0.07%     0.00%      1.28us ( +-   6.54% )

# With oprofile in guest
Analyze events for all VCPUs:

             VM-EXIT    Samples  Samples%     Time%         Avg time 

       EXCEPTION_NMI     161092    60.98%    12.48%      3.15us ( +-   0.56% )
              INVLPG      94184    35.65%     4.30%      1.85us ( +-   1.08% )
      IO_INSTRUCTION       2674     1.01%     2.39%     36.35us ( +-  31.43% )
           CR_ACCESS       2658     1.01%     1.05%     16.00us ( +-   3.06% )
         APIC_ACCESS       1739     0.66%     0.29%      6.80us ( +-   2.24% )
  EXTERNAL_INTERRUPT       1044     0.40%     0.55%     21.58us ( +-  27.84% )
               CPUID        539     0.20%     0.02%      1.28us ( +-   2.73% )
   PENDING_INTERRUPT        159     0.06%     0.01%      1.73us ( +-   1.09% )
                 HLT         73     0.03%    78.92%  43963.08us ( +-  12.33% )
           MSR_WRITE          7     0.00%     0.00%     23.73us ( +-  69.57% )
            MSR_READ          3     0.00%     0.00%      2.74us ( +-   8.44% )

Re: Questing regarding KVM Guest PMU

[Gleb Natapov]

On Sun, Mar 18, 2012 at 09:47:55PM +0530, shashank rachamalla wrote:
> >> I guess things are working fine with perf. But why not with oprofile ?
> >>
> > Looks like it. I never tried oprofile. Will try to reproduce your
> > problem and see what oprofile is doing.
> 
> I am using ubuntu 10.04 with 2.6.32-21-generic kernel as guest and
> oprofile 0.9.6.
> Also, I have tried to capture kvm-events ( perf patch ) in host while
> running oprofile and perf in guest.
> Please see the attachment. I have run the tests in three cases for the
> around 5 secs.
> 
> There are more number of MSR reads and writes in case of perf which I
> think is normal. However, there are very few MSR reads and writes with
> oprofile. Also, the number of NMI exceptions are too high in case of
> oprofile.
> 
Which host kernel are you using? Try latest kvm.git and check if you see
something unusual in dmesg.

--
			Gleb.

Re: Questing regarding KVM Guest PMU

[shashank rachamalla]

On Sun, Mar 18, 2012 at 10:21 PM, Gleb Natapov <gleb@redhat.com> wrote:
> On Sun, Mar 18, 2012 at 09:47:55PM +0530, shashank rachamalla wrote:
>> >> I guess things are working fine with perf. But why not with oprofile ?
>> >>
>> > Looks like it. I never tried oprofile. Will try to reproduce your
>> > problem and see what oprofile is doing.
>>
>> I am using ubuntu 10.04 with 2.6.32-21-generic kernel as guest and
>> oprofile 0.9.6.
>> Also, I have tried to capture kvm-events ( perf patch ) in host while
>> running oprofile and perf in guest.
>> Please see the attachment. I have run the tests in three cases for the
>> around 5 secs.
>>
>> There are more number of MSR reads and writes in case of perf which I
>> think is normal. However, there are very few MSR reads and writes with
>> oprofile. Also, the number of NMI exceptions are too high in case of
>> oprofile.
>>
> Which host kernel are you using? Try latest kvm.git and check if you see
> something unusual in dmesg.

Currenly running 3.3.0-rc5. will try with the latest source from kvm
git and let you know.


>
> --
>                        Gleb.

Re: Questing regarding KVM Guest PMU

[Gleb Natapov]

On Mon, Mar 19, 2012 at 12:20:30PM +0530, shashank rachamalla wrote:
> On Sun, Mar 18, 2012 at 10:21 PM, Gleb Natapov <gleb@redhat.com> wrote:
> > On Sun, Mar 18, 2012 at 09:47:55PM +0530, shashank rachamalla wrote:
> >> >> I guess things are working fine with perf. But why not with oprofile ?
> >> >>
> >> > Looks like it. I never tried oprofile. Will try to reproduce your
> >> > problem and see what oprofile is doing.
> >>
> >> I am using ubuntu 10.04 with 2.6.32-21-generic kernel as guest and
> >> oprofile 0.9.6.
> >> Also, I have tried to capture kvm-events ( perf patch ) in host while
> >> running oprofile and perf in guest.
> >> Please see the attachment. I have run the tests in three cases for the
> >> around 5 secs.
> >>
> >> There are more number of MSR reads and writes in case of perf which I
> >> think is normal. However, there are very few MSR reads and writes with
> >> oprofile. Also, the number of NMI exceptions are too high in case of
> >> oprofile.
> >>
> > Which host kernel are you using? Try latest kvm.git and check if you see
> > something unusual in dmesg.
> 
> Currenly running 3.3.0-rc5. will try with the latest source from kvm
> git and let you know.
> 
> 
Thanks, there were some fixes that didn't make it into 3.3. rdpmc
instruction emulation fix is one of them. If oprofile uses it this can
explain the problem.

--
			Gleb.

Re: Questing regarding KVM Guest PMU

[shashank rachamalla]

On Mon, Mar 19, 2012 at 12:37 PM, Gleb Natapov <gleb@redhat.com> wrote:
> On Mon, Mar 19, 2012 at 12:20:30PM +0530, shashank rachamalla wrote:
>> On Sun, Mar 18, 2012 at 10:21 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> > On Sun, Mar 18, 2012 at 09:47:55PM +0530, shashank rachamalla wrote:
>> >> >> I guess things are working fine with perf. But why not with oprofile ?
>> >> >>
>> >> > Looks like it. I never tried oprofile. Will try to reproduce your
>> >> > problem and see what oprofile is doing.
>> >>
>> >> I am using ubuntu 10.04 with 2.6.32-21-generic kernel as guest and
>> >> oprofile 0.9.6.
>> >> Also, I have tried to capture kvm-events ( perf patch ) in host while
>> >> running oprofile and perf in guest.
>> >> Please see the attachment. I have run the tests in three cases for the
>> >> around 5 secs.
>> >>
>> >> There are more number of MSR reads and writes in case of perf which I
>> >> think is normal. However, there are very few MSR reads and writes with
>> >> oprofile. Also, the number of NMI exceptions are too high in case of
>> >> oprofile.
>> >>
>> > Which host kernel are you using? Try latest kvm.git and check if you see
>> > something unusual in dmesg.
>>
>> Currenly running 3.3.0-rc5. will try with the latest source from kvm
>> git and let you know.
>>
>>
> Thanks, there were some fixes that didn't make it into 3.3. rdpmc
> instruction emulation fix is one of them. If oprofile uses it this can
> explain the problem.
>
I have tried with latest kvm source from git and also with 3.0 guest
kernel but oprofile fails to collect any samples on guest. I am using
a core2duo processor which is considered by oprofile as pentium pro
model.

> --
>                        Gleb.

Re: Questing regarding KVM Guest PMU

[Gleb Natapov]

On Tue, Apr 03, 2012 at 07:20:04PM +0530, shashank rachamalla wrote:
> On Mon, Mar 19, 2012 at 12:37 PM, Gleb Natapov <gleb@redhat.com> wrote:
> > On Mon, Mar 19, 2012 at 12:20:30PM +0530, shashank rachamalla wrote:
> >> On Sun, Mar 18, 2012 at 10:21 PM, Gleb Natapov <gleb@redhat.com> wrote:
> >> > On Sun, Mar 18, 2012 at 09:47:55PM +0530, shashank rachamalla wrote:
> >> >> >> I guess things are working fine with perf. But why not with oprofile ?
> >> >> >>
> >> >> > Looks like it. I never tried oprofile. Will try to reproduce your
> >> >> > problem and see what oprofile is doing.
> >> >>
> >> >> I am using ubuntu 10.04 with 2.6.32-21-generic kernel as guest and
> >> >> oprofile 0.9.6.
> >> >> Also, I have tried to capture kvm-events ( perf patch ) in host while
> >> >> running oprofile and perf in guest.
> >> >> Please see the attachment. I have run the tests in three cases for the
> >> >> around 5 secs.
> >> >>
> >> >> There are more number of MSR reads and writes in case of perf which I
> >> >> think is normal. However, there are very few MSR reads and writes with
> >> >> oprofile. Also, the number of NMI exceptions are too high in case of
> >> >> oprofile.
> >> >>
> >> > Which host kernel are you using? Try latest kvm.git and check if you see
> >> > something unusual in dmesg.
> >>
> >> Currenly running 3.3.0-rc5. will try with the latest source from kvm
> >> git and let you know.
> >>
> >>
> > Thanks, there were some fixes that didn't make it into 3.3. rdpmc
> > instruction emulation fix is one of them. If oprofile uses it this can
> > explain the problem.
> >
> I have tried with latest kvm source from git and also with 3.0 guest
> kernel but oprofile fails to collect any samples on guest. I am using
> a core2duo processor which is considered by oprofile as pentium pro
> model.
> 
core2duo on the host or the guest? What is your qemu command line?

--
			Gleb.

Re: Questing regarding KVM Guest PMU

[shashank rachamalla]

On Tue, Apr 3, 2012 at 10:28 PM, Gleb Natapov <gleb@redhat.com> wrote:
> On Tue, Apr 03, 2012 at 07:20:04PM +0530, shashank rachamalla wrote:
>> On Mon, Mar 19, 2012 at 12:37 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> > On Mon, Mar 19, 2012 at 12:20:30PM +0530, shashank rachamalla wrote:
>> >> On Sun, Mar 18, 2012 at 10:21 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> >> > On Sun, Mar 18, 2012 at 09:47:55PM +0530, shashank rachamalla wrote:
>> >> >> >> I guess things are working fine with perf. But why not with oprofile ?
>> >> >> >>
>> >> >> > Looks like it. I never tried oprofile. Will try to reproduce your
>> >> >> > problem and see what oprofile is doing.
>> >> >>
>> >> >> I am using ubuntu 10.04 with 2.6.32-21-generic kernel as guest and
>> >> >> oprofile 0.9.6.
>> >> >> Also, I have tried to capture kvm-events ( perf patch ) in host while
>> >> >> running oprofile and perf in guest.
>> >> >> Please see the attachment. I have run the tests in three cases for the
>> >> >> around 5 secs.
>> >> >>
>> >> >> There are more number of MSR reads and writes in case of perf which I
>> >> >> think is normal. However, there are very few MSR reads and writes with
>> >> >> oprofile. Also, the number of NMI exceptions are too high in case of
>> >> >> oprofile.
>> >> >>
>> >> > Which host kernel are you using? Try latest kvm.git and check if you see
>> >> > something unusual in dmesg.
>> >>
>> >> Currenly running 3.3.0-rc5. will try with the latest source from kvm
>> >> git and let you know.
>> >>
>> >>
>> > Thanks, there were some fixes that didn't make it into 3.3. rdpmc
>> > instruction emulation fix is one of them. If oprofile uses it this can
>> > explain the problem.
>> >
>> I have tried with latest kvm source from git and also with 3.0 guest
>> kernel but oprofile fails to collect any samples on guest. I am using
>> a core2duo processor which is considered by oprofile as pentium pro
>> model.
>>
> core2duo on the host or the guest? What is your qemu command line?
>
both. qemu command line below.
sudo /usr/local/bin/qemu-system-x86_64 -drive
file=vdisk1.img,if=virtio -cpu host -m 2000 -net nic,model=virtio -net
user

> --
>                        Gleb.

Re: Questing regarding KVM Guest PMU

[shashank rachamalla]

[-- Attachment #1: Type: text/plain, Size: 2480 bytes --]

On Wed, Apr 4, 2012 at 12:13 AM, shashank rachamalla
<shashank.rachamalla@gmail.com> wrote:
> On Tue, Apr 3, 2012 at 10:28 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> On Tue, Apr 03, 2012 at 07:20:04PM +0530, shashank rachamalla wrote:
>>> On Mon, Mar 19, 2012 at 12:37 PM, Gleb Natapov <gleb@redhat.com> wrote:
>>> > On Mon, Mar 19, 2012 at 12:20:30PM +0530, shashank rachamalla wrote:
>>> >> On Sun, Mar 18, 2012 at 10:21 PM, Gleb Natapov <gleb@redhat.com> wrote:
>>> >> > On Sun, Mar 18, 2012 at 09:47:55PM +0530, shashank rachamalla wrote:
>>> >> >> >> I guess things are working fine with perf. But why not with oprofile ?
>>> >> >> >>
>>> >> >> > Looks like it. I never tried oprofile. Will try to reproduce your
>>> >> >> > problem and see what oprofile is doing.
>>> >> >>
>>> >> >> I am using ubuntu 10.04 with 2.6.32-21-generic kernel as guest and
>>> >> >> oprofile 0.9.6.
>>> >> >> Also, I have tried to capture kvm-events ( perf patch ) in host while
>>> >> >> running oprofile and perf in guest.
>>> >> >> Please see the attachment. I have run the tests in three cases for the
>>> >> >> around 5 secs.
>>> >> >>
>>> >> >> There are more number of MSR reads and writes in case of perf which I
>>> >> >> think is normal. However, there are very few MSR reads and writes with
>>> >> >> oprofile. Also, the number of NMI exceptions are too high in case of
>>> >> >> oprofile.
>>> >> >>
>>> >> > Which host kernel are you using? Try latest kvm.git and check if you see
>>> >> > something unusual in dmesg.
>>> >>
>>> >> Currenly running 3.3.0-rc5. will try with the latest source from kvm
>>> >> git and let you know.
>>> >>
>>> >>
>>> > Thanks, there were some fixes that didn't make it into 3.3. rdpmc
>>> > instruction emulation fix is one of them. If oprofile uses it this can
>>> > explain the problem.
>>> >
>>> I have tried with latest kvm source from git and also with 3.0 guest
>>> kernel but oprofile fails to collect any samples on guest. I am using
>>> a core2duo processor which is considered by oprofile as pentium pro
>>> model.
>>>
>> core2duo on the host or the guest? What is your qemu command line?
>>
> both. qemu command line below.
> sudo /usr/local/bin/qemu-system-x86_64 -drive
> file=vdisk1.img,if=virtio -cpu host -m 2000 -net nic,model=virtio -net
> user
>

please find more info ( /proc/cpuinfo and uname of both host and guest
) in attached files.

>> --
>>                        Gleb.

[-- Attachment #2: host_proc.txt --]
[-- Type: text/plain, Size: 1622 bytes --]

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Duo CPU     T7500  @ 2.20GHz
stepping	: 11
microcode	: 0xb3
cpu MHz		: 800.000
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Duo CPU     T7500  @ 2.20GHz
stepping	: 11
microcode	: 0xb3
cpu MHz		: 800.000
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:


[-- Attachment #3: host_uname.txt --]
[-- Type: text/plain, Size: 98 bytes --]

Linux shashank-Aspire-5920 3.3.0-rc5+ #3 SMP Tue Apr 3 22:42:29 IST 2012 i686 i686 i386 GNU/Linux

[-- Attachment #4: guest_proc.txt --]
[-- Type: text/plain, Size: 591 bytes --]

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Duo CPU     T7500  @ 2.20GHz
stepping	: 11
cpu MHz		: 2199.996
cache size	: 4096 KB
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss constant_tsc up arch_perfmon pni ssse3 cx16 hypervisor lahf_lm
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 0 bits virtual
power management:


[-- Attachment #5: guest_uname.txt --]
[-- Type: text/plain, Size: 116 bytes --]

Linux shashank-virtual-machine 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7 14:50:42 UTC 2011 i686 i686 i386 GNU/Linux

Re: Questing regarding KVM Guest PMU

[Gleb Natapov]

On Wed, Apr 04, 2012 at 12:24:17AM +0530, shashank rachamalla wrote:
> On Wed, Apr 4, 2012 at 12:13 AM, shashank rachamalla
> <shashank.rachamalla@gmail.com> wrote:
> > On Tue, Apr 3, 2012 at 10:28 PM, Gleb Natapov <gleb@redhat.com> wrote:
> >> On Tue, Apr 03, 2012 at 07:20:04PM +0530, shashank rachamalla wrote:
> >>> On Mon, Mar 19, 2012 at 12:37 PM, Gleb Natapov <gleb@redhat.com> wrote:
> >>> > On Mon, Mar 19, 2012 at 12:20:30PM +0530, shashank rachamalla wrote:
> >>> >> On Sun, Mar 18, 2012 at 10:21 PM, Gleb Natapov <gleb@redhat.com> wrote:
> >>> >> > On Sun, Mar 18, 2012 at 09:47:55PM +0530, shashank rachamalla wrote:
> >>> >> >> >> I guess things are working fine with perf. But why not with oprofile ?
> >>> >> >> >>
> >>> >> >> > Looks like it. I never tried oprofile. Will try to reproduce your
> >>> >> >> > problem and see what oprofile is doing.
> >>> >> >>
> >>> >> >> I am using ubuntu 10.04 with 2.6.32-21-generic kernel as guest and
> >>> >> >> oprofile 0.9.6.
> >>> >> >> Also, I have tried to capture kvm-events ( perf patch ) in host while
> >>> >> >> running oprofile and perf in guest.
> >>> >> >> Please see the attachment. I have run the tests in three cases for the
> >>> >> >> around 5 secs.
> >>> >> >>
> >>> >> >> There are more number of MSR reads and writes in case of perf which I
> >>> >> >> think is normal. However, there are very few MSR reads and writes with
> >>> >> >> oprofile. Also, the number of NMI exceptions are too high in case of
> >>> >> >> oprofile.
> >>> >> >>
> >>> >> > Which host kernel are you using? Try latest kvm.git and check if you see
> >>> >> > something unusual in dmesg.
> >>> >>
> >>> >> Currenly running 3.3.0-rc5. will try with the latest source from kvm
> >>> >> git and let you know.
> >>> >>
> >>> >>
> >>> > Thanks, there were some fixes that didn't make it into 3.3. rdpmc
> >>> > instruction emulation fix is one of them. If oprofile uses it this can
> >>> > explain the problem.
> >>> >
> >>> I have tried with latest kvm source from git and also with 3.0 guest
> >>> kernel but oprofile fails to collect any samples on guest. I am using
> >>> a core2duo processor which is considered by oprofile as pentium pro
> >>> model.
> >>>
> >> core2duo on the host or the guest? What is your qemu command line?
> >>
> > both. qemu command line below.
> > sudo /usr/local/bin/qemu-system-x86_64 -drive
> > file=vdisk1.img,if=virtio -cpu host -m 2000 -net nic,model=virtio -net
> > user
> >
> 
> please find more info ( /proc/cpuinfo and uname of both host and guest
> ) in attached files.
> 
oprofile does not work for me even on the host. After trying to use it I can
see why perf was written in the first place.

--
			Gleb.

Re: Questing regarding KVM Guest PMU

[shashank rachamalla]

On Wed, Apr 4, 2012 at 12:34 PM, Gleb Natapov <gleb@redhat.com> wrote:
> On Wed, Apr 04, 2012 at 12:24:17AM +0530, shashank rachamalla wrote:
>> On Wed, Apr 4, 2012 at 12:13 AM, shashank rachamalla
>> <shashank.rachamalla@gmail.com> wrote:
>> > On Tue, Apr 3, 2012 at 10:28 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> >> On Tue, Apr 03, 2012 at 07:20:04PM +0530, shashank rachamalla wrote:
>> >>> On Mon, Mar 19, 2012 at 12:37 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> >>> > On Mon, Mar 19, 2012 at 12:20:30PM +0530, shashank rachamalla wrote:
>> >>> >> On Sun, Mar 18, 2012 at 10:21 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> >>> >> > On Sun, Mar 18, 2012 at 09:47:55PM +0530, shashank rachamalla wrote:
>> >>> >> >> >> I guess things are working fine with perf. But why not with oprofile ?
>> >>> >> >> >>
>> >>> >> >> > Looks like it. I never tried oprofile. Will try to reproduce your
>> >>> >> >> > problem and see what oprofile is doing.
>> >>> >> >>
>> >>> >> >> I am using ubuntu 10.04 with 2.6.32-21-generic kernel as guest and
>> >>> >> >> oprofile 0.9.6.
>> >>> >> >> Also, I have tried to capture kvm-events ( perf patch ) in host while
>> >>> >> >> running oprofile and perf in guest.
>> >>> >> >> Please see the attachment. I have run the tests in three cases for the
>> >>> >> >> around 5 secs.
>> >>> >> >>
>> >>> >> >> There are more number of MSR reads and writes in case of perf which I
>> >>> >> >> think is normal. However, there are very few MSR reads and writes with
>> >>> >> >> oprofile. Also, the number of NMI exceptions are too high in case of
>> >>> >> >> oprofile.
>> >>> >> >>
>> >>> >> > Which host kernel are you using? Try latest kvm.git and check if you see
>> >>> >> > something unusual in dmesg.
>> >>> >>
>> >>> >> Currenly running 3.3.0-rc5. will try with the latest source from kvm
>> >>> >> git and let you know.
>> >>> >>
>> >>> >>
>> >>> > Thanks, there were some fixes that didn't make it into 3.3. rdpmc
>> >>> > instruction emulation fix is one of them. If oprofile uses it this can
>> >>> > explain the problem.
>> >>> >
>> >>> I have tried with latest kvm source from git and also with 3.0 guest
>> >>> kernel but oprofile fails to collect any samples on guest. I am using
>> >>> a core2duo processor which is considered by oprofile as pentium pro
>> >>> model.
>> >>>
>> >> core2duo on the host or the guest? What is your qemu command line?
>> >>
>> > both. qemu command line below.
>> > sudo /usr/local/bin/qemu-system-x86_64 -drive
>> > file=vdisk1.img,if=virtio -cpu host -m 2000 -net nic,model=virtio -net
>> > user
>> >
>>
>> please find more info ( /proc/cpuinfo and uname of both host and guest
>> ) in attached files.
>>
> oprofile does not work for me even on the host. After trying to use it I can
> see why perf was written in the first place.
>
ok. seems to be. will move over to perf as its working fine inside guest.

> --
>                        Gleb.

Re: Questing regarding KVM Guest PMU

[Gleb Natapov]

On Wed, Apr 04, 2012 at 03:49:42PM +0530, shashank rachamalla wrote:
> tatus: RO
> Content-Length: 2989
> Lines: 79
> 
> On Wed, Apr 4, 2012 at 12:34 PM, Gleb Natapov <gleb@redhat.com> wrote:
> > On Wed, Apr 04, 2012 at 12:24:17AM +0530, shashank rachamalla wrote:
> >> On Wed, Apr 4, 2012 at 12:13 AM, shashank rachamalla
> >> <shashank.rachamalla@gmail.com> wrote:
> >> > On Tue, Apr 3, 2012 at 10:28 PM, Gleb Natapov <gleb@redhat.com> wrote:
> >> >> On Tue, Apr 03, 2012 at 07:20:04PM +0530, shashank rachamalla wrote:
> >> >>> On Mon, Mar 19, 2012 at 12:37 PM, Gleb Natapov <gleb@redhat.com> wrote:
> >> >>> > On Mon, Mar 19, 2012 at 12:20:30PM +0530, shashank rachamalla wrote:
> >> >>> >> On Sun, Mar 18, 2012 at 10:21 PM, Gleb Natapov <gleb@redhat.com> wrote:
> >> >>> >> > On Sun, Mar 18, 2012 at 09:47:55PM +0530, shashank rachamalla wrote:
> >> >>> >> >> >> I guess things are working fine with perf. But why not with oprofile ?
> >> >>> >> >> >>
> >> >>> >> >> > Looks like it. I never tried oprofile. Will try to reproduce your
> >> >>> >> >> > problem and see what oprofile is doing.
> >> >>> >> >>
> >> >>> >> >> I am using ubuntu 10.04 with 2.6.32-21-generic kernel as guest and
> >> >>> >> >> oprofile 0.9.6.
> >> >>> >> >> Also, I have tried to capture kvm-events ( perf patch ) in host while
> >> >>> >> >> running oprofile and perf in guest.
> >> >>> >> >> Please see the attachment. I have run the tests in three cases for the
> >> >>> >> >> around 5 secs.
> >> >>> >> >>
> >> >>> >> >> There are more number of MSR reads and writes in case of perf which I
> >> >>> >> >> think is normal. However, there are very few MSR reads and writes with
> >> >>> >> >> oprofile. Also, the number of NMI exceptions are too high in case of
> >> >>> >> >> oprofile.
> >> >>> >> >>
> >> >>> >> > Which host kernel are you using? Try latest kvm.git and check if you see
> >> >>> >> > something unusual in dmesg.
> >> >>> >>
> >> >>> >> Currenly running 3.3.0-rc5. will try with the latest source from kvm
> >> >>> >> git and let you know.
> >> >>> >>
> >> >>> >>
> >> >>> > Thanks, there were some fixes that didn't make it into 3.3. rdpmc
> >> >>> > instruction emulation fix is one of them. If oprofile uses it this can
> >> >>> > explain the problem.
> >> >>> >
> >> >>> I have tried with latest kvm source from git and also with 3.0 guest
> >> >>> kernel but oprofile fails to collect any samples on guest. I am using
> >> >>> a core2duo processor which is considered by oprofile as pentium pro
> >> >>> model.
> >> >>>
> >> >> core2duo on the host or the guest? What is your qemu command line?
> >> >>
> >> > both. qemu command line below.
> >> > sudo /usr/local/bin/qemu-system-x86_64 -drive
> >> > file=vdisk1.img,if=virtio -cpu host -m 2000 -net nic,model=virtio -net
> >> > user
> >> >
> >>
> >> please find more info ( /proc/cpuinfo and uname of both host and guest
> >> ) in attached files.
> >>
> > oprofile does not work for me even on the host. After trying to use it I can
> > see why perf was written in the first place.
> >
> ok. seems to be. will move over to perf as its working fine inside guest.
> 
Good riddance IMO. I managed to run it on a guest (but not on my
host!). The thing is buggy. It does not use global ctrl MSR to enable
counters and kvm has all of them disabled by default. I didn't find what
value this MSR should have after reset, so this may be either kvm bug or
real BIOSes enable all counters in global ctrl MSR for PMUv1
compatibility. Doing "wrmsr 0x38f 0x70000000f" solves this problem. The
second problem is that oprofile reprogram PMU counters without
disabling them first and this is explicitly prohibited by Intel SDM.
The patch below solve that, but oprofile is the one who should be fixed.

diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c
index a73f0c1..be05028 100644
--- a/arch/x86/kvm/pmu.c
+++ b/arch/x86/kvm/pmu.c
@@ -396,6 +396,7 @@ int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, u32 index, u64 data)
 				(pmc = get_fixed_pmc(pmu, index))) {
 			data = (s64)(s32)data;
 			pmc->counter += data - read_pmc(pmc);
+			reprogram_gp_counter(pmc, pmc->eventsel);
 			return 0;
 		} else if ((pmc = get_gp_pmc(pmu, index, MSR_P6_EVNTSEL0))) {
 			if (data == pmc->eventsel)

--
			Gleb.

Re: Questing regarding KVM Guest PMU

[shashank rachamalla]

On Wed, Apr 4, 2012 at 3:59 PM, Gleb Natapov <gleb@redhat.com> wrote:
> On Wed, Apr 04, 2012 at 03:49:42PM +0530, shashank rachamalla wrote:
>> tatus: RO
>> Content-Length: 2989
>> Lines: 79
>>
>> On Wed, Apr 4, 2012 at 12:34 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> > On Wed, Apr 04, 2012 at 12:24:17AM +0530, shashank rachamalla wrote:
>> >> On Wed, Apr 4, 2012 at 12:13 AM, shashank rachamalla
>> >> <shashank.rachamalla@gmail.com> wrote:
>> >> > On Tue, Apr 3, 2012 at 10:28 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> >> >> On Tue, Apr 03, 2012 at 07:20:04PM +0530, shashank rachamalla wrote:
>> >> >>> On Mon, Mar 19, 2012 at 12:37 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> >> >>> > On Mon, Mar 19, 2012 at 12:20:30PM +0530, shashank rachamalla wrote:
>> >> >>> >> On Sun, Mar 18, 2012 at 10:21 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> >> >>> >> > On Sun, Mar 18, 2012 at 09:47:55PM +0530, shashank rachamalla wrote:
>> >> >>> >> >> >> I guess things are working fine with perf. But why not with oprofile ?
>> >> >>> >> >> >>
>> >> >>> >> >> > Looks like it. I never tried oprofile. Will try to reproduce your
>> >> >>> >> >> > problem and see what oprofile is doing.
>> >> >>> >> >>
>> >> >>> >> >> I am using ubuntu 10.04 with 2.6.32-21-generic kernel as guest and
>> >> >>> >> >> oprofile 0.9.6.
>> >> >>> >> >> Also, I have tried to capture kvm-events ( perf patch ) in host while
>> >> >>> >> >> running oprofile and perf in guest.
>> >> >>> >> >> Please see the attachment. I have run the tests in three cases for the
>> >> >>> >> >> around 5 secs.
>> >> >>> >> >>
>> >> >>> >> >> There are more number of MSR reads and writes in case of perf which I
>> >> >>> >> >> think is normal. However, there are very few MSR reads and writes with
>> >> >>> >> >> oprofile. Also, the number of NMI exceptions are too high in case of
>> >> >>> >> >> oprofile.
>> >> >>> >> >>
>> >> >>> >> > Which host kernel are you using? Try latest kvm.git and check if you see
>> >> >>> >> > something unusual in dmesg.
>> >> >>> >>
>> >> >>> >> Currenly running 3.3.0-rc5. will try with the latest source from kvm
>> >> >>> >> git and let you know.
>> >> >>> >>
>> >> >>> >>
>> >> >>> > Thanks, there were some fixes that didn't make it into 3.3. rdpmc
>> >> >>> > instruction emulation fix is one of them. If oprofile uses it this can
>> >> >>> > explain the problem.
>> >> >>> >
>> >> >>> I have tried with latest kvm source from git and also with 3.0 guest
>> >> >>> kernel but oprofile fails to collect any samples on guest. I am using
>> >> >>> a core2duo processor which is considered by oprofile as pentium pro
>> >> >>> model.
>> >> >>>
>> >> >> core2duo on the host or the guest? What is your qemu command line?
>> >> >>
>> >> > both. qemu command line below.
>> >> > sudo /usr/local/bin/qemu-system-x86_64 -drive
>> >> > file=vdisk1.img,if=virtio -cpu host -m 2000 -net nic,model=virtio -net
>> >> > user
>> >> >
>> >>
>> >> please find more info ( /proc/cpuinfo and uname of both host and guest
>> >> ) in attached files.
>> >>
>> > oprofile does not work for me even on the host. After trying to use it I can
>> > see why perf was written in the first place.
>> >
>> ok. seems to be. will move over to perf as its working fine inside guest.
>>
> Good riddance IMO. I managed to run it on a guest (but not on my
> host!). The thing is buggy. It does not use global ctrl MSR to enable
> counters and kvm has all of them disabled by default. I didn't find what
> value this MSR should have after reset, so this may be either kvm bug or
> real BIOSes enable all counters in global ctrl MSR for PMUv1
> compatibility. Doing "wrmsr 0x38f 0x70000000f" solves this problem. The
> second problem is that oprofile reprogram PMU counters without
> disabling them first and this is explicitly prohibited by Intel SDM.
> The patch below solve that, but oprofile is the one who should be fixed.
>
> diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c
> index a73f0c1..be05028 100644
> --- a/arch/x86/kvm/pmu.c
> +++ b/arch/x86/kvm/pmu.c
> @@ -396,6 +396,7 @@ int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, u32 index, u64 data)
>                                (pmc = get_fixed_pmc(pmu, index))) {
>                        data = (s64)(s32)data;
>                        pmc->counter += data - read_pmc(pmc);
> +                       reprogram_gp_counter(pmc, pmc->eventsel);
>                        return 0;
>                } else if ((pmc = get_gp_pmc(pmu, index, MSR_P6_EVNTSEL0))) {
>                        if (data == pmc->eventsel)
>
> --
>                        Gleb.

thanks for the patch. will check it out.

Re: Questing regarding KVM Guest PMU

[Avi Kivity]

On 04/04/2012 01:29 PM, Gleb Natapov wrote:
> > >
> > ok. seems to be. will move over to perf as its working fine inside guest.
> > 
> Good riddance IMO. I managed to run it on a guest (but not on my
> host!). The thing is buggy. It does not use global ctrl MSR to enable
> counters and kvm has all of them disabled by default. I didn't find what
> value this MSR should have after reset, so this may be either kvm bug or
> real BIOSes enable all counters in global ctrl MSR for PMUv1
> compatibility. Doing "wrmsr 0x38f 0x70000000f" solves this problem. The
> second problem is that oprofile reprogram PMU counters without
> disabling them first and this is explicitly prohibited by Intel SDM.
> The patch below solve that, but oprofile is the one who should be fixed.

Both should be fixed, there may be other profilers affected.

>
> diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c
> index a73f0c1..be05028 100644
> --- a/arch/x86/kvm/pmu.c
> +++ b/arch/x86/kvm/pmu.c
> @@ -396,6 +396,7 @@ int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, u32 index, u64 data)
>  				(pmc = get_fixed_pmc(pmu, index))) {
>  			data = (s64)(s32)data;
>  			pmc->counter += data - read_pmc(pmc);
> +			reprogram_gp_counter(pmc, pmc->eventsel);
>  			return 0;
>  		} else if ((pmc = get_gp_pmc(pmu, index, MSR_P6_EVNTSEL0))) {
>  			if (data == pmc->eventsel)
>

-- 
error compiling committee.c: too many arguments to function

Re: Questing regarding KVM Guest PMU

[Gleb Natapov]

On Thu, Apr 05, 2012 at 03:27:18PM +0300, Avi Kivity wrote:
> On 04/04/2012 01:29 PM, Gleb Natapov wrote:
> > > >
> > > ok. seems to be. will move over to perf as its working fine inside guest.
> > > 
> > Good riddance IMO. I managed to run it on a guest (but not on my
> > host!). The thing is buggy. It does not use global ctrl MSR to enable
> > counters and kvm has all of them disabled by default. I didn't find what
> > value this MSR should have after reset, so this may be either kvm bug or
> > real BIOSes enable all counters in global ctrl MSR for PMUv1
> > compatibility. Doing "wrmsr 0x38f 0x70000000f" solves this problem. The
> > second problem is that oprofile reprogram PMU counters without
> > disabling them first and this is explicitly prohibited by Intel SDM.
> > The patch below solve that, but oprofile is the one who should be fixed.
> 
> Both should be fixed, there may be other profilers affected.
> 
Global ctrl msr issue I need to investigate further, but second one is
bug that should be fixed in oprofile. Intel spec clearly says:

 EN (Enable Counters) Flag (bit 22) — When set, performance counting
 is enabled in the corresponding performance-monitoring counter; when
 clear, the corresponding counter is disabled. The event logic unit
 for a UMASK must be disabled by setting IA32_PERFEVTSELx[bit 22] = 0,
 before writing to IA32_PMCx.

I suspect that on real HW they got wrong result too. It simply subtly
wrong, so it is not as noticeable as with kvm.

> >
> > diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c
> > index a73f0c1..be05028 100644
> > --- a/arch/x86/kvm/pmu.c
> > +++ b/arch/x86/kvm/pmu.c
> > @@ -396,6 +396,7 @@ int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, u32 index, u64 data)
> >  				(pmc = get_fixed_pmc(pmu, index))) {
> >  			data = (s64)(s32)data;
> >  			pmc->counter += data - read_pmc(pmc);
> > +			reprogram_gp_counter(pmc, pmc->eventsel);
> >  			return 0;
> >  		} else if ((pmc = get_gp_pmc(pmu, index, MSR_P6_EVNTSEL0))) {
> >  			if (data == pmc->eventsel)
> >
> 
> -- 
> error compiling committee.c: too many arguments to function

--
			Gleb.

Re: Questing regarding KVM Guest PMU

[Avi Kivity]

On 04/05/2012 03:37 PM, Gleb Natapov wrote:
> On Thu, Apr 05, 2012 at 03:27:18PM +0300, Avi Kivity wrote:
> > On 04/04/2012 01:29 PM, Gleb Natapov wrote:
> > > > >
> > > > ok. seems to be. will move over to perf as its working fine inside guest.
> > > > 
> > > Good riddance IMO. I managed to run it on a guest (but not on my
> > > host!). The thing is buggy. It does not use global ctrl MSR to enable
> > > counters and kvm has all of them disabled by default. I didn't find what
> > > value this MSR should have after reset, so this may be either kvm bug or
> > > real BIOSes enable all counters in global ctrl MSR for PMUv1
> > > compatibility. Doing "wrmsr 0x38f 0x70000000f" solves this problem. The
> > > second problem is that oprofile reprogram PMU counters without
> > > disabling them first and this is explicitly prohibited by Intel SDM.
> > > The patch below solve that, but oprofile is the one who should be fixed.
> > 
> > Both should be fixed, there may be other profilers affected.
> > 
> Global ctrl msr issue I need to investigate further, but second one is
> bug that should be fixed in oprofile. Intel spec clearly says:
>
>  EN (Enable Counters) Flag (bit 22) — When set, performance counting
>  is enabled in the corresponding performance-monitoring counter; when
>  clear, the corresponding counter is disabled. The event logic unit
>  for a UMASK must be disabled by setting IA32_PERFEVTSELx[bit 22] = 0,
>  before writing to IA32_PMCx.
>
> I suspect that on real HW they got wrong result too. It simply subtly
> wrong, so it is not as noticeable as with kvm.
>

First, there is the standard Linus rant to never break userspace (and
for us a guest kernel is userspace).  Second, the Intel rule may have
been added later (as preparation for a change in behaviour), so it may
actually work correctly on older hardware.

-- 
error compiling committee.c: too many arguments to function

Re: Questing regarding KVM Guest PMU

[Gleb Natapov]

On Thu, Apr 05, 2012 at 03:48:51PM +0300, Avi Kivity wrote:
> On 04/05/2012 03:37 PM, Gleb Natapov wrote:
> > On Thu, Apr 05, 2012 at 03:27:18PM +0300, Avi Kivity wrote:
> > > On 04/04/2012 01:29 PM, Gleb Natapov wrote:
> > > > > >
> > > > > ok. seems to be. will move over to perf as its working fine inside guest.
> > > > > 
> > > > Good riddance IMO. I managed to run it on a guest (but not on my
> > > > host!). The thing is buggy. It does not use global ctrl MSR to enable
> > > > counters and kvm has all of them disabled by default. I didn't find what
> > > > value this MSR should have after reset, so this may be either kvm bug or
> > > > real BIOSes enable all counters in global ctrl MSR for PMUv1
> > > > compatibility. Doing "wrmsr 0x38f 0x70000000f" solves this problem. The
> > > > second problem is that oprofile reprogram PMU counters without
> > > > disabling them first and this is explicitly prohibited by Intel SDM.
> > > > The patch below solve that, but oprofile is the one who should be fixed.
> > > 
> > > Both should be fixed, there may be other profilers affected.
> > > 
> > Global ctrl msr issue I need to investigate further, but second one is
> > bug that should be fixed in oprofile. Intel spec clearly says:
> >
> >  EN (Enable Counters) Flag (bit 22) — When set, performance counting
> >  is enabled in the corresponding performance-monitoring counter; when
> >  clear, the corresponding counter is disabled. The event logic unit
> >  for a UMASK must be disabled by setting IA32_PERFEVTSELx[bit 22] = 0,
> >  before writing to IA32_PMCx.
> >
> > I suspect that on real HW they got wrong result too. It simply subtly
> > wrong, so it is not as noticeable as with kvm.
> >
> 
> First, there is the standard Linus rant to never break userspace (and
We broke nothing. Oprofile is broken and never could have worked
according to Intel spec. (It didn't manage to get any result from it on
real CPU, but may be this is unrelated).

> for us a guest kernel is userspace).  Second, the Intel rule may have
> been added later (as preparation for a change in behaviour), so it may
> actually work correctly on older hardware.
> 
May be (the rule is specified for MPUv1 BTW), but we emulate newer HW,
the one that spec describes and we report this in cpuid leaf 10. I
highly doubt Intel would change CPU in a way that may make old software
to work incorrectly on newer cpus.

--
			Gleb.

Re: Questing regarding KVM Guest PMU

[Avi Kivity]

On 04/05/2012 04:26 PM, Gleb Natapov wrote:
> On Thu, Apr 05, 2012 at 03:48:51PM +0300, Avi Kivity wrote:
> > On 04/05/2012 03:37 PM, Gleb Natapov wrote:
> > > On Thu, Apr 05, 2012 at 03:27:18PM +0300, Avi Kivity wrote:
> > > > On 04/04/2012 01:29 PM, Gleb Natapov wrote:
> > > > > > >
> > > > > > ok. seems to be. will move over to perf as its working fine inside guest.
> > > > > > 
> > > > > Good riddance IMO. I managed to run it on a guest (but not on my
> > > > > host!). The thing is buggy. It does not use global ctrl MSR to enable
> > > > > counters and kvm has all of them disabled by default. I didn't find what
> > > > > value this MSR should have after reset, so this may be either kvm bug or
> > > > > real BIOSes enable all counters in global ctrl MSR for PMUv1
> > > > > compatibility. Doing "wrmsr 0x38f 0x70000000f" solves this problem. The
> > > > > second problem is that oprofile reprogram PMU counters without
> > > > > disabling them first and this is explicitly prohibited by Intel SDM.
> > > > > The patch below solve that, but oprofile is the one who should be fixed.
> > > > 
> > > > Both should be fixed, there may be other profilers affected.
> > > > 
> > > Global ctrl msr issue I need to investigate further, but second one is
> > > bug that should be fixed in oprofile. Intel spec clearly says:
> > >
> > >  EN (Enable Counters) Flag (bit 22) — When set, performance counting
> > >  is enabled in the corresponding performance-monitoring counter; when
> > >  clear, the corresponding counter is disabled. The event logic unit
> > >  for a UMASK must be disabled by setting IA32_PERFEVTSELx[bit 22] = 0,
> > >  before writing to IA32_PMCx.
> > >
> > > I suspect that on real HW they got wrong result too. It simply subtly
> > > wrong, so it is not as noticeable as with kvm.
> > >
> > 
> > First, there is the standard Linus rant to never break userspace (and
> We broke nothing. Oprofile is broken and never could have worked
> according to Intel spec. (It didn't manage to get any result from it on
> real CPU, but may be this is unrelated).

But it did work sometime in the past, I've used it.

> > for us a guest kernel is userspace).  Second, the Intel rule may have
> > been added later (as preparation for a change in behaviour), so it may
> > actually work correctly on older hardware.
> > 
> May be (the rule is specified for MPUv1 BTW), but we emulate newer HW,
> the one that spec describes and we report this in cpuid leaf 10. I
> highly doubt Intel would change CPU in a way that may make old software
> to work incorrectly on newer cpus.

The non-architectural PMU is version specific, so they could have made
this change before arch PMU was introduced.

-- 
error compiling committee.c: too many arguments to function

Re: Questing regarding KVM Guest PMU

[Gleb Natapov]

On Thu, Apr 05, 2012 at 04:28:48PM +0300, Avi Kivity wrote:
> On 04/05/2012 04:26 PM, Gleb Natapov wrote:
> > On Thu, Apr 05, 2012 at 03:48:51PM +0300, Avi Kivity wrote:
> > > On 04/05/2012 03:37 PM, Gleb Natapov wrote:
> > > > On Thu, Apr 05, 2012 at 03:27:18PM +0300, Avi Kivity wrote:
> > > > > On 04/04/2012 01:29 PM, Gleb Natapov wrote:
> > > > > > > >
> > > > > > > ok. seems to be. will move over to perf as its working fine inside guest.
> > > > > > > 
> > > > > > Good riddance IMO. I managed to run it on a guest (but not on my
> > > > > > host!). The thing is buggy. It does not use global ctrl MSR to enable
> > > > > > counters and kvm has all of them disabled by default. I didn't find what
> > > > > > value this MSR should have after reset, so this may be either kvm bug or
> > > > > > real BIOSes enable all counters in global ctrl MSR for PMUv1
> > > > > > compatibility. Doing "wrmsr 0x38f 0x70000000f" solves this problem. The
> > > > > > second problem is that oprofile reprogram PMU counters without
> > > > > > disabling them first and this is explicitly prohibited by Intel SDM.
> > > > > > The patch below solve that, but oprofile is the one who should be fixed.
> > > > > 
> > > > > Both should be fixed, there may be other profilers affected.
> > > > > 
> > > > Global ctrl msr issue I need to investigate further, but second one is
> > > > bug that should be fixed in oprofile. Intel spec clearly says:
> > > >
> > > >  EN (Enable Counters) Flag (bit 22) — When set, performance counting
> > > >  is enabled in the corresponding performance-monitoring counter; when
> > > >  clear, the corresponding counter is disabled. The event logic unit
> > > >  for a UMASK must be disabled by setting IA32_PERFEVTSELx[bit 22] = 0,
> > > >  before writing to IA32_PMCx.
> > > >
> > > > I suspect that on real HW they got wrong result too. It simply subtly
> > > > wrong, so it is not as noticeable as with kvm.
> > > >
> > > 
> > > First, there is the standard Linus rant to never break userspace (and
> > We broke nothing. Oprofile is broken and never could have worked
> > according to Intel spec. (It didn't manage to get any result from it on
> > real CPU, but may be this is unrelated).
> 
> But it did work sometime in the past, I've used it.
> 
May be it used NMI based profiling. We should ask oprofile developers.
As I said I am almost sure my inability to run it on a host is probably
PEBKAC, although I ran the same script exactly on the host and the
guest (the script is from the first email of this thread)

> > > for us a guest kernel is userspace).  Second, the Intel rule may have
> > > been added later (as preparation for a change in behaviour), so it may
> > > actually work correctly on older hardware.
> > > 
> > May be (the rule is specified for MPUv1 BTW), but we emulate newer HW,
> > the one that spec describes and we report this in cpuid leaf 10. I
> > highly doubt Intel would change CPU in a way that may make old software
> > to work incorrectly on newer cpus.
> 
> The non-architectural PMU is version specific, so they could have made
> this change before arch PMU was introduced.
> 
So Table 18-11 section 18.4.4.3 Writing a PEBS Interrupt Service Routine
has:
 For Processors based on Intel Core microarchitecture "Counters MUST be
 stopped before writing".

"Must" is written in all caps in the spec. For NetBurst this is optional.
Really, I fail to see how this is not oprofile bug.

--
			Gleb.

Re: Questing regarding KVM Guest PMU

[Gleb Natapov]

On Thu, Apr 05, 2012 at 04:48:57PM +0300, Gleb Natapov wrote:
> On Thu, Apr 05, 2012 at 04:28:48PM +0300, Avi Kivity wrote:
> > On 04/05/2012 04:26 PM, Gleb Natapov wrote:
> > > On Thu, Apr 05, 2012 at 03:48:51PM +0300, Avi Kivity wrote:
> > > > On 04/05/2012 03:37 PM, Gleb Natapov wrote:
> > > > > On Thu, Apr 05, 2012 at 03:27:18PM +0300, Avi Kivity wrote:
> > > > > > On 04/04/2012 01:29 PM, Gleb Natapov wrote:
> > > > > > > > >
> > > > > > > > ok. seems to be. will move over to perf as its working fine inside guest.
> > > > > > > > 
> > > > > > > Good riddance IMO. I managed to run it on a guest (but not on my
> > > > > > > host!). The thing is buggy. It does not use global ctrl MSR to enable
> > > > > > > counters and kvm has all of them disabled by default. I didn't find what
> > > > > > > value this MSR should have after reset, so this may be either kvm bug or
> > > > > > > real BIOSes enable all counters in global ctrl MSR for PMUv1
> > > > > > > compatibility. Doing "wrmsr 0x38f 0x70000000f" solves this problem. The
> > > > > > > second problem is that oprofile reprogram PMU counters without
> > > > > > > disabling them first and this is explicitly prohibited by Intel SDM.
> > > > > > > The patch below solve that, but oprofile is the one who should be fixed.
> > > > > > 
> > > > > > Both should be fixed, there may be other profilers affected.
> > > > > > 
> > > > > Global ctrl msr issue I need to investigate further, but second one is
> > > > > bug that should be fixed in oprofile. Intel spec clearly says:
> > > > >
> > > > >  EN (Enable Counters) Flag (bit 22) — When set, performance counting
> > > > >  is enabled in the corresponding performance-monitoring counter; when
> > > > >  clear, the corresponding counter is disabled. The event logic unit
> > > > >  for a UMASK must be disabled by setting IA32_PERFEVTSELx[bit 22] = 0,
> > > > >  before writing to IA32_PMCx.
> > > > >
> > > > > I suspect that on real HW they got wrong result too. It simply subtly
> > > > > wrong, so it is not as noticeable as with kvm.
> > > > >
> > > > 
> > > > First, there is the standard Linus rant to never break userspace (and
> > > We broke nothing. Oprofile is broken and never could have worked
> > > according to Intel spec. (It didn't manage to get any result from it on
> > > real CPU, but may be this is unrelated).
> > 
> > But it did work sometime in the past, I've used it.
> > 
> May be it used NMI based profiling. We should ask oprofile developers.
> As I said I am almost sure my inability to run it on a host is probably
> PEBKAC, although I ran the same script exactly on the host and the
> guest (the script is from the first email of this thread)
> 
After upgrading the kernel to latest git from whatever it was there the
same script works and counts CPU_CLK_UNHALT events.

--
			Gleb.

Re: Questing regarding KVM Guest PMU

[Avi Kivity]

On 04/05/2012 04:57 PM, Gleb Natapov wrote:
> > > 
> > May be it used NMI based profiling. We should ask oprofile developers.
> > As I said I am almost sure my inability to run it on a host is probably
> > PEBKAC, although I ran the same script exactly on the host and the
> > guest (the script is from the first email of this thread)
> > 
> After upgrading the kernel to latest git from whatever it was there the
> same script works and counts CPU_CLK_UNHALT events.
>

This is even while it violates the Intel guidelines?

-- 
error compiling committee.c: too many arguments to function

Re: Questing regarding KVM Guest PMU

[Gleb Natapov]

On Thu, Apr 05, 2012 at 05:38:40PM +0300, Avi Kivity wrote:
> On 04/05/2012 04:57 PM, Gleb Natapov wrote:
> > > > 
> > > May be it used NMI based profiling. We should ask oprofile developers.
> > > As I said I am almost sure my inability to run it on a host is probably
> > > PEBKAC, although I ran the same script exactly on the host and the
> > > guest (the script is from the first email of this thread)
> > > 
> > After upgrading the kernel to latest git from whatever it was there the
> > same script works and counts CPU_CLK_UNHALT events.
> >
> 
> This is even while it violates the Intel guidelines?
> 
Yes, but who says the result is correct :) It seems that we handle
global ctrl msr wrong. That is counter can be enabled either in global
ctrl or in eventsel. Trying to confirm that.

--
			Gleb.

Re: Questing regarding KVM Guest PMU

[shashank rachamalla]

On Thu, Apr 5, 2012 at 8:11 PM, Gleb Natapov <gleb@redhat.com> wrote:
> On Thu, Apr 05, 2012 at 05:38:40PM +0300, Avi Kivity wrote:
>> On 04/05/2012 04:57 PM, Gleb Natapov wrote:
>> > > >
>> > > May be it used NMI based profiling. We should ask oprofile developers.
>> > > As I said I am almost sure my inability to run it on a host is probably
>> > > PEBKAC, although I ran the same script exactly on the host and the
>> > > guest (the script is from the first email of this thread)
>> > >
>> > After upgrading the kernel to latest git from whatever it was there the
>> > same script works and counts CPU_CLK_UNHALT events.
>> >
>>
>> This is even while it violates the Intel guidelines?
>>
> Yes, but who says the result is correct :) It seems that we handle
> global ctrl msr wrong. That is counter can be enabled either in global
> ctrl or in eventsel. Trying to confirm that.
>
if that becomes true then will global ctrl msr have any significance ?
> --
>                        Gleb.

Re: Questing regarding KVM Guest PMU

[Gleb Natapov]

On Fri, Apr 06, 2012 at 10:43:17AM +0530, shashank rachamalla wrote:
> On Thu, Apr 5, 2012 at 8:11 PM, Gleb Natapov <gleb@redhat.com> wrote:
> > On Thu, Apr 05, 2012 at 05:38:40PM +0300, Avi Kivity wrote:
> >> On 04/05/2012 04:57 PM, Gleb Natapov wrote:
> >> > > >
> >> > > May be it used NMI based profiling. We should ask oprofile developers.
> >> > > As I said I am almost sure my inability to run it on a host is probably
> >> > > PEBKAC, although I ran the same script exactly on the host and the
> >> > > guest (the script is from the first email of this thread)
> >> > >
> >> > After upgrading the kernel to latest git from whatever it was there the
> >> > same script works and counts CPU_CLK_UNHALT events.
> >> >
> >>
> >> This is even while it violates the Intel guidelines?
> >>
> > Yes, but who says the result is correct :) It seems that we handle
> > global ctrl msr wrong. That is counter can be enabled either in global
> > ctrl or in eventsel. Trying to confirm that.
> >
> if that becomes true then will global ctrl msr have any significance ?
When it is in use yes.

--
			Gleb.

Re: Questing regarding KVM Guest PMU

[Gleb Natapov]

On Fri, Apr 06, 2012 at 09:50:50AM +0300, Gleb Natapov wrote:
> On Fri, Apr 06, 2012 at 10:43:17AM +0530, shashank rachamalla wrote:
> > On Thu, Apr 5, 2012 at 8:11 PM, Gleb Natapov <gleb@redhat.com> wrote:
> > > On Thu, Apr 05, 2012 at 05:38:40PM +0300, Avi Kivity wrote:
> > >> On 04/05/2012 04:57 PM, Gleb Natapov wrote:
> > >> > > >
> > >> > > May be it used NMI based profiling. We should ask oprofile developers.
> > >> > > As I said I am almost sure my inability to run it on a host is probably
> > >> > > PEBKAC, although I ran the same script exactly on the host and the
> > >> > > guest (the script is from the first email of this thread)
> > >> > >
> > >> > After upgrading the kernel to latest git from whatever it was there the
> > >> > same script works and counts CPU_CLK_UNHALT events.
> > >> >
> > >>
> > >> This is even while it violates the Intel guidelines?
> > >>
> > > Yes, but who says the result is correct :) It seems that we handle
> > > global ctrl msr wrong. That is counter can be enabled either in global
> > > ctrl or in eventsel. Trying to confirm that.
> > >
> > if that becomes true then will global ctrl msr have any significance ?
> When it is in use yes.
> 
I was wrong. We do handle global ctrl msr correctly, I just ran my test
incorrectly. If I disable global ctrl on all cpus (for i in `seq 0 15`;
do wrmsr -p $i 0x38f 0; done) oprofile stops working.

--
			Gleb.

Re: Questing regarding KVM Guest PMU

[Gleb Natapov]

On Sun, Apr 08, 2012 at 06:27:03PM +0300, Gleb Natapov wrote:
> On Fri, Apr 06, 2012 at 09:50:50AM +0300, Gleb Natapov wrote:
> > On Fri, Apr 06, 2012 at 10:43:17AM +0530, shashank rachamalla wrote:
> > > On Thu, Apr 5, 2012 at 8:11 PM, Gleb Natapov <gleb@redhat.com> wrote:
> > > > On Thu, Apr 05, 2012 at 05:38:40PM +0300, Avi Kivity wrote:
> > > >> On 04/05/2012 04:57 PM, Gleb Natapov wrote:
> > > >> > > >
> > > >> > > May be it used NMI based profiling. We should ask oprofile developers.
> > > >> > > As I said I am almost sure my inability to run it on a host is probably
> > > >> > > PEBKAC, although I ran the same script exactly on the host and the
> > > >> > > guest (the script is from the first email of this thread)
> > > >> > >
> > > >> > After upgrading the kernel to latest git from whatever it was there the
> > > >> > same script works and counts CPU_CLK_UNHALT events.
> > > >> >
> > > >>
> > > >> This is even while it violates the Intel guidelines?
> > > >>
> > > > Yes, but who says the result is correct :) It seems that we handle
> > > > global ctrl msr wrong. That is counter can be enabled either in global
> > > > ctrl or in eventsel. Trying to confirm that.
> > > >
> > > if that becomes true then will global ctrl msr have any significance ?
> > When it is in use yes.
> > 
> I was wrong. We do handle global ctrl msr correctly, I just ran my test
> incorrectly. If I disable global ctrl on all cpus (for i in `seq 0 15`;
> do wrmsr -p $i 0x38f 0; done) oprofile stops working.
> 
After searching high and low I finally found the following in
"Performance Monitoring Unit Sharing Guide" white paper:

  Known starting state: Software requires a known starting
  state. After CPU reset, all counters and control registers are
  disabled and clear/reset to 0.  The only exception to this is the
  IA32_PERF_GLOBAL_CTRL control MSR, all programmable counter global
  enable bits are reset to 1.

Patch will follow.

--
			Gleb.

Re: Questing regarding KVM Guest PMU

[shashank rachamalla]

On Mon, Apr 9, 2012 at 2:56 PM, Gleb Natapov <gleb@redhat.com> wrote:
> On Sun, Apr 08, 2012 at 06:27:03PM +0300, Gleb Natapov wrote:
>> On Fri, Apr 06, 2012 at 09:50:50AM +0300, Gleb Natapov wrote:
>> > On Fri, Apr 06, 2012 at 10:43:17AM +0530, shashank rachamalla wrote:
>> > > On Thu, Apr 5, 2012 at 8:11 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> > > > On Thu, Apr 05, 2012 at 05:38:40PM +0300, Avi Kivity wrote:
>> > > >> On 04/05/2012 04:57 PM, Gleb Natapov wrote:
>> > > >> > > >
>> > > >> > > May be it used NMI based profiling. We should ask oprofile developers.
>> > > >> > > As I said I am almost sure my inability to run it on a host is probably
>> > > >> > > PEBKAC, although I ran the same script exactly on the host and the
>> > > >> > > guest (the script is from the first email of this thread)
>> > > >> > >
>> > > >> > After upgrading the kernel to latest git from whatever it was there the
>> > > >> > same script works and counts CPU_CLK_UNHALT events.
>> > > >> >
>> > > >>
>> > > >> This is even while it violates the Intel guidelines?
>> > > >>
>> > > > Yes, but who says the result is correct :) It seems that we handle
>> > > > global ctrl msr wrong. That is counter can be enabled either in global
>> > > > ctrl or in eventsel. Trying to confirm that.
>> > > >
>> > > if that becomes true then will global ctrl msr have any significance ?
>> > When it is in use yes.
>> >
>> I was wrong. We do handle global ctrl msr correctly, I just ran my test
>> incorrectly. If I disable global ctrl on all cpus (for i in `seq 0 15`;
>> do wrmsr -p $i 0x38f 0; done) oprofile stops working.
>>
> After searching high and low I finally found the following in
> "Performance Monitoring Unit Sharing Guide" white paper:
>
>  Known starting state: Software requires a known starting
>  state. After CPU reset, all counters and control registers are
>  disabled and clear/reset to 0.  The only exception to this is the
>  IA32_PERF_GLOBAL_CTRL control MSR, all programmable counter global
>  enable bits are reset to 1.
>
> Patch will follow.
>

Thanks for the patch. global ctrl msr is properly set inside guest and
i can run oprofile as well.

> --
>                        Gleb.