[XENOPROFILE] Xenoprof passive profiling broken under cs 13062

[XENOPROFILE] Xenoprof passive profiling broken under cs 13062

[Ray Bryant]

Renato,

If one creates an HVM domain (say domain 1) and then does

echo 1 > /dev/oprofile/passive_domains

one then gets the following on the Xen serial console:

(XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode enabledisn't 
supported yet
(XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode enabledisn't 
supported yet
(XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode enabledisn't 
supported yet
(XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode enabledisn't 
supported yet
(XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode enabledisn't 
supported yet
(XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode enabledisn't 
supported yet
(XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode enabledisn't 
supported yet
(XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode enabledisn't 
supported yet
(XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode enabledisn't 
supported yet
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at drivers/xen/xenoprof/xenoprofile.c:429
invalid opcode: 0000 [1] SMP
CPU 1
Modules linked in: xt_physdev iptable_filter ip_tables x_tables tun bridge 
usbmouse usbhid ide_cd cdrom floppy snd_atiixd
Pid: 5313, comm: bash Not tainted 2.6.16.33-xen #2
RIP: e030:[<ffffffff802806b6>] <ffffffff802806b6>{xenoprof_set_passive+190}
RSP: e02b:ffff880014ccfe88  EFLAGS: 00010292
RAX: 00000000f000ff53 RBX: 0000000000000000 RCX: ffffc20000940000
RDX: ffff88000138a160 RSI: 0000000000000000 RDI: ffffffff80353b90
RBP: 0000000000000000 R08: ffff880014ccfe08 R09: ffffffffffffffff
R10: 0000000000007ff0 R11: ffff880017c6fa80 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
FS:  00002b05deb40f60(0000) GS:ffffffff803c0080(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000
Process bash (pid: 5313, threadinfo ffff880014cce000, task ffff88001fdeb820)
Stack: ffffffff8048c200 ffffffff803824c0 0000000000000001 ffffffff8048c200
       0000000000000002 ffff880014ccff50 0000000000000001 ffffffff8027de5c
       ffff8800189b1002 0000000000000001
Call Trace: <ffffffff8027de5c>{oprofile_set_passive+63}
       <ffffffff8027f9a7>{pdomain_write+291} <ffffffff8017a023>{vfs_write+212}
       <ffffffff8017a180>{sys_write+69} <ffffffff8010b382>{system_call+134}
       <ffffffff8010b2fc>{system_call+0}

Code: 0f 0b 68 22 48 32 80 c2 ad 01 89 c2 4c 89 e0 41 ff c5 48 c1
RIP <ffffffff802806b6>{xenoprof_set_passive+190} RSP <ffff880014ccfe88>

This for a 64 bit hypervisor and a 32 bit guest (although the above is all 
dom0 code related to mapping the xenoprof buffer, so I doubt the bittedness 
of the guest makes much difference); both AMD V and VT have the same problem.

I've tried this on change sets 12548, 13008 and 13062 (latest as of this
afternoon).   So this particular problem has been around for a bit.

Any ideas as to what went wrong here?

Thanks,
-- 
Ray Bryant
AMD Performance Labs                   Austin, Tx
512-602-0038 (o)                 512-507-7807 (c)

RE: [XENOPROFILE] Xenoprof passive profiling broken undercs 13062

[Santos, Jose Renato G]

Ray,

Try this patch

Please let me know if it fixes the problem

BTW, I have now xen running on a AMD machine with SVM support here.
I did some tests with 64bit PV guests but have not tried hvm guests yet
What is the status of profiling HVM guests? Are you still having
problems?
If you are still having problems I can try to reproduce the problem
here...

Regards 

Renato
===========================================================
Signed-off-by: Jose Renato Santos <jsantos@hpl.hp.com>

diff -r 37141c3a3d39 xen/common/xenoprof.c
--- a/xen/common/xenoprof.c	Mon Dec 11 15:06:53 2006 +0000
+++ b/xen/common/xenoprof.c	Fri Dec 15 17:58:03 2006 -0800
@@ -384,7 +384,7 @@ static int add_passive_list(XEN_GUEST_HA
     d->xenoprof->domain_type = XENOPROF_DOMAIN_PASSIVE;
     passive.nbuf = d->xenoprof->nbuf;
     passive.bufsize = d->xenoprof->bufsize;
-    if ( !shadow_mode_translate(d) )
+    if ( !shadow_mode_translate(current->domain) )
         passive.buf_gmaddr = __pa(d->xenoprof->rawbuf);
     else
         xenoprof_shared_gmfn_with_guest(




> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Ray Bryant
> Sent: Friday, December 15, 2006 3:21 PM
> To: Santos@mpdtxmail.amd.com; Santos, Jose Renato G
> Cc: Isaku Yamahata; xen-devel@lists.xensource.com
> Subject: [Xen-devel] [XENOPROFILE] Xenoprof passive profiling 
> broken undercs 13062
> 
> Renato,
> 
> If one creates an HVM domain (say domain 1) and then does
> 
> echo 1 > /dev/oprofile/passive_domains
> 
> one then gets the following on the Xen serial console:
> 
> (XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode 
> enabledisn't supported yet
> (XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode 
> enabledisn't supported yet
> (XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode 
> enabledisn't supported yet
> (XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode 
> enabledisn't supported yet
> (XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode 
> enabledisn't supported yet
> (XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode 
> enabledisn't supported yet
> (XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode 
> enabledisn't supported yet
> (XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode 
> enabledisn't supported yet
> (XEN) xenoprof.c:143:d0 xenoprof/x86 with autotranslated mode 
> enabledisn't supported yet
> ----------- [cut here ] --------- [please bite here ] 
> --------- Kernel BUG at drivers/xen/xenoprof/xenoprofile.c:429
> invalid opcode: 0000 [1] SMP
> CPU 1
> Modules linked in: xt_physdev iptable_filter ip_tables 
> x_tables tun bridge usbmouse usbhid ide_cd cdrom floppy snd_atiixd
> Pid: 5313, comm: bash Not tainted 2.6.16.33-xen #2
> RIP: e030:[<ffffffff802806b6>] 
> <ffffffff802806b6>{xenoprof_set_passive+190}
> RSP: e02b:ffff880014ccfe88  EFLAGS: 00010292
> RAX: 00000000f000ff53 RBX: 0000000000000000 RCX: ffffc20000940000
> RDX: ffff88000138a160 RSI: 0000000000000000 RDI: ffffffff80353b90
> RBP: 0000000000000000 R08: ffff880014ccfe08 R09: ffffffffffffffff
> R10: 0000000000007ff0 R11: ffff880017c6fa80 R12: 0000000000000000
> R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
> FS:  00002b05deb40f60(0000) GS:ffffffff803c0080(0000) 
> knlGS:0000000000000000
> CS:  e033 DS: 0000 ES: 0000
> Process bash (pid: 5313, threadinfo ffff880014cce000, task 
> ffff88001fdeb820)
> Stack: ffffffff8048c200 ffffffff803824c0 0000000000000001 
> ffffffff8048c200
>        0000000000000002 ffff880014ccff50 0000000000000001 
> ffffffff8027de5c
>        ffff8800189b1002 0000000000000001 Call Trace: 
> <ffffffff8027de5c>{oprofile_set_passive+63}
>        <ffffffff8027f9a7>{pdomain_write+291} 
> <ffffffff8017a023>{vfs_write+212}
>        <ffffffff8017a180>{sys_write+69} 
> <ffffffff8010b382>{system_call+134}
>        <ffffffff8010b2fc>{system_call+0}
> 
> Code: 0f 0b 68 22 48 32 80 c2 ad 01 89 c2 4c 89 e0 41 ff c5 
> 48 c1 RIP <ffffffff802806b6>{xenoprof_set_passive+190} RSP 
> <ffff880014ccfe88>
> 
> This for a 64 bit hypervisor and a 32 bit guest (although the 
> above is all dom0 code related to mapping the xenoprof 
> buffer, so I doubt the bittedness of the guest makes much 
> difference); both AMD V and VT have the same problem.
> 
> I've tried this on change sets 12548, 13008 and 13062 (latest 
> as of this
> afternoon).   So this particular problem has been around for a bit.
> 
> Any ideas as to what went wrong here?
> 
> Thanks,
> --
> Ray Bryant
> AMD Performance Labs                   Austin, Tx
> 512-602-0038 (o)                 512-507-7807 (c)
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: [XENOPROFILE] Xenoprof passive profiling broken undercs 13062

[Ray Bryant]

On Saturday 16 December 2006 03:24, Santos, Jose Renato G wrote:
> Ray,
>
> Try this patch
>
> Please let me know if it fixes the problem
>
> BTW, I have now xen running on a AMD machine with SVM support here.
> I did some tests with 64bit PV guests but have not tried hvm guests yet
> What is the status of profiling HVM guests? Are you still having
> problems?
> If you are still having problems I can try to reproduce the problem
> here...
>
> Regards
>
> Renato

Yes, that patch appears to fix the problem.   I'm surprised, though, as the 
test appears to be backwards from what I thought "shadow_mode_translate()" 
returned.

WRT profiling HVM guests:  with oprofile-0.9.2 plus your 
oprofile-0.9.2-xen.patch, it appears to work OK on an AMD RevG.  I haven't 
tried it on other processors yet.

-- 
Ray Bryant
AMD Performance Labs                   Austin, Tx
512-602-0038 (o)                 512-507-7807 (c)

RE: [XENOPROFILE] Xenoprof passive profiling broken undercs 13062

[Santos, Jose Renato G]

 

> -----Original Message-----
> From: Ray Bryant [mailto:raybry@mpdtxmail.amd.com] 
> Sent: Monday, December 18, 2006 10:07 AM
> To: Santos, Jose Renato G
> Cc: Isaku Yamahata; xen-devel@lists.xensource.com; Woller, Thomas
> Subject: Re: [Xen-devel] [XENOPROFILE] Xenoprof passive 
> profiling broken undercs 13062
> 
> On Saturday 16 December 2006 03:24, Santos, Jose Renato G wrote:
> > Ray,
> >
> > Try this patch
> >
> > Please let me know if it fixes the problem
> >
> > BTW, I have now xen running on a AMD machine with SVM support here.
> > I did some tests with 64bit PV guests but have not tried hvm guests 
> > yet What is the status of profiling HVM guests? Are you 
> still having 
> > problems?
> > If you are still having problems I can try to reproduce the problem 
> > here...
> >
> > Regards
> >
> > Renato
> 
> Yes, that patch appears to fix the problem.   I'm surprised, 
> though, as the 
> test appears to be backwards from what I thought 
> "shadow_mode_translate()" 

  The problem is that we map xenoprof buffer for the passive hvm guest
  into dom0 and not into the guest since dom0 collects the profile
samples
  for the passive guest. The code was checking the mode for the guest
which is
  irrelevant; it should check the mode for dom0 (or coordinator) which
is
  mapping the buffer.
  With the provided patch, "shadow_mode_translate()" should now be
false,
  as expected, since it is now checking dom0 mode.
  
  
> returned.
> 
> WRT profiling HVM guests:  with oprofile-0.9.2 plus your 
> oprofile-0.9.2-xen.patch, it appears to work OK on an AMD 
> RevG.  I haven't tried it on other processors yet.
> 
  Good. Nice to hear that

  Regards

  Renato

> --
> Ray Bryant
> AMD Performance Labs                   Austin, Tx
> 512-602-0038 (o)                 512-507-7807 (c)
> 
> 
> 

Re: [XENOPROFILE] and "xm vcpu-pin" doesn't work as expected

[Ray Bryant]

Renato,

I've been trying to profile an HVM guest and its corresponding Xen usage so I 
used the "xm vcpu-pin" command to pin the HVM guest to CPU 1 and dom0 to CPU 
0 (this is a single socket, dual core setup, but that probably doesn't matter 
here).

I then used the opsetup --separate=cpu option to split out the profile samples 
by CPU, fully expecting to see no "domain1-xen" samples on CPU 0.   However, 
what I actually see is 90% (more or less) of the samples on CPU 0 and the 
remainder on CPU 1.

This problem occurs on both AMD V and VT.

So, I guess the question is:

(1)   Is xenoprof reporting the correct CPU, or
(2)   Does "xm vcpu-pin" really do what it claims it does?

Any ideas on this?   For the moment, I'm ignoring the issue, but I would like 
it better if I could isolate the VCPU's to a dedicated CPU in order to reduce 
measurement overhead in the guest.   (e. g. by forcing the trace daemon to 
run on CPU 0 and the measured guest to run on CPU 1.)

BTW, this is on changeset 13062.

-- 
Ray Bryant
AMD Performance Labs                   Austin, Tx
512-602-0038 (o)                 512-507-7807 (c)

RE: [XENOPROFILE] and "xm vcpu-pin" doesn't work as expected

[Santos, Jose Renato G]

Ray,

That is normal behavior. In Xen when --separate=cpu is used, samples are
separated by "virtual" cpu. That means, CPU0 samples will have all
samples for vcpu=0 for dom0 and vcpu=0 for the guest. This seems to be
more usefull to users profiling user and kernel code which may not be
aware of Xen scheduling decisions which may move vcpus around. It would
be nice to add the capability of separating samples by physical cpu in
Xen, but I think no one is working on this. Patches would be greatly
appreciated, though. :)

Regards

Renato
 

> -----Original Message-----
> From: Ray Bryant [mailto:raybry@mpdtxmail.amd.com] 
> Sent: Friday, January 19, 2007 1:07 PM
> To: Santos, Jose Renato G
> Cc: xen-devel@lists.xensource.com; Woller, Thomas; Thomas Friebel
> Subject: Re: [XENOPROFILE] and "xm vcpu-pin" doesn't work as expected
> 
> Renato,
> 
> I've been trying to profile an HVM guest and its 
> corresponding Xen usage so I used the "xm vcpu-pin" command 
> to pin the HVM guest to CPU 1 and dom0 to CPU 0 (this is a 
> single socket, dual core setup, but that probably doesn't 
> matter here).
> 
> I then used the opsetup --separate=cpu option to split out 
> the profile samples 
> by CPU, fully expecting to see no "domain1-xen" samples on 
> CPU 0.   However, 
> what I actually see is 90% (more or less) of the samples on 
> CPU 0 and the remainder on CPU 1.
> 
> This problem occurs on both AMD V and VT.
> 
> So, I guess the question is:
> 
> (1)   Is xenoprof reporting the correct CPU, or
> (2)   Does "xm vcpu-pin" really do what it claims it does?
> 
> Any ideas on this?   For the moment, I'm ignoring the issue, 
> but I would like 
> it better if I could isolate the VCPU's to a dedicated CPU in 
> order to reduce 
> measurement overhead in the guest.   (e. g. by forcing the 
> trace daemon to 
> run on CPU 0 and the measured guest to run on CPU 1.)
> 
> BTW, this is on changeset 13062.
> 
> --
> Ray Bryant
> AMD Performance Labs                   Austin, Tx
> 512-602-0038 (o)                 512-507-7807 (c)
> 
> 
>