Revisiting the Xenoprof problem

Revisiting the Xenoprof problem

[Venkataraman, Meenakshi]

Hi all,

There was a discussion previously on oprofile not working in Dom U.
Oprofile prints the following message and exits.

-------- snip -----------

Using default event: CPU_CLK_UNHALTED:100000:0:1:1 Using 2.6+ OProfile
kernel interface.
Reading module info.
Failed to open profile device: Operation not permitted Couldn't start
oprofiled.
Check the log file "/var/lib/oprofile/oprofiled.log" and kernel syslog

-------- snip -----------

There was an email exchange a year ago on the same problem. (see here:
http://lists.xensource.com/archives/html/xen-devel/2006-05/msg01168.html
)


I've followed the sequence of steps mentioned in the thread, and yet I
get the same problem. I'm starting the daemon in Dom0 before I start
oprofile in the guest.

Thoughts and/or solutions, anyone?

Thanks in advance!
Meenakshi

RE: Revisiting the Xenoprof problem

[Santos, Jose Renato G]

Are you able to profile dom0 alone?
I suggest that you try this first.
Then try using passive domain profiing which enables guest profiling but does not require running oprofile in domU. It is much simpler and easier to use. Take a look at  http://www.xen.org/files/summit_3/xenoprof_tutorial.pdf for a xenoprof tutorial. It is a little old but it should be usefull (mostly the xen and oprofile versions have changed since then; otherwise everything still applies)

Renato

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of
> Venkataraman, Meenakshi
> Sent: Wednesday, November 14, 2007 9:32 AM
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] Revisiting the Xenoprof problem
>
> Hi all,
>
> There was a discussion previously on oprofile not working in Dom U.
> Oprofile prints the following message and exits.
>
> -------- snip -----------
>
> Using default event: CPU_CLK_UNHALTED:100000:0:1:1 Using 2.6+
> OProfile kernel interface.
> Reading module info.
> Failed to open profile device: Operation not permitted
> Couldn't start oprofiled.
> Check the log file "/var/lib/oprofile/oprofiled.log" and kernel syslog
>
> -------- snip -----------
>
> There was an email exchange a year ago on the same problem. (see here:
> http://lists.xensource.com/archives/html/xen-devel/2006-05/msg
01168.html
> )
>
>
> I've followed the sequence of steps mentioned in the thread,
> and yet I get the same problem. I'm starting the daemon in
> Dom0 before I start oprofile in the guest.
>
> Thoughts and/or solutions, anyone?
>
> Thanks in advance!
> Meenakshi
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

RE: Revisiting the Xenoprof problem

[Venkataraman, Meenakshi]

In answer to your questions...

> Are you able to profile dom0 alone?
>I suggest that you try this first.

I tried what you suggest above, I'm not able to do it without tweaking
the kernel. I found that I was unable to get any samples from opcontrol
for even just domain 0. The problem is that oprofile thinks that it is
on a P-III CPU, while it is actually running on a core 2.

If I fake the events and bitmasks to appear as if they're for an
i386/P-III but actually put the events and bitmasks for a core 2,
opcontrol works for the CPU_CLK_UNHALTED event. I haven't tested with
other events. Opreport is still not working.

The problem is traceable to the hypervisor, which is the entity that
determines the cpu_type during bootup. I'm trying to figure out why the
hypervisor is behaving strangely.

Has anyone come across such a problem?

>Then try using passive domain profiing which enables guest profiling
but >does not require running oprofile in domU. It is much simpler and
easier to >use. Take a look at
http://www.xen.org/files/summit_3/xenoprof_tutorial.pdf >for a xenoprof
tutorial. It is a little old but it should be usefull >(mostly the xen
and oprofile versions have changed since then; otherwise >everything
still applies)

I'll take a look at the tutorial you point to.

Thanks a ton,
Meenakshi


-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Santos, Jose
Renato G
Sent: Wednesday, November 14, 2007 11:32 PM
To: Venkataraman, Meenakshi; xen-devel@lists.xensource.com
Subject: RE: [Xen-devel] Revisiting the Xenoprof problem

Are you able to profile dom0 alone?
I suggest that you try this first.
Then try using passive domain profiing which enables guest profiling but
does not require running oprofile in domU. It is much simpler and easier
to use. Take a look at
http://www.xen.org/files/summit_3/xenoprof_tutorial.pdf for a xenoprof
tutorial. It is a little old but it should be usefull (mostly the xen
and oprofile versions have changed since then; otherwise everything
still applies)

Renato

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of
> Venkataraman, Meenakshi
> Sent: Wednesday, November 14, 2007 9:32 AM
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] Revisiting the Xenoprof problem
>
> Hi all,
>
> There was a discussion previously on oprofile not working in Dom U.
> Oprofile prints the following message and exits.
>
> -------- snip -----------
>
> Using default event: CPU_CLK_UNHALTED:100000:0:1:1 Using 2.6+
> OProfile kernel interface.
> Reading module info.
> Failed to open profile device: Operation not permitted
> Couldn't start oprofiled.
> Check the log file "/var/lib/oprofile/oprofiled.log" and kernel syslog
>
> -------- snip -----------
>
> There was an email exchange a year ago on the same problem. (see here:
> http://lists.xensource.com/archives/html/xen-devel/2006-05/msg
01168.html
> )
>
>
> I've followed the sequence of steps mentioned in the thread,
> and yet I get the same problem. I'm starting the daemon in
> Dom0 before I start oprofile in the guest.
>
> Thoughts and/or solutions, anyone?
>
> Thanks in advance!
> Meenakshi
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Revisiting the Xenoprof problem

[Santos, Jose Renato G]

What versions of Xen and oprofile are you running?
Any oprofile related message in the kernel log or in the output of "xm dmesg"
What is on /dev/oprofile/cpu_type after you run opcontrol?
Also take a look at /var/lib/oprofile/samples/oprofiled.log for any error message

Renato

> -----Original Message-----
> From: Venkataraman, Meenakshi
> [mailto:meenakshi.venkataraman@intel.com]
> Sent: Wednesday, November 14, 2007 11:47 PM
> To: Santos, Jose Renato G; xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] Revisiting the Xenoprof problem
>
> In answer to your questions...
>
> > Are you able to profile dom0 alone?
> >I suggest that you try this first.
>
> I tried what you suggest above, I'm not able to do it without
> tweaking the kernel. I found that I was unable to get any
> samples from opcontrol for even just domain 0. The problem is
> that oprofile thinks that it is on a P-III CPU, while it is
> actually running on a core 2.
>
> If I fake the events and bitmasks to appear as if they're for
> an i386/P-III but actually put the events and bitmasks for a
> core 2, opcontrol works for the CPU_CLK_UNHALTED event. I
> haven't tested with other events. Opreport is still not working.
>
> The problem is traceable to the hypervisor, which is the
> entity that determines the cpu_type during bootup. I'm trying
> to figure out why the hypervisor is behaving strangely.
>
> Has anyone come across such a problem?
>
> >Then try using passive domain profiing which enables guest profiling
> but >does not require running oprofile in domU. It is much
> simpler and easier to >use. Take a look at
> http://www.xen.org/files/summit_3/xenoprof_tutorial.pdf >for
> a xenoprof tutorial. It is a little old but it should be
> usefull >(mostly the xen and oprofile versions have changed
> since then; otherwise >everything still applies)
>
> I'll take a look at the tutorial you point to.
>
> Thanks a ton,
> Meenakshi
>
>
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of
> Santos, Jose Renato G
> Sent: Wednesday, November 14, 2007 11:32 PM
> To: Venkataraman, Meenakshi; xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] Revisiting the Xenoprof problem
>
> Are you able to profile dom0 alone?
> I suggest that you try this first.
> Then try using passive domain profiing which enables guest
> profiling but does not require running oprofile in domU. It
> is much simpler and easier to use. Take a look at
> http://www.xen.org/files/summit_3/xenoprof_tutorial.pdf for a
> xenoprof tutorial. It is a little old but it should be
> usefull (mostly the xen and oprofile versions have changed
> since then; otherwise everything still applies)
>
> Renato
>
> > -----Original Message-----
> > From: xen-devel-bounces@lists.xensource.com
> > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of
> > Venkataraman, Meenakshi
> > Sent: Wednesday, November 14, 2007 9:32 AM
> > To: xen-devel@lists.xensource.com
> > Subject: [Xen-devel] Revisiting the Xenoprof problem
> >
> > Hi all,
> >
> > There was a discussion previously on oprofile not working in Dom U.
> > Oprofile prints the following message and exits.
> >
> > -------- snip -----------
> >
> > Using default event: CPU_CLK_UNHALTED:100000:0:1:1 Using
> 2.6+ OProfile
> > kernel interface.
> > Reading module info.
> > Failed to open profile device: Operation not permitted
> Couldn't start
> > oprofiled.
> > Check the log file "/var/lib/oprofile/oprofiled.log" and
> kernel syslog
> >
> > -------- snip -----------
> >
> > There was an email exchange a year ago on the same problem.
> (see here:
> > http://lists.xensource.com/archives/html/xen-devel/2006-05/msg
> 01168.html
> > )
> >
> >
> > I've followed the sequence of steps mentioned in the
> thread, and yet I
> > get the same problem. I'm starting the daemon in Dom0
> before I start
> > oprofile in the guest.
> >
> > Thoughts and/or solutions, anyone?
> >
> > Thanks in advance!
> > Meenakshi
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

RE: Revisiting the Xenoprof problem

[Venkataraman, Meenakshi]

>What versions of Xen and oprofile are you running?
I'm running xen 3.1.0 and oprofile 0.9.3 (patched for xen).

>Any oprofile related message in the kernel log or in the output of "xm
>dmesg"
No...none. Although the cpu type is correctly recognized as a xeon in xm
dmesg.

>What is on /dev/oprofile/cpu_type after you run opcontrol?
"i386/piii"

>Also take a look at /var/lib/oprofile/samples/oprofiled.log for any
error message
There are a lot of CPU_SWITCH messages in during some runs in the
oprofiled.log. I saw another post with the same problem on an embedded
processor, but couldn't find any follow up to it.

Meenakshi

RE: Revisiting the Xenoprof problem

[Santos, Jose Renato G]

> -----Original Message-----
> From: Venkataraman, Meenakshi
> [mailto:meenakshi.venkataraman@intel.com]
> Sent: Thursday, November 15, 2007 1:32 AM
> To: Santos, Jose Renato G; xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] Revisiting the Xenoprof problem
>
>
> >What versions of Xen and oprofile are you running?
> I'm running xen 3.1.0 and oprofile 0.9.3 (patched for xen).
>
> >Any oprofile related message in the kernel log or in the
> output of "xm
> >dmesg"
> No...none. Although the cpu type is correctly recognized as a
> xeon in xm dmesg.
>
> >What is on /dev/oprofile/cpu_type after you run opcontrol?
> "i386/piii"

  I think the problem is probably that Xen 3.1.0 does not have xenoprof support for core2 machines. Support for core2 was added in c/s 12478 (nov 17,2006)(not sure if 3.1.0 included that patch, but it looks like it does not). You should try using a more recent version of Xen or apply that patch.

Renato

Re: Revisiting the Xenoprof problem

[William Cohen]

Venkataraman, Meenakshi wrote:
>> What versions of Xen and oprofile are you running?
> I'm running xen 3.1.0 and oprofile 0.9.3 (patched for xen).
> 
>> Any oprofile related message in the kernel log or in the output of "xm
>> dmesg"
> No...none. Although the cpu type is correctly recognized as a xeon in xm
> dmesg.
> 
>> What is on /dev/oprofile/cpu_type after you run opcontrol?
> "i386/piii"
> 
>> Also take a look at /var/lib/oprofile/samples/oprofiled.log for any
> error message
> There are a lot of CPU_SWITCH messages in during some runs in the
> oprofiled.log. I saw another post with the same problem on an embedded
> processor, but couldn't find any follow up to it.
> 
> Meenakshi

Is xen 3.1.0 okay to run with oprofile 0.9.3. There were some conflicts in the 
defines between xenoprof and the defines added for the cell processor in 
oprofile 0.9.3.

https://bugzilla.redhat.com/show_bug.cgi?id=250852

-Will

RE: Revisiting the Xenoprof problem

[Santos, Jose Renato G]

Running oprofile 0.9.3 still requires a patch to Xen http://xenoprof.sourceforge.net/xenoprof-oprofile-0.9.3-compat.patch
The plan was to have this patch into mainline Xen once the oprofile patches to support Xen passive domains were accepted into Oprofile main tree
It seems that this is not going to happen anytime soon as nobody is working on it.

I briefly discussed this with John Levon last week.
We agreed to reserve a domain_switch code in Oprofile and thus avoid future incompatibilities while we don't have the proper passive domain support included.
I will work on a patch for that.
After that we should think on a plan to enable oprofile 0.9.3 in Xen without breaking instalations with oprofile 0.9.2 and 0.9.1.

Regards

Renato

-----Original Message-----
From: William Cohen [mailto:wcohen@redhat.com]
Sent: Tuesday, November 20, 2007 12:49 PM
To: Santos, Jose Renato G
Cc: Venkataraman, Meenakshi; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] Revisiting the Xenoprof problem

Venkataraman, Meenakshi wrote:
>> What versions of Xen and oprofile are you running?
> I'm running xen 3.1.0 and oprofile 0.9.3 (patched for xen).
>
>> Any oprofile related message in the kernel log or in the output of
>> "xm dmesg"
> No...none. Although the cpu type is correctly recognized as a xeon in
> xm dmesg.
>
>> What is on /dev/oprofile/cpu_type after you run opcontrol?
> "i386/piii"
>
>> Also take a look at /var/lib/oprofile/samples/oprofiled.log for any
> error message
> There are a lot of CPU_SWITCH messages in during some runs in the
> oprofiled.log. I saw another post with the same problem on an embedded
> processor, but couldn't find any follow up to it.
>
> Meenakshi

Is xen 3.1.0 okay to run with oprofile 0.9.3. There were some conflicts in the defines between xenoprof and the defines added for the cell processor in oprofile 0.9.3.

https://bugzilla.redhat.com/show_bug.cgi?id=250852

-Will

Re: Revisiting the Xenoprof problem

[William Cohen]

Santos, Jose Renato G wrote:
> Running oprofile 0.9.3 still requires a patch to Xen http://xenoprof.sourceforge.net/xenoprof-oprofile-0.9.3-compat.patch
> The plan was to have this patch into mainline Xen once the oprofile patches to support Xen passive domains were accepted into Oprofile main tree
> It seems that this is not going to happen anytime soon as nobody is working on it.
> 
> I briefly discussed this with John Levon last week.
> We agreed to reserve a domain_switch code in Oprofile and thus avoid future incompatibilities while we don't have the proper passive domain support included.
> I will work on a patch for that.
> After that we should think on a plan to enable oprofile 0.9.3 in Xen without breaking instalations with oprofile 0.9.2 and 0.9.1.
> 
> Regards
> 
> Renato

So the xen passive domain support needs to be accepted into oprofile before the 
matching patches get accepted into Xen? Is there going to be a placeholder or 
some other entry in oprofile cvs, so people know that a define is reserved for 
Xenoprof in  daemon/opd_interface.h?

Any thoughts on how to determine whether the old or new xenoprof interface is 
being used. The oprofile kernel-user space interface doesn't have versioning 
information.

-Will

RE: Revisiting the Xenoprof problem

[Santos, Jose Renato G]

>
> So the xen passive domain support needs to be accepted into
> oprofile before the matching patches get accepted into Xen?

  That was the original plan. However since no one is working on modifying the patches to attend John Levon requests I think this is not going to happen anytime soon. So I am suggesting to change the original plan and merge the Xen fix to support oprofile 0.9.3. now. We should reserve the cpu buffer escape codes for Xen in mainline Oprofile (to avoid the same problem in future Oprofile versions) and get the changes to Xen merged.

The main question is how to change Xen to support newer versions of Oprofile without breaking instalations using older versions.

Here is a suggestion.
We could create a new xenoprof operation (in xenoprof hypercall) to tell Xen to use the new version of the CPU escape codes (used to represent domain switches). We would then modify oprofile patches for passive domain support to call this hypercall for oprofile versions 0.9.3 and later. By default Xen would use the older version of the escape code if the new xenoprof hypercall is not invoked, enabling instalations with oprofile 0.9.2 and older to continue working, since they will not invoke the new hypercall.

Does this seems reasonable?
Keir, would this be acceptable?
If we reach agreement I can create the Xen and Oprofile patches

Thanks

Renato

> Is there going to be a placeholder or some other entry in
> oprofile cvs, so people know that a define is reserved for
> Xenoprof in  daemon/opd_interface.h?
>
> Any thoughts on how to determine whether the old or new
> xenoprof interface is being used. The oprofile kernel-user
> space interface doesn't have versioning information.
>
> -Will
>

Re: Revisiting the Xenoprof problem

[Keir Fraser]

On 28/11/07 21:44, "Santos, Jose Renato G" <joserenato.santos@hp.com> wrote:

> Does this seems reasonable?
> Keir, would this be acceptable?
> If we reach agreement I can create the Xen and Oprofile patches

Fine by me.

 -- Keir