event TRC_MEM_PAGE_GRANT_TRANSFER

event TRC_MEM_PAGE_GRANT_TRANSFER

[Cesar Fernandes]

[-- Attachment #1.1: Type: text/plain, Size: 298 bytes --]

Greetings,

I'm using xen 3.0.4_1. I need to log  the event TRC_MEM_PAGE_GRANT_TRANSFER,
which, I believe, is the number of page exchanges between domU and dom0. It
seems that this event is not being logged in xen buffer, therefore,
xenmon/xentrace is not working either.

Thanks in advance,
Cesar

[-- Attachment #1.2: Type: text/html, Size: 331 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: event TRC_MEM_PAGE_GRANT_TRANSFER

[Rob Gardner]

Cesar Fernandes wrote:
> Greetings,
> 
> I'm using xen 3.0.4_1. I need to log  the event 
> TRC_MEM_PAGE_GRANT_TRANSFER, which, I believe, is the number of page 
> exchanges between domU and dom0. It seems that this event is not being 
> logged in xen buffer, therefore, xenmon/xentrace is not working either.


Yeah I noticed that I/O count has not been working lately in xenmon but 
haven't had a chance yet to see why. The TRC_MEM_PAGE_GRANT_TRANSFER 
event is being "logged" but the reason it doesn't show up in the trace 
buffer may be because that event is never actually happening. It seems 
that there has been some change in the mechanism used to move I/O pages 
between domains.

So some research needs to be done to figure out exactly what code should 
be instrumented with some trace call in order to provide xenmon with 
data on the number of I/O's or I/O pages. If anyone can help, it would 
be appreciated.

Rob Gardner

Re: event TRC_MEM_PAGE_GRANT_TRANSFER

[Keir Fraser]

On 23/1/07 17:51, "Rob Gardner" <rob.gardner@hp.com> wrote:

> Yeah I noticed that I/O count has not been working lately in xenmon but
> haven't had a chance yet to see why. The TRC_MEM_PAGE_GRANT_TRANSFER
> event is being "logged" but the reason it doesn't show up in the trace
> buffer may be because that event is never actually happening. It seems
> that there has been some change in the mechanism used to move I/O pages
> between domains.

Grant transfers are no longer used to move network data from netback to
netfront (except for backward compatibility with old netfront drivers).

 -- Keir

Re: event TRC_MEM_PAGE_GRANT_TRANSFER

[Rob Gardner]

Keir Fraser wrote:

> 
> Grant transfers are no longer used to move network data from netback to
> netfront (except for backward compatibility with old netfront drivers).


Yeah, got that. ;)  Could you explain what mechanisms are currently used 
to move data for net I/O and disk I/O between domains, and in 
particular, can you suggest where in the code I could put trace calls to 
be able to count I/O's? Thanks.

Rob

Re: event TRC_MEM_PAGE_GRANT_TRANSFER

[Keir Fraser]

On 23/1/07 19:31, "Rob Gardner" <rob.gardner@hp.com> wrote:

>> 
>> Grant transfers are no longer used to move network data from netback to
>> netfront (except for backward compatibility with old netfront drivers).
> 
> 
> Yeah, got that. ;)  Could you explain what mechanisms are currently used
> to move data for net I/O and disk I/O between domains, and in
> particular, can you suggest where in the code I could put trace calls to
> be able to count I/O's? Thanks.

Everything is done via grant-map/unmap commands as it always was, except
network receive (netback->netfront) which is done via grant-copy commands
(one per contiguous fragment of network packet).

 -- Keir

Re: event TRC_MEM_PAGE_GRANT_TRANSFER

[Ashish Gupta]

[-- Attachment #1.1: Type: text/plain, Size: 1415 bytes --]

I am using xentrace to understand performance bottlenecks for an application
inside domU.
My question is how can I distinguish between network IO events and disk IO
events using xentrace ?
A second related question is can I figure out disk queue waiting times and
serving times (and similarly for the network) to figure out any bottlenecks
or any external stress on these resources that may be causing the guest
machine to be slowing down ?

The post below seems to be the closest to what I want...but I am not sure
exactly how to extract the above information. Is it already implemented in
Xentrace for the current stable Xen release ?

I greatly appreciate any advise/insight from fellow members here.

cheers,
Ashish

On 23/1/07 19:31, "Rob Gardner" <rob.gardner@xxxxxx> wrote:

>*> *
>*> Grant transfers are no longer used to move network data from netback to*
>*> netfront (except for backward compatibility with old netfront drivers).*
>* *
>* *
>* Yeah, got that. ;)  Could you explain what mechanisms are currently used*
>* to move data for net I/O and disk I/O between domains, and in*
>* particular, can you suggest where in the code I could put trace calls to*
>* be able to count I/O's? Thanks.*

Everything is done via grant-map/unmap commands as it always was, except
network receive (netback->netfront) which is done via grant-copy commands
(one per contiguous fragment of network packet).

 -- Keir

[-- Attachment #1.2: Type: text/html, Size: 1647 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: event TRC_MEM_PAGE_GRANT_TRANSFER

[Mark Williamson]

> I am using xentrace to understand performance bottlenecks for an
> application inside domU.
> My question is how can I distinguish between network IO events and disk IO
> events using xentrace ?

I don't know if there are trace events currently generated for grant copy 
operations - if not, you could add them and use these to judge the amount of 
incoming network traffic.

Outgoing network traffic and disk IO are harder to distinguish since they both 
just use temporary sharing grants.  It might be easier to use some sort of IO 
monitoring tools within dom0 and the domU in question, similarly to as you 
would on a native Linux system.

> A second related question is can I figure out disk queue waiting times and
> serving times (and similarly for the network) to figure out any bottlenecks
> or any external stress on these resources that may be causing the guest
> machine to be slowing down ?

Again, it's worth taking a look at native Linux tools.  XenMon may provide 
some feedback but I imagine you might have already tried this?

For more complete profiling, Xenoprof allows you to run oprofile against 
multiple domains (and Xen itself) at once).

> I greatly appreciate any advise/insight from fellow members here.

Sorry not to be more specific.

Cheers,
Mark

> cheers,
> Ashish
>
> On 23/1/07 19:31, "Rob Gardner" <rob.gardner@xxxxxx> wrote:
> >*> *
> >*> Grant transfers are no longer used to move network data from netback
> > to* *> netfront (except for backward compatibility with old netfront
> > drivers).* * *
> >* *
> >* Yeah, got that. ;)  Could you explain what mechanisms are currently
> > used* * to move data for net I/O and disk I/O between domains, and in*
> >* particular, can you suggest where in the code I could put trace calls
> > to* * be able to count I/O's? Thanks.*
>
> Everything is done via grant-map/unmap commands as it always was, except
> network receive (netback->netfront) which is done via grant-copy commands
> (one per contiguous fragment of network packet).
>
>  -- Keir



-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: event TRC_MEM_PAGE_GRANT_TRANSFER

[Ashish Gupta]

[-- Attachment #1.1: Type: text/plain, Size: 3153 bytes --]

Thanks Mark !  Actually the goal here is to remain black box....I am working
on generic online techniques for a project, not really for a specific
application.

Can't I also add tracing in the code that actually does the packet IO and
disk IO in xen ? I need to figure out stats for both incoming and outgoing
traffic. I have never really modified or played with the xen code so kind of
warming up to what's really needed instead of getting lost in the code.

As for native Linux tools, if I run them in dom0 , will that be able to tell
me all about disk and  network IO for the guest ? I am not allowed to run
any thing inside domU to meet the black box requirement.

cheers,
Ashish

On 4/24/07, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:
>
> > I am using xentrace to understand performance bottlenecks for an
> > application inside domU.
> > My question is how can I distinguish between network IO events and disk
> IO
> > events using xentrace ?
>
> I don't know if there are trace events currently generated for grant copy
> operations - if not, you could add them and use these to judge the amount
> of
> incoming network traffic.
>
> Outgoing network traffic and disk IO are harder to distinguish since they
> both
> just use temporary sharing grants.  It might be easier to use some sort of
> IO
> monitoring tools within dom0 and the domU in question, similarly to as you
> would on a native Linux system.
>
> > A second related question is can I figure out disk queue waiting times
> and
> > serving times (and similarly for the network) to figure out any
> bottlenecks
> > or any external stress on these resources that may be causing the guest
> > machine to be slowing down ?
>
> Again, it's worth taking a look at native Linux tools.  XenMon may provide
> some feedback but I imagine you might have already tried this?
>
> For more complete profiling, Xenoprof allows you to run oprofile against
> multiple domains (and Xen itself) at once).
>
> > I greatly appreciate any advise/insight from fellow members here.
>
> Sorry not to be more specific.
>
> Cheers,
> Mark
>
> > cheers,
> > Ashish
> >
> > On 23/1/07 19:31, "Rob Gardner" <rob.gardner@xxxxxx> wrote:
> > >*> *
> > >*> Grant transfers are no longer used to move network data from netback
> > > to* *> netfront (except for backward compatibility with old netfront
> > > drivers).* * *
> > >* *
> > >* Yeah, got that. ;)  Could you explain what mechanisms are currently
> > > used* * to move data for net I/O and disk I/O between domains, and in*
> > >* particular, can you suggest where in the code I could put trace calls
> > > to* * be able to count I/O's? Thanks.*
> >
> > Everything is done via grant-map/unmap commands as it always was, except
> > network receive (netback->netfront) which is done via grant-copy
> commands
> > (one per contiguous fragment of network packet).
> >
> >  -- Keir
>
>
>
> --
> Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
> Mark: To answer a question with a question: What use is a skateboard?
> Dave: Skateboards have wheels.
> Mark: My wheel has a wheel!
>



-- 
http://www.cs.northwestern.edu/~agupta

[-- Attachment #1.2: Type: text/html, Size: 3820 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: event TRC_MEM_PAGE_GRANT_TRANSFER

[Mark Williamson]

> Thanks Mark !  Actually the goal here is to remain black box....I am
> working on generic online techniques for a project, not really for a
> specific application.

OK.  Working in a blackbox fashion makes virtualisation extremely useful.

> Can't I also add tracing in the code that actually does the packet IO and
> disk IO in xen ? I need to figure out stats for both incoming and outgoing
> traffic. I have never really modified or played with the xen code so kind
> of warming up to what's really needed instead of getting lost in the code.

This code is in dom0, not in Xen itself so you won't be able to do it with the 
tracebuffer - you'd need to set up a dom0-based solution.

> As for native Linux tools, if I run them in dom0 , will that be able to
> tell me all about disk and  network IO for the guest ? I am not allowed to
> run any thing inside domU to meet the black box requirement.

Well, if you measure the amount of traffic the vifs have seen that should tell 
you how much network IO had occurred.  Block-wise, I'm not sure but I think 
each domain gets a block IO thread for purposes of being able to use the IO 
schedulers to regulate each domain...  If I'm right and that patch got 
applied, you may be able to view IO stats for these kernel threads.  
Otherwise, you could maybe find and apply the patch to dom0 yourself.

FWIW I think Xentop (and therefore libxenstat - which you could link against 
although it's not guaranteed not to change between Xen releases) provides 
access to some IO performance statistics - maybe that would server your 
needs.

Hope that helps,
Mark

> cheers,
> Ashish
>
> On 4/24/07, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:
> > > I am using xentrace to understand performance bottlenecks for an
> > > application inside domU.
> > > My question is how can I distinguish between network IO events and disk
> >
> > IO
> >
> > > events using xentrace ?
> >
> > I don't know if there are trace events currently generated for grant copy
> > operations - if not, you could add them and use these to judge the amount
> > of
> > incoming network traffic.
> >
> > Outgoing network traffic and disk IO are harder to distinguish since they
> > both
> > just use temporary sharing grants.  It might be easier to use some sort
> > of IO
> > monitoring tools within dom0 and the domU in question, similarly to as
> > you would on a native Linux system.
> >
> > > A second related question is can I figure out disk queue waiting times
> >
> > and
> >
> > > serving times (and similarly for the network) to figure out any
> >
> > bottlenecks
> >
> > > or any external stress on these resources that may be causing the guest
> > > machine to be slowing down ?
> >
> > Again, it's worth taking a look at native Linux tools.  XenMon may
> > provide some feedback but I imagine you might have already tried this?
> >
> > For more complete profiling, Xenoprof allows you to run oprofile against
> > multiple domains (and Xen itself) at once).
> >
> > > I greatly appreciate any advise/insight from fellow members here.
> >
> > Sorry not to be more specific.
> >
> > Cheers,
> > Mark
> >
> > > cheers,
> > > Ashish
> > >
> > > On 23/1/07 19:31, "Rob Gardner" <rob.gardner@xxxxxx> wrote:
> > > >*> *
> > > >*> Grant transfers are no longer used to move network data from
> > > > netback to* *> netfront (except for backward compatibility with old
> > > > netfront drivers).* * *
> > > >* *
> > > >* Yeah, got that. ;)  Could you explain what mechanisms are currently
> > > > used* * to move data for net I/O and disk I/O between domains, and
> > > > in* * particular, can you suggest where in the code I could put trace
> > > > calls to* * be able to count I/O's? Thanks.*
> > >
> > > Everything is done via grant-map/unmap commands as it always was,
> > > except network receive (netback->netfront) which is done via grant-copy
> >
> > commands
> >
> > > (one per contiguous fragment of network packet).
> > >
> > >  -- Keir
> >
> > --
> > Dave: Just a question. What use is a unicyle with no seat?  And no
> > pedals! Mark: To answer a question with a question: What use is a
> > skateboard? Dave: Skateboards have wheels.
> > Mark: My wheel has a wheel!



-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: event TRC_MEM_PAGE_GRANT_TRANSFER

[Ashish Gupta]

[-- Attachment #1.1: Type: text/plain, Size: 5542 bytes --]

>
>
>
> This code is in dom0, not in Xen itself so you won't be able to do it with
> the
> tracebuffer - you'd need to set up a dom0-based solution.


Good point. I am not sure if I can add Xentrace instrumentation (or some
other) to dom0 code ?

> As for native Linux tools, if I run them in dom0 , will that be able to
> > tell me all about disk and  network IO for the guest ? I am not allowed
> to
> > run any thing inside domU to meet the black box requirement.
>
> Well, if you measure the amount of traffic the vifs have seen that should
> tell
> you how much network IO had occurred.  Block-wise, I'm not sure but I
> think
> each domain gets a block IO thread for purposes of being able to use the
> IO
> schedulers to regulate each domain...  If I'm right and that patch got
> applied, you may be able to view IO stats for these kernel threads.
> Otherwise, you could maybe find and apply the patch to dom0 yourself.


I will try finding this patch...IO stats for these threads can definitely
help. As you said global info may not be that difficult...its getting this
info at the domain level that's the real issue.

For example something I am trying to find is this: Does the network send
operation block for a certain application either due to network congestion
or sockets blocking on the other end ? This requires me too see the incoming
send requests to dom0 from the guest and see if dom0 can actually send them
at a fast enough rate. I am not sure if this strategy will actually work so
just exploring.


FWIW I think Xentop (and therefore libxenstat - which you could link against
> although it's not guaranteed not to change between Xen releases) provides
> access to some IO performance statistics - maybe that would server your
> needs.


xentop outputs netrx and nettx info per domain --> this can be quite useful.

What is VBD in xentop output ? Does it correspond to page switches ? I tried
finding more details about what xentop reports but I couldn't find it
online.

I think using this with iostat and vmstat could also tell me something
useful. But iostat will give me global info though, so if I can correlate
that with the domain level info from xentop, that might work at some level

BTW, all the replies here are immensely useful...I am learning smth. new
from every comment. Once I am able to get something done, hopefully I can
summarize my findings for tracking Xen stats for benefits of Xen community.
Thanks again.


cheers,
Ashish


Hope that helps,
> Mark
>
> > cheers,
> > Ashish
> >
> > On 4/24/07, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:
> > > > I am using xentrace to understand performance bottlenecks for an
> > > > application inside domU.
> > > > My question is how can I distinguish between network IO events and
> disk
> > >
> > > IO
> > >
> > > > events using xentrace ?
> > >
> > > I don't know if there are trace events currently generated for grant
> copy
> > > operations - if not, you could add them and use these to judge the
> amount
> > > of
> > > incoming network traffic.
> > >
> > > Outgoing network traffic and disk IO are harder to distinguish since
> they
> > > both
> > > just use temporary sharing grants.  It might be easier to use some
> sort
> > > of IO
> > > monitoring tools within dom0 and the domU in question, similarly to as
> > > you would on a native Linux system.
> > >
> > > > A second related question is can I figure out disk queue waiting
> times
> > >
> > > and
> > >
> > > > serving times (and similarly for the network) to figure out any
> > >
> > > bottlenecks
> > >
> > > > or any external stress on these resources that may be causing the
> guest
> > > > machine to be slowing down ?
> > >
> > > Again, it's worth taking a look at native Linux tools.  XenMon may
> > > provide some feedback but I imagine you might have already tried this?
> > >
> > > For more complete profiling, Xenoprof allows you to run oprofile
> against
> > > multiple domains (and Xen itself) at once).
> > >
> > > > I greatly appreciate any advise/insight from fellow members here.
> > >
> > > Sorry not to be more specific.
> > >
> > > Cheers,
> > > Mark
> > >
> > > > cheers,
> > > > Ashish
> > > >
> > > > On 23/1/07 19:31, "Rob Gardner" <rob.gardner@xxxxxx> wrote:
> > > > >*> *
> > > > >*> Grant transfers are no longer used to move network data from
> > > > > netback to* *> netfront (except for backward compatibility with
> old
> > > > > netfront drivers).* * *
> > > > >* *
> > > > >* Yeah, got that. ;)  Could you explain what mechanisms are
> currently
> > > > > used* * to move data for net I/O and disk I/O between domains, and
> > > > > in* * particular, can you suggest where in the code I could put
> trace
> > > > > calls to* * be able to count I/O's? Thanks.*
> > > >
> > > > Everything is done via grant-map/unmap commands as it always was,
> > > > except network receive (netback->netfront) which is done via
> grant-copy
> > >
> > > commands
> > >
> > > > (one per contiguous fragment of network packet).
> > > >
> > > >  -- Keir
> > >
> > > --
> > > Dave: Just a question. What use is a unicyle with no seat?  And no
> > > pedals! Mark: To answer a question with a question: What use is a
> > > skateboard? Dave: Skateboards have wheels.
> > > Mark: My wheel has a wheel!
>
>
>
> --
> Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
> Mark: To answer a question with a question: What use is a skateboard?
> Dave: Skateboards have wheels.
> Mark: My wheel has a wheel!
>



-- 
http://www.cs.northwestern.edu/~agupta

[-- Attachment #1.2: Type: text/html, Size: 7170 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel