Is there data copy between dom0 and domU

Is there data copy between dom0 and domU

[xu cong]

[-- Attachment #1.1: Type: text/plain, Size: 198 bytes --]

When applications running in domU receive/send data from/to network, is
there data copy via shared memory between dom0 and domU? Or the zero copy
is used. How about the disk I/O operations? Thanks.

[-- Attachment #1.2: Type: text/html, Size: 223 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Is there data copy between dom0 and domU

[Wei Liu]

On Wed, Apr 23, 2014 at 12:06:51PM -0400, xu cong wrote:
> When applications running in domU receive/send data from/to network, is
> there data copy via shared memory between dom0 and domU? Or the zero copy
> is used. How about the disk I/O operations? Thanks.

If you're using latest kernel (3.14-ish), guest TX path is zero-copy,
otherwise it's using copy. For RX path it always copies.

I think disk always copies data. But I'm not sure.

Wei.

Re: Is there data copy between dom0 and domU

[Roger Pau Monné]

On 23/04/14 18:14, Wei Liu wrote:
> On Wed, Apr 23, 2014 at 12:06:51PM -0400, xu cong wrote:
>> When applications running in domU receive/send data from/to network, is
>> there data copy via shared memory between dom0 and domU? Or the zero copy
>> is used. How about the disk I/O operations? Thanks.
> 
> If you're using latest kernel (3.14-ish), guest TX path is zero-copy,
> otherwise it's using copy. For RX path it always copies.
> 
> I think disk always copies data. But I'm not sure.

For disks, up to Linux 3.8 data is never copied and starting from 3.8
data can be copied (in the frontend) if both DomU and Dom0 are running
Linux kernel >= 3.8.

Roger.

Re: Is there data copy between dom0 and domU

[xu cong]

[-- Attachment #1.1: Type: text/plain, Size: 881 bytes --]

2014-04-23 12:14 GMT-04:00 Wei Liu <wei.liu2@citrix.com>:

> On Wed, Apr 23, 2014 at 12:06:51PM -0400, xu cong wrote:
> > When applications running in domU receive/send data from/to network, is
> > there data copy via shared memory between dom0 and domU? Or the zero copy
> > is used. How about the disk I/O operations? Thanks.
>
> If you're using latest kernel (3.14-ish), guest TX path is zero-copy,
> otherwise it's using copy. For RX path it always copies.
>
> I think disk always copies data. But I'm not sure.
>
> Wei.
>


I am using the 3.12. So the cpu utilization (top output) of vifx.0 process
in dom0 is for data copy and irq processing, right? I found that if the vif
irq processing is on the same core where the corresponding domU is running,
the CPU utilization of dom0 is lower and the application IOPS is higher. I
think maybe it is caused by less IPIs are issued.

[-- Attachment #1.2: Type: text/html, Size: 1378 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Is there data copy between dom0 and domU

[xu cong]

[-- Attachment #1.1: Type: text/plain, Size: 1450 bytes --]

2014-04-23 12:22 GMT-04:00 Roger Pau Monné <roger.pau@citrix.com>:

> On 23/04/14 18:14, Wei Liu wrote:
> > On Wed, Apr 23, 2014 at 12:06:51PM -0400, xu cong wrote:
> >> When applications running in domU receive/send data from/to network, is
> >> there data copy via shared memory between dom0 and domU? Or the zero
> copy
> >> is used. How about the disk I/O operations? Thanks.
> >
> > If you're using latest kernel (3.14-ish), guest TX path is zero-copy,
> > otherwise it's using copy. For RX path it always copies.
> >
> > I think disk always copies data. But I'm not sure.
>
> For disks, up to Linux 3.8 data is never copied and starting from 3.8
> data can be copied (in the frontend) if both DomU and Dom0 are running
> Linux kernel >= 3.8.
>
> Roger.
>
>
What's the vif irq for? The following is my understanding, I am sure it is
correct or not.
When a packet arrives at physical machine, dom0 picks it up from NIC and
puts it to ring buffer, then notify the destination domU. (Are all of them
done by vif kernel thread?) So the vif irq is issued when dom0 notifies the
destination domU, because vif is the virtual device of domU. If it is
correct, the vif irq (xen-dyn-event) is an event (softirq?) on the event
channel between dom0 and domU. While, if the vif irq processing does not
happens on the same core where destination domU is running, one IPI is
issued to finish the notification.

Regards,
Cong

[-- Attachment #1.2: Type: text/html, Size: 2066 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Is there data copy between dom0 and domU

[Wei Liu]

On Wed, Apr 23, 2014 at 03:40:25PM -0400, xu cong wrote:
> 2014-04-23 12:22 GMT-04:00 Roger Pau Monné <roger.pau@citrix.com>:
> 
> > On 23/04/14 18:14, Wei Liu wrote:
> > > On Wed, Apr 23, 2014 at 12:06:51PM -0400, xu cong wrote:
> > >> When applications running in domU receive/send data from/to network, is
> > >> there data copy via shared memory between dom0 and domU? Or the zero
> > copy
> > >> is used. How about the disk I/O operations? Thanks.
> > >
> > > If you're using latest kernel (3.14-ish), guest TX path is zero-copy,
> > > otherwise it's using copy. For RX path it always copies.
> > >
> > > I think disk always copies data. But I'm not sure.
> >
> > For disks, up to Linux 3.8 data is never copied and starting from 3.8
> > data can be copied (in the frontend) if both DomU and Dom0 are running
> > Linux kernel >= 3.8.
> >
> > Roger.
> >
> >
> What's the vif irq for? The following is my understanding, I am sure it is
> correct or not.
> When a packet arrives at physical machine, dom0 picks it up from NIC and
> puts it to ring buffer, then notify the destination domU. (Are all of them

Yes.

> done by vif kernel thread?) So the vif irq is issued when dom0 notifies the

For the guest RX path, yes.

> destination domU, because vif is the virtual device of domU. If it is
> correct, the vif irq (xen-dyn-event) is an event (softirq?) on the event
> channel between dom0 and domU. While, if the vif irq processing does not
> happens on the same core where destination domU is running, one IPI is
> issued to finish the notification.
> 

Softirq in Linux refers to a specific type of bottom-half mechanism; IPI
in general means inter-processor interrupt. IIRC xen-dyn-event refers to
event channel event. You might want to distinguish several concepts
before moving on.

Wei.

> Regards,
> Cong

Re: Is there data copy between dom0 and domU

[xu cong]

[-- Attachment #1.1: Type: text/plain, Size: 2211 bytes --]

2014-04-24 12:52 GMT-04:00 Wei Liu <wei.liu2@citrix.com>:

> On Wed, Apr 23, 2014 at 03:40:25PM -0400, xu cong wrote:
> > 2014-04-23 12:22 GMT-04:00 Roger Pau Monné <roger.pau@citrix.com>:
> >
> > > On 23/04/14 18:14, Wei Liu wrote:
> > > > On Wed, Apr 23, 2014 at 12:06:51PM -0400, xu cong wrote:
> > > >> When applications running in domU receive/send data from/to
> network, is
> > > >> there data copy via shared memory between dom0 and domU? Or the zero
> > > copy
> > > >> is used. How about the disk I/O operations? Thanks.
> > > >
> > > > If you're using latest kernel (3.14-ish), guest TX path is zero-copy,
> > > > otherwise it's using copy. For RX path it always copies.
> > > >
> > > > I think disk always copies data. But I'm not sure.
> > >
> > > For disks, up to Linux 3.8 data is never copied and starting from 3.8
> > > data can be copied (in the frontend) if both DomU and Dom0 are running
> > > Linux kernel >= 3.8.
> > >
> > > Roger.
> > >
> > >
> > What's the vif irq for? The following is my understanding, I am sure it
> is
> > correct or not.
> > When a packet arrives at physical machine, dom0 picks it up from NIC and
> > puts it to ring buffer, then notify the destination domU. (Are all of
> them
>
> Yes.
>
> > done by vif kernel thread?) So the vif irq is issued when dom0 notifies
> the
>
> For the guest RX path, yes.
>
> > destination domU, because vif is the virtual device of domU. If it is
> > correct, the vif irq (xen-dyn-event) is an event (softirq?) on the event
> > channel between dom0 and domU. While, if the vif irq processing does not
> > happens on the same core where destination domU is running, one IPI is
> > issued to finish the notification.
> >
>
> Softirq in Linux refers to a specific type of bottom-half mechanism; IPI
> in general means inter-processor interrupt. IIRC xen-dyn-event refers to
> event channel event. You might want to distinguish several concepts
> before moving on.
>
> Wei.
>
> > Regards,
> > Cong
>

Thanks for your clarification. Is there any overhead if two ends of event
channel are not on the same core compared with two ends running on the same
core?

Regards,
Cong

[-- Attachment #1.2: Type: text/html, Size: 2972 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Is there data copy between dom0 and domU

[Ian Campbell]

On Thu, 2014-04-24 at 15:19 -0400, xu cong wrote:

> Is there any overhead if two ends of event channel are not on the same
> core compared with two ends running on the same core?

Why don't you take a look at the code?

Ian