[Question]About KVM network zero-copy feature!

[Question]About KVM network zero-copy feature!

[Peter Huang(Peng)]

Hi,All

I searched from git-log, and found that until now we have vhost TX zero-copy experiment feature, how
about RX zero-copy?

For XEN, net-back also only has TX zero-copy, Is there any reason that RX zero-copy still not implemented?

Thanks!

Re: [Question]About KVM network zero-copy feature!

[Stephen Hemminger]

On Fri, 10 Aug 2012 11:34:32 +0800
"Peter Huang(Peng)" <peter.huangpeng@huawei.com> wrote:

> Hi,All
> 
> I searched from git-log, and found that until now we have vhost TX zero-copy experiment feature, how
> about RX zero-copy?
> 
> For XEN, net-back also only has TX zero-copy, Is there any reason that RX zero-copy still not implemented?
> 
There is no guarantee that packet will ever be read by receiver. This means zero-copy could
create memory back pressure stalls.

Re: [Question]About KVM network zero-copy feature!

[Robert Vineyard]

> On Fri, 10 Aug 2012 11:34:32 +0800
> "Peter Huang(Peng)" <peter.huangpeng@huawei.com> wrote:
>
>> I searched from git-log, and found that until now we have vhost TX zero-copy experiment feature, how
>> about RX zero-copy?

On 08/11/2012 04:55 PM, Stephen Hemminger wrote:
> There is no guarantee that packet will ever be read by receiver. This means zero-copy could
> create memory back pressure stalls.

It would be handy if this could be an optional feature, perhaps not 
enabled by default due to the problem with stalls you mentioned. I would 
love to see RX zero-copy implemented natively in KVM, as it might 
alleviate the need for custom solutions like vPF_RING:

http://www.ntop.org/products/pf_ring/vpf_ring/

Every time a packet is copied, especially from kernel space to user 
space, there is an opportunity for it to be dropped on its way to the 
receiving application - which is unacceptable when monitoring high-speed 
networks for security or bandwidth accounting purposes.

I am attempting to find a highly-efficient way to deploy virtualized 
network monitoring sensors (Snort, for example). Ideally I want to 
exploit symmetric hardware-based RSS and SR-IOV functionality for 
load-balancing and packet distribution completely in ASIC. I've found 
other existing work in this area (also using custom drivers) indicating 
significant performance gains in the non-virtualized case:

http://www.ndsl.kaist.edu/~shinae/papers/TR-symRSS.pdf

Is there any interest in exploring native RX zero-copy within the 
mainline KVM networking code?

Thanks,
Robert Vineyard

Re: [Question]About KVM network zero-copy feature!

[Robert Vineyard]

> On Fri, 10 Aug 2012 11:34:32 +0800
> "Peter Huang(Peng)" <peter.huangpeng@huawei.com> wrote:
>
>> I searched from git-log, and found that until now we have vhost TX zero-copy experiment feature, how
>> about RX zero-copy?

On 08/11/2012 04:55 PM, Stephen Hemminger wrote:
> There is no guarantee that packet will ever be read by receiver. This means zero-copy could
> create memory back pressure stalls.

It would be handy if this could be an optional feature, perhaps not 
enabled by default due to the problem with stalls you mentioned. I would 
love to see RX zero-copy implemented natively in KVM, as it might 
alleviate the need for custom solutions like vPF_RING:

http://www.ntop.org/products/pf_ring/vpf_ring/

Every time a packet is copied, especially from kernel space to user 
space, there is an opportunity for it to be dropped on its way to the 
receiving application - which is unacceptable when monitoring high-speed 
networks for security or bandwidth accounting purposes.

I am attempting to find a highly-efficient way to deploy virtualized 
network monitoring sensors (Snort, for example). Ideally I want to 
exploit symmetric hardware-based RSS and SR-IOV functionality for 
load-balancing and packet distribution completely in ASIC. I've found 
other existing work in this area (also using custom drivers) indicating 
significant performance gains in the non-virtualized case:

http://www.ndsl.kaist.edu/~shinae/papers/TR-symRSS.pdf

Is there any interest in exploring native RX zero-copy within the 
mainline KVM networking code?

Thanks,
Robert Vineyard

Re: [Question]About KVM network zero-copy feature!

[Robert Vineyard]

(adding Xin Xiaohui to the conversation for comment)

According to the NetworkingTodo page on the KVM wiki, zero-copy RX for 
macvtap is in fact on the roadmap, assigned to Xin:

http://www.linux-kvm.org/page/NetworkingTodo

The Release Notes for RHEL 6.2 (originally published on 12/06/2011) also 
specifically mention macvtap/vhost zero-copy capabilities as being 
included as a Technology Preview:

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/6.2_Release_Notes/virtualization.html

Since I've been unable to find much information on the current status of 
this development, can anyone confirm if this functionality is still in 
the works? If so, is there any planned ETA?

Thanks,
Robert Vineyard


On 08/11/2012 05:54 PM, Robert Vineyard wrote:
>> On Fri, 10 Aug 2012 11:34:32 +0800
>> "Peter Huang(Peng)" <peter.huangpeng@huawei.com> wrote:
>>
>>> I searched from git-log, and found that until now we have vhost TX
>>> zero-copy experiment feature, how
>>> about RX zero-copy?
>
> On 08/11/2012 04:55 PM, Stephen Hemminger wrote:
>> There is no guarantee that packet will ever be read by receiver. This
>> means zero-copy could
>> create memory back pressure stalls.
>
> It would be handy if this could be an optional feature, perhaps not
> enabled by default due to the problem with stalls you mentioned. I would
> love to see RX zero-copy implemented natively in KVM, as it might
> alleviate the need for custom solutions like vPF_RING:
>
> http://www.ntop.org/products/pf_ring/vpf_ring/
>
> Every time a packet is copied, especially from kernel space to user
> space, there is an opportunity for it to be dropped on its way to the
> receiving application - which is unacceptable when monitoring high-speed
> networks for security or bandwidth accounting purposes.
>
> I am attempting to find a highly-efficient way to deploy virtualized
> network monitoring sensors (Snort, for example). Ideally I want to
> exploit symmetric hardware-based RSS and SR-IOV functionality for
> load-balancing and packet distribution completely in ASIC. I've found
> other existing work in this area (also using custom drivers) indicating
> significant performance gains in the non-virtualized case:
>
> http://www.ndsl.kaist.edu/~shinae/papers/TR-symRSS.pdf
>
> Is there any interest in exploring native RX zero-copy within the
> mainline KVM networking code?
>
> Thanks,
> Robert Vineyard
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Question]About KVM network zero-copy feature!

[Robert Vineyard]

(adding Xin Xiaohui to the conversation for comment)

According to the NetworkingTodo page on the KVM wiki, zero-copy RX for 
macvtap is in fact on the roadmap, assigned to Xin:

http://www.linux-kvm.org/page/NetworkingTodo

The Release Notes for RHEL 6.2 (originally published on 12/06/2011) also 
specifically mention macvtap/vhost zero-copy capabilities as being 
included as a Technology Preview:

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/6.2_Release_Notes/virtualization.html

Since I've been unable to find much information on the current status of 
this development, can anyone confirm if this functionality is still in 
the works? If so, is there any planned ETA?

Thanks,
Robert Vineyard


On 08/11/2012 05:54 PM, Robert Vineyard wrote:
>> On Fri, 10 Aug 2012 11:34:32 +0800
>> "Peter Huang(Peng)" <peter.huangpeng@huawei.com> wrote:
>>
>>> I searched from git-log, and found that until now we have vhost TX
>>> zero-copy experiment feature, how
>>> about RX zero-copy?
>
> On 08/11/2012 04:55 PM, Stephen Hemminger wrote:
>> There is no guarantee that packet will ever be read by receiver. This
>> means zero-copy could
>> create memory back pressure stalls.
>
> It would be handy if this could be an optional feature, perhaps not
> enabled by default due to the problem with stalls you mentioned. I would
> love to see RX zero-copy implemented natively in KVM, as it might
> alleviate the need for custom solutions like vPF_RING:
>
> http://www.ntop.org/products/pf_ring/vpf_ring/
>
> Every time a packet is copied, especially from kernel space to user
> space, there is an opportunity for it to be dropped on its way to the
> receiving application - which is unacceptable when monitoring high-speed
> networks for security or bandwidth accounting purposes.
>
> I am attempting to find a highly-efficient way to deploy virtualized
> network monitoring sensors (Snort, for example). Ideally I want to
> exploit symmetric hardware-based RSS and SR-IOV functionality for
> load-balancing and packet distribution completely in ASIC. I've found
> other existing work in this area (also using custom drivers) indicating
> significant performance gains in the non-virtualized case:
>
> http://www.ndsl.kaist.edu/~shinae/papers/TR-symRSS.pdf
>
> Is there any interest in exploring native RX zero-copy within the
> mainline KVM networking code?
>
> Thanks,
> Robert Vineyard
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Question]About KVM network zero-copy feature!

[Michael S. Tsirkin]

On Sat, Aug 11, 2012 at 08:42:44PM -0400, Robert Vineyard wrote:
> (adding Xin Xiaohui to the conversation for comment)
> 
> According to the NetworkingTodo page on the KVM wiki, zero-copy RX
> for macvtap is in fact on the roadmap, assigned to Xin:
> 
> http://www.linux-kvm.org/page/NetworkingTodo

AFAIK Xin left Intel and is not working on it.
Contributions are welcome.

> The Release Notes for RHEL 6.2 (originally published on 12/06/2011)
> also specifically mention macvtap/vhost zero-copy capabilities as
> being included as a Technology Preview:
> 
> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/6.2_Release_Notes/virtualization.html
> 
> Since I've been unable to find much information on the current
> status of this development, can anyone confirm if this functionality
> is still in the works? If so, is there any planned ETA?
> 
> Thanks,
> Robert Vineyard

I think this means TX.

-- 
MST

Re: [Question]About KVM network zero-copy feature!

[Michael S. Tsirkin]

On Sat, Aug 11, 2012 at 08:42:44PM -0400, Robert Vineyard wrote:
> (adding Xin Xiaohui to the conversation for comment)
> 
> According to the NetworkingTodo page on the KVM wiki, zero-copy RX
> for macvtap is in fact on the roadmap, assigned to Xin:
> 
> http://www.linux-kvm.org/page/NetworkingTodo

AFAIK Xin left Intel and is not working on it.
Contributions are welcome.

> The Release Notes for RHEL 6.2 (originally published on 12/06/2011)
> also specifically mention macvtap/vhost zero-copy capabilities as
> being included as a Technology Preview:
> 
> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/6.2_Release_Notes/virtualization.html
> 
> Since I've been unable to find much information on the current
> status of this development, can anyone confirm if this functionality
> is still in the works? If so, is there any planned ETA?
> 
> Thanks,
> Robert Vineyard

I think this means TX.

-- 
MST

Re: [Question]About KVM network zero-copy feature!

[Peter Huang(Peng)]

Hi, Michael

IIt will be usefull if we can implement rx zero-copy, could you give
 me some help if you know some technical details or some
references on the internet?

I am wondering may be I can take a deep look on it first then decide
if I can take it over or not.

Thanks a lot.

On 2012/8/12 17:37, Michael S. Tsirkin wrote:
> On Sat, Aug 11, 2012 at 08:42:44PM -0400, Robert Vineyard wrote:
>> (adding Xin Xiaohui to the conversation for comment)
>>
>> According to the NetworkingTodo page on the KVM wiki, zero-copy RX
>> for macvtap is in fact on the roadmap, assigned to Xin:
>>
>> http://www.linux-kvm.org/page/NetworkingTodo
> AFAIK Xin left Intel and is not working on it.
> Contributions are welcome.
>

Re: [Question]About KVM network zero-copy feature!

[Michael S. Tsirkin]

Last patchset version was this:
[PATCH v16 00/17] Provide a zero-copy method on KVM virtio-net

IIRC the main issue was the need to integrate the patchset with
macvtap (as opposed to using a separate device).

On Mon, Aug 13, 2012 at 09:24:54AM +0800, Peter Huang(Peng) wrote:
> Hi, Michael
> 
> IIt will be usefull if we can implement rx zero-copy, could you give
>  me some help if you know some technical details or some
> references on the internet?
> 
> I am wondering may be I can take a deep look on it first then decide
> if I can take it over or not.
> 
> Thanks a lot.
> 
> On 2012/8/12 17:37, Michael S. Tsirkin wrote:
> > On Sat, Aug 11, 2012 at 08:42:44PM -0400, Robert Vineyard wrote:
> >> (adding Xin Xiaohui to the conversation for comment)
> >>
> >> According to the NetworkingTodo page on the KVM wiki, zero-copy RX
> >> for macvtap is in fact on the roadmap, assigned to Xin:
> >>
> >> http://www.linux-kvm.org/page/NetworkingTodo
> > AFAIK Xin left Intel and is not working on it.
> > Contributions are welcome.
> >

Re: [Question]About KVM network zero-copy feature!

[Robert Vineyard]

On 08/12/2012 05:37 AM, Michael S. Tsirkin wrote:
> AFAIK Xin left Intel and is not working on it.
> Contributions are welcome.

That's too bad... do you know of anyone else (at Intel or otherwise) who 
might be familiar enough with the existing codebase to get me started?

 From the code I've looked at, it appears that among other things 
they're using the splice(2) / vmsplice(2) system calls to effect 
"copying" without actually copying data. If I understand the semantics 
correctly, these calls are basically shuffling pointers around to avoid 
unnecessary memcpy(3) / mmap(2) calls. I've even seen a "zero-copy" 
version of sendfile(2) that essentially wraps it around a call to splice(2).

I may be able to hack something together based on the current zero-copy 
TX implementation, but as I'm still wrapping my head around several of 
the concepts I just described, it may be awhile before I can produce 
anything useful. I have quite a bit of experience developing for Linux 
in C, but this would be my first attempt at writing kernel/device-driver 
code.

>> The Release Notes for RHEL 6.2 (originally published on 12/06/2011)
>> also specifically mention macvtap/vhost zero-copy capabilities as
>> being included as a Technology Preview:
>>
>> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/6.2_Release_Notes/virtualization.html
>
> I think this means TX.

It does. I meant to clarify that point in my original email... yes, only 
TX zero-copy is currently implemented, and it is still marked as 
"experimental". Outside of the custom solutions like PF_RING that I 
mentioned, I don't know that I've seen zero-copy for RX.

-- Robert Vineyard

Re: [Question]About KVM network zero-copy feature!

[Robert Vineyard]

On 08/12/2012 05:37 AM, Michael S. Tsirkin wrote:
> AFAIK Xin left Intel and is not working on it.
> Contributions are welcome.

That's too bad... do you know of anyone else (at Intel or otherwise) who 
might be familiar enough with the existing codebase to get me started?

 From the code I've looked at, it appears that among other things 
they're using the splice(2) / vmsplice(2) system calls to effect 
"copying" without actually copying data. If I understand the semantics 
correctly, these calls are basically shuffling pointers around to avoid 
unnecessary memcpy(3) / mmap(2) calls. I've even seen a "zero-copy" 
version of sendfile(2) that essentially wraps it around a call to splice(2).

I may be able to hack something together based on the current zero-copy 
TX implementation, but as I'm still wrapping my head around several of 
the concepts I just described, it may be awhile before I can produce 
anything useful. I have quite a bit of experience developing for Linux 
in C, but this would be my first attempt at writing kernel/device-driver 
code.

>> The Release Notes for RHEL 6.2 (originally published on 12/06/2011)
>> also specifically mention macvtap/vhost zero-copy capabilities as
>> being included as a Technology Preview:
>>
>> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/6.2_Release_Notes/virtualization.html
>
> I think this means TX.

It does. I meant to clarify that point in my original email... yes, only 
TX zero-copy is currently implemented, and it is still marked as 
"experimental". Outside of the custom solutions like PF_RING that I 
mentioned, I don't know that I've seen zero-copy for RX.

-- Robert Vineyard

Re: [Question]About KVM network zero-copy feature!

[Robert Vineyard]

On 08/12/2012 11:45 PM, Robert Vineyard wrote:
> On 08/12/2012 05:37 AM, Michael S. Tsirkin wrote:
>> Contributions are welcome.
>
> That's too bad... do you know of anyone else (at Intel or otherwise) who
> might be familiar enough with the existing codebase to get me started?

I suppose I should have done my homework... *ahem* time for me to finish 
RTFM on git...

I'll come back after digging through more kernel code :-)

-- Robert

Re: [Question]About KVM network zero-copy feature!

[Robert Vineyard]

On 08/12/2012 11:45 PM, Robert Vineyard wrote:
> On 08/12/2012 05:37 AM, Michael S. Tsirkin wrote:
>> Contributions are welcome.
>
> That's too bad... do you know of anyone else (at Intel or otherwise) who
> might be familiar enough with the existing codebase to get me started?

I suppose I should have done my homework... *ahem* time for me to finish 
RTFM on git...

I'll come back after digging through more kernel code :-)

-- Robert

[Question]About KVM network zero-copy feature!

[Peter Huang(Peng)]

Hi,All

I searched from git-log, and found that until now we have vhost TX zero-copy experiment feature, how
about RX zero-copy?

For XEN, net-back also only has TX zero-copy, Is there any reason that RX zero-copy still not implemented?

Thanks!