[PATCH 0/3] qemu-kvm: vhost net support

[PATCH 0/3] qemu-kvm: vhost net support

[Michael S. Tsirkin]

This adds support for vhost-net virtio kernel backend.

This is RFC, but works without issues for me.

Still needs to be split up, tested and benchmarked properly,
but posting it here in case people want to test drive
the kernel bits I posted.

Michael S. Tsirkin (3):
  qemu-kvm: move virtio-pci.o to near pci.o
  virtio: move features to an inline function
  qemu-kvm: vhost-net implementation

 Makefile.hw         |    2 +-
 Makefile.target     |    3 +-
 hw/vhost_net.c      |  181 +++++++++++++++++++++++++++++++++++++++++++++++++++
 hw/vhost_net.h      |   30 +++++++++
 hw/virtio-balloon.c |    2 +-
 hw/virtio-blk.c     |    2 +-
 hw/virtio-console.c |    2 +-
 hw/virtio-net.c     |   34 +++++++++-
 hw/virtio-pci.c     |   43 +++++++++++-
 hw/virtio.c         |   19 ------
 hw/virtio.h         |   38 ++++++++++-
 net.c               |    8 ++-
 net.h               |    1 +
 qemu-kvm.c          |    8 --
 qemu-kvm.h          |    9 +++
 15 files changed, 340 insertions(+), 42 deletions(-)
 create mode 100644 hw/vhost_net.c
 create mode 100644 hw/vhost_net.h

Re: [PATCH 0/3] qemu-kvm: vhost net support

[Anthony Liguori]

Michael S. Tsirkin wrote:
> This adds support for vhost-net virtio kernel backend.
>
> This is RFC, but works without issues for me.
>
> Still needs to be split up, tested and benchmarked properly,
> but posting it here in case people want to test drive
> the kernel bits I posted.
>   

Any rough idea on performance?  Better or worse than userspace?

Regards,

Anthony Liguori

Re: [PATCH 0/3] qemu-kvm: vhost net support

[Michael S. Tsirkin]

On Mon, Aug 10, 2009 at 03:51:12PM -0500, Anthony Liguori wrote:
> Michael S. Tsirkin wrote:
>> This adds support for vhost-net virtio kernel backend.
>>
>> This is RFC, but works without issues for me.
>>
>> Still needs to be split up, tested and benchmarked properly,
>> but posting it here in case people want to test drive
>> the kernel bits I posted.
>>   
>
> Any rough idea on performance?  Better or worse than userspace?
>
> Regards,
>
> Anthony Liguori

Well, I definitely see some gain in latency.
Here's a simple test over a 1G ethernet link (host to guest):

Native:
[root@qus18 ~]# netperf -H 11.0.0.1 -t udp_rr
UDP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 11.0.0.1 (11.0.0.1) port 0 AF_INET
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

126976 126976 1        1       10.00    10393.23
124928 124928


vhost virtio:
[root@qus18 ~]# netperf -H 11.0.0.3 -t udp_rr
UDP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 11.0.0.3 (11.0.0.3) port 0 AF_INET
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

126976 126976 1        1       10.00    8169.58
124928 124928

Userspace virtio:
[root@qus18 ~]# netperf -H 11.0.0.3 -t udp_rr
UDP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 11.0.0.3 (11.0.0.3) port 0 AF_INET
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

126976 126976 1        1       10.00    2029.49
124928 124928


Part of it might be that tx mitigation does not come into play with vhost. I
need to disable it in qemu and see.

-- 
MST

Re: [PATCH 0/3] qemu-kvm: vhost net support

[Gregory Haskins]

[-- Attachment #1.1: Type: text/plain, Size: 402 bytes --]

Michael S. Tsirkin wrote:
> This adds support for vhost-net virtio kernel backend.
> 
> This is RFC, but works without issues for me.
> 
> Still needs to be split up, tested and benchmarked properly,
> but posting it here in case people want to test drive
> the kernel bits I posted.

This has a large degree of rejects against qemu-kvm.git/master.  What
tree does this apply to?

-Greg


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 267 bytes --]

[-- Attachment #2: Type: text/plain, Size: 184 bytes --]

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/virtualization

Re: [PATCH 0/3] qemu-kvm: vhost net support

[Michael S. Tsirkin]

On Wed, Aug 12, 2009 at 04:27:44PM -0400, Gregory Haskins wrote:
> Michael S. Tsirkin wrote:
> > This adds support for vhost-net virtio kernel backend.
> > 
> > This is RFC, but works without issues for me.
> > 
> > Still needs to be split up, tested and benchmarked properly,
> > but posting it here in case people want to test drive
> > the kernel bits I posted.
> 
> This has a large degree of rejects against qemu-kvm.git/master.  What
> tree does this apply to?
> 
> -Greg
> 

Likely that tree has advanced since.
This is on top of commit b6bbd41fac4b6fb0efc65e083d2151ce1521f615.

-- 
MST

Re: [PATCH 0/3] qemu-kvm: vhost net support

[Gregory Haskins]

[-- Attachment #1.1: Type: text/plain, Size: 1250 bytes --]

Michael S. Tsirkin wrote:
> On Wed, Aug 12, 2009 at 04:27:44PM -0400, Gregory Haskins wrote:
>> Michael S. Tsirkin wrote:
>>> This adds support for vhost-net virtio kernel backend.
>>>
>>> This is RFC, but works without issues for me.
>>>
>>> Still needs to be split up, tested and benchmarked properly,
>>> but posting it here in case people want to test drive
>>> the kernel bits I posted.
>> This has a large degree of rejects against qemu-kvm.git/master.  What
>> tree does this apply to?
>>
>> -Greg
>>
> 
> Likely that tree has advanced since.
> This is on top of commit b6bbd41fac4b6fb0efc65e083d2151ce1521f615.
> 


Hmm....better, but I still get rejects.  Of particular concern is this
one in net.c:

@@ -1903,7 +1903,7 @@ static TAPState *net_tap_init(VLANState *vlan,
const char *model,
 typedef struct RAWState {
     VLANClientState *vc;
     int fd;
-    uint8_t buf[4096];
+    uint8_t buf[65000];
     int promisc;
 } RAWState;


I do not see any occurrence of "RAWState" in b6bbd41f (or master, for
that matter).  There is probably an operator error somewhere in here ;),
but any help getting this working is appreciated.  Do you have a git
tree I can pull somewhere?

Kind Regards,
-Greg


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 267 bytes --]

[-- Attachment #2: Type: text/plain, Size: 184 bytes --]

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/virtualization

Re: [PATCH 0/3] qemu-kvm: vhost net support

[Michael S. Tsirkin]

On Thu, Aug 13, 2009 at 07:35:52AM -0400, Gregory Haskins wrote:
> Michael S. Tsirkin wrote:
> > On Wed, Aug 12, 2009 at 04:27:44PM -0400, Gregory Haskins wrote:
> >> Michael S. Tsirkin wrote:
> >>> This adds support for vhost-net virtio kernel backend.
> >>>
> >>> This is RFC, but works without issues for me.
> >>>
> >>> Still needs to be split up, tested and benchmarked properly,
> >>> but posting it here in case people want to test drive
> >>> the kernel bits I posted.
> >> This has a large degree of rejects against qemu-kvm.git/master.  What
> >> tree does this apply to?
> >>
> >> -Greg
> >>
> > 
> > Likely that tree has advanced since.
> > This is on top of commit b6bbd41fac4b6fb0efc65e083d2151ce1521f615.
> > 
> 
> 
> Hmm....better, but I still get rejects.  Of particular concern is this
> one in net.c:
> 
> @@ -1903,7 +1903,7 @@ static TAPState *net_tap_init(VLANState *vlan,
> const char *model,
>  typedef struct RAWState {
>      VLANClientState *vc;
>      int fd;
> -    uint8_t buf[4096];
> +    uint8_t buf[65000];
>      int promisc;
>  } RAWState;
> 
> 
> I do not see any occurrence of "RAWState" in b6bbd41f (or master, for
> that matter).  There is probably an operator error somewhere in here ;),

Yes. Mine :)

> but any help getting this working is appreciated.

I reposted a clean one which is against latest bits earlier today.
Look for PATCHv2 in your inbox.

> Do you have a git tree I can pull somewhere?
> Kind Regards,
> -Greg
> 

Thanks for the patience,

-- 
MST

Re: [PATCH 0/3] qemu-kvm: vhost net support

[Or Gerlitz]

Michael S. Tsirkin wrote:
> Well, I definitely see some gain in latency.
> Here's a simple test over a 1G ethernet link (host to guest):

> Native:
> 126976 126976 1        1       10.00    10393.23

> vhost virtio:
> 126976 126976 1        1       10.00    8169.58

> Userspace virtio:
> 126976 126976 1        1       10.00    2029.49

Hi Michael,

Can you provide udp stream numbers for small packets, so we can see the 
packets-per-second gain? e.g -- -m 22 which should account for 64 byte frames.

Or.

Re: [PATCH 0/3] qemu-kvm: vhost net support

[Rusty Russell]

On Thu, 13 Aug 2009 03:05:48 am Michael S. Tsirkin wrote:
> On Mon, Aug 10, 2009 at 03:51:12PM -0500, Anthony Liguori wrote:
> > Any rough idea on performance?  Better or worse than userspace?
> 
> Well, I definitely see some gain in latency.
...
> Part of it might be that tx mitigation does not come into play with vhost. I
> need to disable it in qemu and see.

A simple test I've found reasonably indicative when looking at base performance (before stuff like TSO get involved) is to do pings.  Throughput with "ping -s 1024 -l 120 -c 1000000 -f -q" and latency with "ping -c 1000000 -f -q".

(You may need to cut -l down if your ring is too small).

Cheers,
Rusty.

Re: [PATCH 0/3] qemu-kvm: vhost net support

[Michael S. Tsirkin]

On Thu, Aug 20, 2009 at 04:49:47PM +0930, Rusty Russell wrote:
> On Thu, 13 Aug 2009 03:05:48 am Michael S. Tsirkin wrote:
> > On Mon, Aug 10, 2009 at 03:51:12PM -0500, Anthony Liguori wrote:
> > > Any rough idea on performance?  Better or worse than userspace?
> > 
> > Well, I definitely see some gain in latency.
> ...
> > Part of it might be that tx mitigation does not come into play with vhost. I
> > need to disable it in qemu and see.
> 
> A simple test I've found reasonably indicative when looking at base performance (before stuff like TSO get involved) is to do pings.  Throughput with "ping -s 1024 -l 120 -c 1000000 -f -q" and latency with "ping -c 1000000 -f -q".
> 
> (You may need to cut -l down if your ring is too small).
> 
> Cheers,
> Rusty.

OK, will try that.

-- 
MST

Re: [PATCH 0/3] qemu-kvm: vhost net support

[Michael S. Tsirkin]

On Thu, Aug 20, 2009 at 04:49:47PM +0930, Rusty Russell wrote:
> On Thu, 13 Aug 2009 03:05:48 am Michael S. Tsirkin wrote:
> > On Mon, Aug 10, 2009 at 03:51:12PM -0500, Anthony Liguori wrote:
> > > Any rough idea on performance?  Better or worse than userspace?
> > 
> > Well, I definitely see some gain in latency.
> ...
> > Part of it might be that tx mitigation does not come into play with vhost. I
> > need to disable it in qemu and see.
> 
> A simple test I've found reasonably indicative when looking at base performance (before stuff like TSO get involved) is to do pings.  Throughput with "ping -s 1024 -l 120 -c 1000000 -f -q" and latency with "ping -c 1000000 -f -q".
> 
> (You may need to cut -l down if your ring is too small).
> 
> Cheers,
> Rusty.

Surprisingly, I seem to get better-than-userspace throughput with this
benchmark as well.  Here's a test run locally, host-to-guest,
over a veth link:

vhost:
[root@qus19 ~]# ping -c 1000000 -f -q 11.0.0.4
PING 11.0.0.4 (11.0.0.4) 56(84) bytes of data.

--- 11.0.0.4 ping statistics ---
1000000 packets transmitted, 1000000 received, 0% packet loss, time 10230ms
rtt min/avg/max/mdev = 0.005/0.006/0.035/0.000 ms, ipg/ewma 0.010/0.006 ms
[root@qus19 ~]# ping -s 1024 -l 120 -c 1000000 -f -q 11.0.0.4
PING 11.0.0.4 (11.0.0.4) 1024(1052) bytes of data.

--- 11.0.0.4 ping statistics ---
1000000 packets transmitted, 1000000 received, 0% packet loss, time 11270ms
rtt min/avg/max/mdev = 0.006/0.006/0.022/0.002 ms, pipe 120, ipg/ewma 0.011/0.007 ms

qemu:

[root@qus19 ~]# ping -c 1000000 -f -q 12.0.0.2
PING 12.0.0.2 (12.0.0.2) 56(84) bytes of data.

--- 12.0.0.2 ping statistics ---
1000000 packets transmitted, 1000000 received, 0% packet loss, time 59974ms
rtt min/avg/max/mdev = 0.025/0.044/8.076/0.012 ms, pipe 2, ipg/ewma 0.059/0.045 ms

[root@qus19 ~]# ping -s 1024 -l 120 -c 1000000 -f -q 12.0.0.2
PING 12.0.0.2 (12.0.0.2) 1024(1052) bytes of data.

--- 12.0.0.2 ping statistics ---
1000000 packets transmitted, 833496 received, 16% packet loss, time 38841ms
rtt min/avg/max/mdev = 0.133/4.482/9.728/0.771 ms, pipe 120, ipg/ewma 0.038/4.144 ms

Will try over an external link next week when I get access to
a back-to-back setup (don't want to bother with external
switches at this point).

-- 
MST

Re: [PATCH 0/3] qemu-kvm: vhost net support

[Avi Kivity]

On 08/20/2009 08:00 PM, Michael S. Tsirkin wrote:
>
> Surprisingly, I seem to get better-than-userspace throughput with this
> benchmark as well.  Here's a test run locally, host-to-guest,
> over a veth link:
>
> vhost:
> [root@qus19 ~]# ping -c 1000000 -f -q 11.0.0.4
> PING 11.0.0.4 (11.0.0.4) 56(84) bytes of data.
>
> --- 11.0.0.4 ping statistics ---
> 1000000 packets transmitted, 1000000 received, 0% packet loss, time 10230ms
> rtt min/avg/max/mdev = 0.005/0.006/0.035/0.000 ms, ipg/ewma 0.010/0.006 ms
>    

6 microseconds is very impressive for round-trip time.

> [root@qus19 ~]# ping -s 1024 -l 120 -c 1000000 -f -q 11.0.0.4
> PING 11.0.0.4 (11.0.0.4) 1024(1052) bytes of data.
>
> --- 11.0.0.4 ping statistics ---
> 1000000 packets transmitted, 1000000 received, 0% packet loss, time 11270ms
> rtt min/avg/max/mdev = 0.006/0.006/0.022/0.002 ms, pipe 120, ipg/ewma 0.011/0.007 ms
>
>    

That's only 100MB/sec, but understandable given the lack of interrupt 
mitigation.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [PATCH 0/3] qemu-kvm: vhost net support

[Michael S. Tsirkin]

On Thu, Aug 20, 2009 at 08:24:37PM +0300, Avi Kivity wrote:
> On 08/20/2009 08:00 PM, Michael S. Tsirkin wrote:
>>
>> Surprisingly, I seem to get better-than-userspace throughput with this
>> benchmark as well.  Here's a test run locally, host-to-guest,
>> over a veth link:

Bleh, I knew it's too good to be true. botched the setup.
Pls disregard this, will post real numbers soon.
Sorry about the noise,

-- 
MST