[Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking on tap

[Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking on tap

[Wangkai (Kevin,C)]

When used a tap as net driver for vm, if too many packets was delivered to the 
guest os via tap interface, the guest os will be blocked on io events for a long
time, while tap driver was busying process packets.

kvm vcpu thread block on io lock call trace:
  __lll_lock_wait
  _L_lock_1004
  __pthread_mutex_lock
  qemu_mutex_lock
  kvm_cpu_exec
  qemu_kvm_cpu_thread_fn
  start_thread

qemu io thread call trace:
  ...
  qemu_net_queue_send
  tap_send
  qemu_iohandler_poll
  main_loop_wait
  main_loop
  

I think the qemu io lock time should be as small as possible, and the io work
slice should be limited at a particular ration or time.

---
Signed-off-by: Wangkai <wangkai86@huawei.com>

diff --git a/net/tap.c b/net/tap.c
index a40f7f0..df9a0eb 100644
--- a/net/tap.c
+++ b/net/tap.c
@@ -189,6 +189,7 @@ static void tap_send(void *opaque)
 {
     TAPState *s = opaque;
     int size;
+    int pkt = 0;
 
     while (qemu_can_send_packet(&s->nc)) {
         uint8_t *buf = s->buf;
@@ -210,6 +211,11 @@ static void tap_send(void *opaque)
         } else if (size < 0) {
             break;
         }
+
+        /* limit io block time slice for 50 packets */
+        pkt++;
+        if (pkt >= 50)
+            break;
     }
 }
 
---
2.0.0

Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking on tap

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1172 bytes --]

On Fri, Jul 11, 2014 at 01:05:30AM +0000, Wangkai (Kevin,C) wrote:
> When used a tap as net driver for vm, if too many packets was delivered to the 
> guest os via tap interface, the guest os will be blocked on io events for a long
> time, while tap driver was busying process packets.
> 
> kvm vcpu thread block on io lock call trace:
>   __lll_lock_wait
>   _L_lock_1004
>   __pthread_mutex_lock
>   qemu_mutex_lock
>   kvm_cpu_exec
>   qemu_kvm_cpu_thread_fn
>   start_thread
> 
> qemu io thread call trace:
>   ...
>   qemu_net_queue_send
>   tap_send
>   qemu_iohandler_poll
>   main_loop_wait
>   main_loop
>   
> 
> I think the qemu io lock time should be as small as possible, and the io work
> slice should be limited at a particular ration or time.
> 
> ---
> Signed-off-by: Wangkai <wangkai86@huawei.com>

How many packets are you seeing in a single tap_send() call?

Have you profiled the tap_send() code path?  Maybe it is performing some
operation that is very slow.

By the way, if you want good performance you should use vhost_net
instead of userspace vhost_net.  Userspace virtio-net is not very
optimized.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking on tap

[Wangkai (Kevin,C)]

> -----Original Message-----
> From: Stefan Hajnoczi [mailto:stefanha@redhat.com]
> Sent: Friday, July 11, 2014 9:04 PM
> To: Wangkai (Kevin,C)
> Cc: qemu-devel@nongnu.org; aliguori@amazon.com; Lee yang
> Subject: Re: [PATCH] Tap: fix vcpu long time io blocking on tap
> 
> On Fri, Jul 11, 2014 at 01:05:30AM +0000, Wangkai (Kevin,C) wrote:
> > When used a tap as net driver for vm, if too many packets was
> > delivered to the guest os via tap interface, the guest os will be
> > blocked on io events for a long time, while tap driver was busying
> process packets.
> >
> > kvm vcpu thread block on io lock call trace:
> >   __lll_lock_wait
> >   _L_lock_1004
> >   __pthread_mutex_lock
> >   qemu_mutex_lock
> >   kvm_cpu_exec
> >   qemu_kvm_cpu_thread_fn
> >   start_thread
> >
> > qemu io thread call trace:
> >   ...
> >   qemu_net_queue_send
> >   tap_send
> >   qemu_iohandler_poll
> >   main_loop_wait
> >   main_loop
> >
> >
> > I think the qemu io lock time should be as small as possible, and the
> > io work slice should be limited at a particular ration or time.
> >
> > ---
> > Signed-off-by: Wangkai <wangkai86@huawei.com>
> 
> How many packets are you seeing in a single tap_send() call?
> 
> Have you profiled the tap_send() code path?  Maybe it is performing
> some operation that is very slow.
> 
> By the way, if you want good performance you should use vhost_net
> instead of userspace vhost_net.  Userspace virtio-net is not very
> optimized.
> 
> Stefan


Hi Stefan,

I am not use profile, just debug with gdb and code review.

When packets delivered, I found the VM was hung, and I check qemu run

State by gdb, I see the call trace for IO thread and vcpu thread, and

I add debug info to check how many packets within tap_send, the info below:

total recv 393520 time 1539821 us 
total recv 1270 time 4931 us
total recv 257872 time 995828 us 
total recv 10745 time 41438 us
total recv 505387 time 2000925 us
total recv 535 time 1891 us
total recv 168934 time 646008 us


regards,
Wangkai

Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking on tap

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2690 bytes --]

On Mon, Jul 14, 2014 at 01:55:05AM +0000, Wangkai (Kevin,C) wrote:
> 
> 
> > -----Original Message-----
> > From: Stefan Hajnoczi [mailto:stefanha@redhat.com]
> > Sent: Friday, July 11, 2014 9:04 PM
> > To: Wangkai (Kevin,C)
> > Cc: qemu-devel@nongnu.org; aliguori@amazon.com; Lee yang
> > Subject: Re: [PATCH] Tap: fix vcpu long time io blocking on tap
> > 
> > On Fri, Jul 11, 2014 at 01:05:30AM +0000, Wangkai (Kevin,C) wrote:
> > > When used a tap as net driver for vm, if too many packets was
> > > delivered to the guest os via tap interface, the guest os will be
> > > blocked on io events for a long time, while tap driver was busying
> > process packets.
> > >
> > > kvm vcpu thread block on io lock call trace:
> > >   __lll_lock_wait
> > >   _L_lock_1004
> > >   __pthread_mutex_lock
> > >   qemu_mutex_lock
> > >   kvm_cpu_exec
> > >   qemu_kvm_cpu_thread_fn
> > >   start_thread
> > >
> > > qemu io thread call trace:
> > >   ...
> > >   qemu_net_queue_send
> > >   tap_send
> > >   qemu_iohandler_poll
> > >   main_loop_wait
> > >   main_loop
> > >
> > >
> > > I think the qemu io lock time should be as small as possible, and the
> > > io work slice should be limited at a particular ration or time.
> > >
> > > ---
> > > Signed-off-by: Wangkai <wangkai86@huawei.com>
> > 
> > How many packets are you seeing in a single tap_send() call?
> > 
> > Have you profiled the tap_send() code path?  Maybe it is performing
> > some operation that is very slow.
> > 
> > By the way, if you want good performance you should use vhost_net
> > instead of userspace vhost_net.  Userspace virtio-net is not very
> > optimized.
> > 
> > Stefan
> 
> 
> Hi Stefan,
> 
> I am not use profile, just debug with gdb and code review.

It's worth understanding the root cause for this behavior because
something is probably wrong.

> When packets delivered, I found the VM was hung, and I check qemu run
> 
> State by gdb, I see the call trace for IO thread and vcpu thread, and
> 
> I add debug info to check how many packets within tap_send, the info below:
> 
> total recv 393520 time 1539821 us 
> total recv 1270 time 4931 us
> total recv 257872 time 995828 us 
> total recv 10745 time 41438 us
> total recv 505387 time 2000925 us

505387 packets or 505387 bytes?

If that's packets, then even with small 64-byte packets that would mean
32 MB of pending data!

Are you running a networking benchmark where you'd expect lots of
packets?

Have you checked how lot the time between tap_send() calls is?  Perhaps
something else in QEMU is blocking the event loop so packets accumulate
in the host kernel.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking on tap

[Wangkai (Kevin,C)]

> -----Original Message-----
> From: Stefan Hajnoczi [mailto:stefanha@redhat.com]
> Sent: Monday, July 14, 2014 4:43 PM
> To: Wangkai (Kevin,C)
> Cc: qemu-devel@nongnu.org; aliguori@amazon.com; Lee yang
> Subject: Re: [PATCH] Tap: fix vcpu long time io blocking on tap
> 
> On Mon, Jul 14, 2014 at 01:55:05AM +0000, Wangkai (Kevin,C) wrote:
> >
> >
> > > -----Original Message-----
> > > From: Stefan Hajnoczi [mailto:stefanha@redhat.com]
> > > Sent: Friday, July 11, 2014 9:04 PM
> > > To: Wangkai (Kevin,C)
> > > Cc: qemu-devel@nongnu.org; aliguori@amazon.com; Lee yang
> > > Subject: Re: [PATCH] Tap: fix vcpu long time io blocking on tap
> > >
> > > On Fri, Jul 11, 2014 at 01:05:30AM +0000, Wangkai (Kevin,C) wrote:
> > > > When used a tap as net driver for vm, if too many packets was
> > > > delivered to the guest os via tap interface, the guest os will be
> > > > blocked on io events for a long time, while tap driver was
> busying
> > > process packets.
> > > >
> > > > kvm vcpu thread block on io lock call trace:
> > > >   __lll_lock_wait
> > > >   _L_lock_1004
> > > >   __pthread_mutex_lock
> > > >   qemu_mutex_lock
> > > >   kvm_cpu_exec
> > > >   qemu_kvm_cpu_thread_fn
> > > >   start_thread
> > > >
> > > > qemu io thread call trace:
> > > >   ...
> > > >   qemu_net_queue_send
> > > >   tap_send
> > > >   qemu_iohandler_poll
> > > >   main_loop_wait
> > > >   main_loop
> > > >
> > > >
> > > > I think the qemu io lock time should be as small as possible, and
> > > > the io work slice should be limited at a particular ration or
> time.
> > > >
> > > > ---
> > > > Signed-off-by: Wangkai <wangkai86@huawei.com>
> > >
> > > How many packets are you seeing in a single tap_send() call?
> > >
> > > Have you profiled the tap_send() code path?  Maybe it is performing
> > > some operation that is very slow.
> > >
> > > By the way, if you want good performance you should use vhost_net
> > > instead of userspace vhost_net.  Userspace virtio-net is not very
> > > optimized.
> > >
> > > Stefan
> >
> >
> > Hi Stefan,
> >
> > I am not use profile, just debug with gdb and code review.
> 
> It's worth understanding the root cause for this behavior because
> something is probably wrong.
> 
> > When packets delivered, I found the VM was hung, and I check qemu run
> >
> > State by gdb, I see the call trace for IO thread and vcpu thread, and
> >
> > I add debug info to check how many packets within tap_send, the info
> below:
> >
> > total recv 393520 time 1539821 us
> > total recv 1270 time 4931 us
> > total recv 257872 time 995828 us
> > total recv 10745 time 41438 us
> > total recv 505387 time 2000925 us
> 
> 505387 packets or 505387 bytes?
> 
> If that's packets, then even with small 64-byte packets that would mean
> 32 MB of pending data!
> 
> Are you running a networking benchmark where you'd expect lots of
> packets?
> 
> Have you checked how lot the time between tap_send() calls is?  Perhaps
> something else in QEMU is blocking the event loop so packets accumulate
> in the host kernel.
> 
> Stefan

Hi Stefan,

Here the detail network:

+--------------------------------------------+
| The host add tap1 and eth10 to bridge 'br1'|                     +--------+
| +------------+                             |                     |  send  |
| |   VM  eth1-+-tap1 --- bridge --- eth10 --+---------------------+ packets|
| +------------+                             |                     |        |
+--------------------------------------------+                     +--------+

Qemu start vm by virtio, use tap interface, option is:
-net nic,vlan=101, model=virtio -net tap,vlan=101,ifname=tap1,script=no,downscript=no

And add tap1 and eth10 to bridge br1 in the host:
Brctl addif br1 tap1
Brctl addif br1 eth10

total recv 505387 time 2000925 us:
means call tap_send once dealing 505387 packets, the packet payload was 300 bytes, and
time use for tap_send() was 2,000,925 micro-seconds, time was measured by record time stamp
at function tap_send() start and end.

We just test the performance of VM.

Regards
Wangkai

Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking on tap

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1816 bytes --]

On Mon, Jul 14, 2014 at 10:44:58AM +0000, Wangkai (Kevin,C) wrote:
> Here the detail network:
> 
> +--------------------------------------------+
> | The host add tap1 and eth10 to bridge 'br1'|                     +--------+
> | +------------+                             |                     |  send  |
> | |   VM  eth1-+-tap1 --- bridge --- eth10 --+---------------------+ packets|
> | +------------+                             |                     |        |
> +--------------------------------------------+                     +--------+
> 
> Qemu start vm by virtio, use tap interface, option is:
> -net nic,vlan=101, model=virtio -net tap,vlan=101,ifname=tap1,script=no,downscript=no

Use the newer -netdev/-device syntax to get offload support and slightly
better performance:

-netdev tap,id=tap0,ifname=tap1,script=no,downscript=no \
-device virtio-net-pci,netdev=tap0

> And add tap1 and eth10 to bridge br1 in the host:
> Brctl addif br1 tap1
> Brctl addif br1 eth10
> 
> total recv 505387 time 2000925 us:
> means call tap_send once dealing 505387 packets, the packet payload was 300 bytes, and
> time use for tap_send() was 2,000,925 micro-seconds, time was measured by record time stamp
> at function tap_send() start and end.
> 
> We just test the performance of VM.

That is 150 MB of incoming packets in a single tap_send().  Network rx
queues are maybe a few 1000 packets so I wonder what is going on here.

Maybe more packets are arriving while QEMU is reading them and we keep
looping.  That's strange though because the virtio-net rx virtqueue
should fill up (it only has 256 entries).

Can you investigate more and find out exactly what is going on?  It's
not clear yet that adding a budget is the solution or just hiding a
deeper problem.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking on tap

[Wangkai (Kevin,C)]

> -----Original Message-----
> From: Stefan Hajnoczi [mailto:stefanha@gmail.com]
> Sent: Tuesday, July 15, 2014 11:00 PM
> To: Wangkai (Kevin,C)
> Cc: Stefan Hajnoczi; Lee yang; qemu-devel@nongnu.org;
> aliguori@amazon.com
> Subject: Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking
> on tap
> 
> On Mon, Jul 14, 2014 at 10:44:58AM +0000, Wangkai (Kevin,C) wrote:
> > Here the detail network:
> >
> > +--------------------------------------------+
> > | The host add tap1 and eth10 to bridge 'br1'|                     +-
> -------+
> > | +------------+                             |                     |
> send  |
> > | |   VM  eth1-+-tap1 --- bridge --- eth10 --+---------------------+
> > | | packets|
> > | +------------+                             |                     |
> |
> > +--------------------------------------------+
> > +--------------------------------------------+ +--------+
> >
> > Qemu start vm by virtio, use tap interface, option is:
> > -net nic,vlan=101, model=virtio -net
> > tap,vlan=101,ifname=tap1,script=no,downscript=no
> 
> Use the newer -netdev/-device syntax to get offload support and
> slightly better performance:
> 
> -netdev tap,id=tap0,ifname=tap1,script=no,downscript=no \ -device
> virtio-net-pci,netdev=tap0
> 
> > And add tap1 and eth10 to bridge br1 in the host:
> > Brctl addif br1 tap1
> > Brctl addif br1 eth10
> >
> > total recv 505387 time 2000925 us:
> > means call tap_send once dealing 505387 packets, the packet payload
> > was 300 bytes, and time use for tap_send() was 2,000,925
> > micro-seconds, time was measured by record time stamp at function
> tap_send() start and end.
> >
> > We just test the performance of VM.
> 
> That is 150 MB of incoming packets in a single tap_send().  Network rx
> queues are maybe a few 1000 packets so I wonder what is going on here.
> 
> Maybe more packets are arriving while QEMU is reading them and we keep
> looping.  That's strange though because the virtio-net rx virtqueue
> should fill up (it only has 256 entries).
> 
> Can you investigate more and find out exactly what is going on?  It's
> not clear yet that adding a budget is the solution or just hiding a
> deeper problem.
> 
> Stefan
[Wangkai (Kevin,C)] 


Hi Stefan,

Yes, I have one machine keep sending packets 300M/second, about 1M packets 
per second, I am checking the code, you mean virtqueue has only 256 entries,
is this member keep the value? 
vq->vring.num

I will check and add some debug info to check this problem.

I have got the most time taking by the virtio_net_receive()
virtio_net_receive
qemu_deliver_packet
qemu_net_queue_send
net_hub_port_receive
qemu_deliver_packet
qemu_net_queue_send
tap_send
qemu_iohandler_poll
main_loop_wait
main_loop
main

regards
Wangkai

Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking on tap

[Wangkai (Kevin,C)]

> -----Original Message-----
> From: Stefan Hajnoczi [mailto:stefanha@gmail.com]
> Sent: Tuesday, July 15, 2014 11:00 PM
> To: Wangkai (Kevin,C)
> Cc: Stefan Hajnoczi; Lee yang; qemu-devel@nongnu.org;
> aliguori@amazon.com
> Subject: Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking
> on tap
> 
> On Mon, Jul 14, 2014 at 10:44:58AM +0000, Wangkai (Kevin,C) wrote:
> > Here the detail network:
> >
> > +--------------------------------------------+
> > | The host add tap1 and eth10 to bridge 'br1'|                     +-
> -------+
> > | +------------+                             |                     |
> send  |
> > | |   VM  eth1-+-tap1 --- bridge --- eth10 --+---------------------+
> > | | packets|
> > | +------------+                             |                     |
> |
> > +--------------------------------------------+
> > +--------------------------------------------+ +--------+
> >
> > Qemu start vm by virtio, use tap interface, option is:
> > -net nic,vlan=101, model=virtio -net
> > tap,vlan=101,ifname=tap1,script=no,downscript=no
> 
> Use the newer -netdev/-device syntax to get offload support and
> slightly better performance:
> 
> -netdev tap,id=tap0,ifname=tap1,script=no,downscript=no \ -device
> virtio-net-pci,netdev=tap0
> 
> > And add tap1 and eth10 to bridge br1 in the host:
> > Brctl addif br1 tap1
> > Brctl addif br1 eth10
> >
> > total recv 505387 time 2000925 us:
> > means call tap_send once dealing 505387 packets, the packet payload
> > was 300 bytes, and time use for tap_send() was 2,000,925
> > micro-seconds, time was measured by record time stamp at function
> tap_send() start and end.
> >
> > We just test the performance of VM.
> 
> That is 150 MB of incoming packets in a single tap_send().  Network rx
> queues are maybe a few 1000 packets so I wonder what is going on here.
> 
> Maybe more packets are arriving while QEMU is reading them and we keep
> looping.  That's strange though because the virtio-net rx virtqueue
> should fill up (it only has 256 entries).
> 
> Can you investigate more and find out exactly what is going on?  It's
> not clear yet that adding a budget is the solution or just hiding a
> deeper problem.
> 
> Stefan
[Wangkai (Kevin,C)] 

Hi Stefan,

I think I have found the problem, why 256 entries virtqueue cannot prevent
packets receiving.

I have start one SMP guest, which have 2 cores, one core was pending on
Io, and the other core was receiving the packets, and QEMU filling the
virtqueue while the guest kernel was moving the packets out from the 
queue and process.

They were racing, only if the guest got enough packets and receive slower
than QEMU sending, the virtqueue full, then finish once receive.

And I have tried -netdev/-device syntax start guest again, got very few 
Improvement.

Regards
Wangkai

Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking on tap

[Jason Wang]

On 07/17/2014 11:43 AM, Wangkai (Kevin,C) wrote:
>
>> -----Original Message-----
>> From: Stefan Hajnoczi [mailto:stefanha@gmail.com]
>> Sent: Tuesday, July 15, 2014 11:00 PM
>> To: Wangkai (Kevin,C)
>> Cc: Stefan Hajnoczi; Lee yang; qemu-devel@nongnu.org;
>> aliguori@amazon.com
>> Subject: Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking
>> on tap
>>
>> On Mon, Jul 14, 2014 at 10:44:58AM +0000, Wangkai (Kevin,C) wrote:
>>> Here the detail network:
>>>
>>> +--------------------------------------------+
>>> | The host add tap1 and eth10 to bridge 'br1'|                     +-
>> -------+
>>> | +------------+                             |                     |
>> send  |
>>> | |   VM  eth1-+-tap1 --- bridge --- eth10 --+---------------------+
>>> | | packets|
>>> | +------------+                             |                     |
>> |
>>> +--------------------------------------------+
>>> +--------------------------------------------+ +--------+
>>>
>>> Qemu start vm by virtio, use tap interface, option is:
>>> -net nic,vlan=101, model=virtio -net
>>> tap,vlan=101,ifname=tap1,script=no,downscript=no
>> Use the newer -netdev/-device syntax to get offload support and
>> slightly better performance:
>>
>> -netdev tap,id=tap0,ifname=tap1,script=no,downscript=no \ -device
>> virtio-net-pci,netdev=tap0
>>
>>> And add tap1 and eth10 to bridge br1 in the host:
>>> Brctl addif br1 tap1
>>> Brctl addif br1 eth10
>>>
>>> total recv 505387 time 2000925 us:
>>> means call tap_send once dealing 505387 packets, the packet payload
>>> was 300 bytes, and time use for tap_send() was 2,000,925
>>> micro-seconds, time was measured by record time stamp at function
>> tap_send() start and end.
>>> We just test the performance of VM.
>> That is 150 MB of incoming packets in a single tap_send().  Network rx
>> queues are maybe a few 1000 packets so I wonder what is going on here.
>>
>> Maybe more packets are arriving while QEMU is reading them and we keep
>> looping.  That's strange though because the virtio-net rx virtqueue
>> should fill up (it only has 256 entries).
>>
>> Can you investigate more and find out exactly what is going on?  It's
>> not clear yet that adding a budget is the solution or just hiding a
>> deeper problem.
>>
>> Stefan
> [Wangkai (Kevin,C)] 
>
> Hi Stefan,
>
> I think I have found the problem, why 256 entries virtqueue cannot prevent
> packets receiving.
>
> I have start one SMP guest, which have 2 cores, one core was pending on
> Io, and the other core was receiving the packets, and QEMU filling the
> virtqueue while the guest kernel was moving the packets out from the 
> queue and process.
>
> They were racing, only if the guest got enough packets and receive slower
> than QEMU sending, the virtqueue full, then finish once receive.
I hit the similar issue in the past. Using pktgen to inject packets from
tap to guest directly, then guest was slow to respond any other io event.

This is similar to the tx and we have tx_burst or tx_timer to solve it.
But for rx, probably we could do some limitation in tap_send() like this
patch since e1000 may have the same issue.

Generically, we need some mechanism to guarantee the fairness between
devices to prevent one from starving the others.
>
> And I have tried -netdev/-device syntax start guest again, got very few 
> Improvement.
>
> Regards
> Wangkai
>

Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking on tap

[Wangkai (Kevin,C)]

> -----Original Message-----
> From: Jason Wang [mailto:jasowang@redhat.com]
> Sent: Thursday, July 17, 2014 1:37 PM
> To: Wangkai (Kevin,C); Stefan Hajnoczi
> Cc: Lee yang; qemu-devel@nongnu.org; Stefan Hajnoczi;
> aliguori@amazon.com; Michael S. Tsirkin
> Subject: Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking
> on tap
> 
> On 07/17/2014 11:43 AM, Wangkai (Kevin,C) wrote:
> >
> >> -----Original Message-----
> >> From: Stefan Hajnoczi [mailto:stefanha@gmail.com]
> >> Sent: Tuesday, July 15, 2014 11:00 PM
> >> To: Wangkai (Kevin,C)
> >> Cc: Stefan Hajnoczi; Lee yang; qemu-devel@nongnu.org;
> >> aliguori@amazon.com
> >> Subject: Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io
> blocking
> >> on tap
> >>
> >> On Mon, Jul 14, 2014 at 10:44:58AM +0000, Wangkai (Kevin,C) wrote:
> >>> Here the detail network:
> >>>
> >>> +--------------------------------------------+
> >>> | The host add tap1 and eth10 to bridge 'br1'|
> +-
> >> -------+
> >>> | +------------+                             |
> |
> >> send  |
> >>> | |   VM  eth1-+-tap1 --- bridge --- eth10 --+---------------------
> +
> >>> | | packets|
> >>> | +------------+                             |
> |
> >> |
> >>> +--------------------------------------------+
> >>> +--------------------------------------------+ +--------+
> >>>
> >>> Qemu start vm by virtio, use tap interface, option is:
> >>> -net nic,vlan=101, model=virtio -net
> >>> tap,vlan=101,ifname=tap1,script=no,downscript=no
> >> Use the newer -netdev/-device syntax to get offload support and
> >> slightly better performance:
> >>
> >> -netdev tap,id=tap0,ifname=tap1,script=no,downscript=no \ -device
> >> virtio-net-pci,netdev=tap0
> >>
> >>> And add tap1 and eth10 to bridge br1 in the host:
> >>> Brctl addif br1 tap1
> >>> Brctl addif br1 eth10
> >>>
> >>> total recv 505387 time 2000925 us:
> >>> means call tap_send once dealing 505387 packets, the packet payload
> >>> was 300 bytes, and time use for tap_send() was 2,000,925
> >>> micro-seconds, time was measured by record time stamp at function
> >> tap_send() start and end.
> >>> We just test the performance of VM.
> >> That is 150 MB of incoming packets in a single tap_send().  Network
> >> rx queues are maybe a few 1000 packets so I wonder what is going on
> here.
> >>
> >> Maybe more packets are arriving while QEMU is reading them and we
> >> keep looping.  That's strange though because the virtio-net rx
> >> virtqueue should fill up (it only has 256 entries).
> >>
> >> Can you investigate more and find out exactly what is going on?
> It's
> >> not clear yet that adding a budget is the solution or just hiding a
> >> deeper problem.
> >>
> >> Stefan
> > [Wangkai (Kevin,C)]
> >
> > Hi Stefan,
> >
> > I think I have found the problem, why 256 entries virtqueue cannot
> > prevent packets receiving.
> >
> > I have start one SMP guest, which have 2 cores, one core was pending
> > on Io, and the other core was receiving the packets, and QEMU filling
> > the virtqueue while the guest kernel was moving the packets out from
> > the queue and process.
> >
> > They were racing, only if the guest got enough packets and receive
> > slower than QEMU sending, the virtqueue full, then finish once
> receive.
> I hit the similar issue in the past. Using pktgen to inject packets
> from tap to guest directly, then guest was slow to respond any other io
> event.
> 
> This is similar to the tx and we have tx_burst or tx_timer to solve it.
> But for rx, probably we could do some limitation in tap_send() like
> this patch since e1000 may have the same issue.
> 
> Generically, we need some mechanism to guarantee the fairness between
> devices to prevent one from starving the others.
[Wangkai (Kevin,C)] 

Yes, the QEMU io shared one thread, and locked, fairness should be 
guaranteed between io events and the io lock should be as small as
possible.

> >
> > And I have tried -netdev/-device syntax start guest again, got very
> > few Improvement.
> >
> > Regards
> > Wangkai
> >

Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking on tap

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1377 bytes --]

On Fri, Jul 11, 2014 at 01:05:30AM +0000, Wangkai (Kevin,C) wrote:
> When used a tap as net driver for vm, if too many packets was delivered to the 
> guest os via tap interface, the guest os will be blocked on io events for a long
> time, while tap driver was busying process packets.

Please tweak this description to explain clearly that the tap_send()
loop can execute for a very long time if more packets are received by
the host during tap_send().

It seems that the guest does not require the QEMU global mutex often
during virtio-net receive, otherwise the rx virtqueue would fill up and
tap_send() would return.

> diff --git a/net/tap.c b/net/tap.c
> index a40f7f0..df9a0eb 100644
> --- a/net/tap.c
> +++ b/net/tap.c
> @@ -189,6 +189,7 @@ static void tap_send(void *opaque)
>  {
>      TAPState *s = opaque;
>      int size;
> +    int pkt = 0;
>  
>      while (qemu_can_send_packet(&s->nc)) {
>          uint8_t *buf = s->buf;
> @@ -210,6 +211,11 @@ static void tap_send(void *opaque)
>          } else if (size < 0) {
>              break;
>          }
> +
> +        /* limit io block time slice for 50 packets */
> +        pkt++;
> +        if (pkt >= 50)
> +            break;

Please use scripts/checkpatch.pl to check coding style.  QEMU always
uses curlies around if statement bodies, even when they are only one
line.

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]