Re: [RFC PATCH 0/4] Implement multiqueue virtio-net

Michael S. Tsirkin
2010-09-08T10:55:18Z

On Wed, Sep 08, 2010 at 02:53:03PM +0530, Krishna Kumar2 wrote:
> "Michael S. Tsirkin" <mst@redhat.com> wrote on 09/08/2010 01:40:11 PM:
>
> >
> _______________________________________________________________________________
>
> > > TCP (#numtxqs=2)
> > > N# BW1 BW2 (%) SD1 SD2 (%) RSD1 RSD2
> (%)
> > >
> >
> _______________________________________________________________________________
>
> > > 4 26387 40716 (54.30) 20 28 (40.00) 86i 85
> (-1.16)
> > > 8 24356 41843 (71.79) 88 129 (46.59) 372 362
> (-2.68)
> > > 16 23587 40546 (71.89) 375 564 (50.40) 1558 1519
> (-2.50)
> > > 32 22927 39490 (72.24) 1617 2171 (34.26) 6694 5722
> (-14.52)
> > > 48 23067 39238 (70.10) 3931 5170 (31.51) 15823 13552
> (-14.35)
> > > 64 22927 38750 (69.01) 7142 9914 (38.81) 28972 26173
> (-9.66)
> > > 96 22568 38520 (70.68) 16258 27844 (71.26) 65944 73031
> (10.74)
> >
> > That's a significant hit in TCP SD. Is it caused by the imbalance between
> > number of queues for TX and RX? Since you mention RX is complete,
> > maybe measure with a balanced TX/RX?
>
> Yes, I am not sure why it is so high.

Any errors at higher levels? Are any packets reordered?

> I found the same with #RX=#TX
> too. As a hack, I tried ixgbe without MQ (set "indices=1" before
> calling alloc_etherdev_mq, not sure if that is entirely correct) -
> here too SD worsened by around 40%. I can't explain it, since the
> virtio-net driver runs lock free once sch_direct_xmit gets
> HARD_TX_LOCK for the specific txq. Maybe the SD calculation is not strictly
> correct since
> more threads are now running parallel and load is higher? Eg, if you
> compare SD between
> #netperfs = 8 vs 16 for original code (cut-n-paste relevant columns
> only) ...
>
> N# BW SD
> 8 24356 88
> 16 23587 375
>
> ... SD has increased more than 4 times for the same BW.
>
> > What happens with a single netperf?
> > host -> guest performance with TCP and small packet speed
> > are also worth measuring.
>
> OK, I will do this and send the results later today.
>
> > At some level, host/guest communication is easy in that we don't really
> > care which queue is used. I would like to give some thought (and
> > testing) to how is this going to work with a real NIC card and packet
> > steering at the backend.
> > Any idea?
>
> I have done a little testing with guest -> remote server both
> using a bridge and with macvtap (mq is required only for rx).
> I didn't understand what you mean by packet steering though,
> is it whether packets go out of the NIC on different queues?
> If so, I verified that is the case by putting a counter and
> displaying through /debug interface on the host. dev_queue_xmit
> on the host handles it by calling dev_pick_tx().
>
> > > Guest interrupts for a 4 TXQ device after a 5 min test:
> > > # egrep "virtio0|CPU" /proc/interrupts
> > > CPU0 CPU1 CPU2 CPU3
> > > 40: 0 0 0 0 PCI-MSI-edge virtio0-config
> > > 41: 126955 126912 126505 126940 PCI-MSI-edge virtio0-input
> > > 42: 108583 107787 107853 107716 PCI-MSI-edge virtio0-output.0
> > > 43: 300278 297653 299378 300554 PCI-MSI-edge virtio0-output.1
> > > 44: 372607 374884 371092 372011 PCI-MSI-edge virtio0-output.2
> > > 45: 162042 162261 163623 162923 PCI-MSI-edge virtio0-output.3
> >
> > Does this mean each interrupt is constantly bouncing between CPUs?
>
> Yes. I didn't do *any* tuning for the tests. The only "tuning"
> was to use 64K IO size with netperf. When I ran default netperf
> (16K), I got a little lesser improvement in BW and worse(!) SD
> than with 64K.
>
> Thanks,
>
> - KK