From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=34260 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PS5oA-0000Eg-Cj for qemu-devel@nongnu.org; Mon, 13 Dec 2010 05:39:07 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PS5o9-00053m-33 for qemu-devel@nongnu.org; Mon, 13 Dec 2010 05:39:06 -0500 Received: from mx1.redhat.com ([209.132.183.28]:25328) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PS5o8-000536-JC for qemu-devel@nongnu.org; Mon, 13 Dec 2010 05:39:05 -0500 Date: Mon, 13 Dec 2010 12:38:36 +0200 From: "Michael S. Tsirkin" Message-ID: <20101213103836.GG25590@redhat.com> References: <1292166128-10874-1-git-send-email-stefanha@linux.vnet.ibm.com> <20101212204127.GA24726@redhat.com> <20101212204228.GA24786@redhat.com> <20101212205634.GA24986@redhat.com> <20101212210959.GA25136@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: Content-Transfer-Encoding: quoted-printable Subject: [Qemu-devel] Re: [PATCH v5 0/4] virtio: Use ioeventfd for virtqueue notify List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Stefan Hajnoczi Cc: qemu-devel@nongnu.org On Mon, Dec 13, 2010 at 10:24:51AM +0000, Stefan Hajnoczi wrote: > On Sun, Dec 12, 2010 at 9:09 PM, Michael S. Tsirkin wr= ote: > > On Sun, Dec 12, 2010 at 10:56:34PM +0200, Michael S. Tsirkin wrote: > >> On Sun, Dec 12, 2010 at 10:42:28PM +0200, Michael S. Tsirkin wrote: > >> > On Sun, Dec 12, 2010 at 10:41:28PM +0200, Michael S. Tsirkin wrote= : > >> > > On Sun, Dec 12, 2010 at 03:02:04PM +0000, Stefan Hajnoczi wrote: > >> > > > See below for the v5 changelog. > >> > > > > >> > > > Due to lack of connectivity I am sending from GMail. =A0Git sh= ould retain my > >> > > > stefanha@linux.vnet.ibm.com From address. > >> > > > > >> > > > Virtqueue notify is currently handled synchronously in userspa= ce virtio. =A0This > >> > > > prevents the vcpu from executing guest code while hardware emu= lation code > >> > > > handles the notify. > >> > > > > >> > > > On systems that support KVM, the ioeventfd mechanism can be us= ed to make > >> > > > virtqueue notify a lightweight exit by deferring hardware emul= ation to the > >> > > > iothread and allowing the VM to continue execution. =A0This mo= del is similar to > >> > > > how vhost receives virtqueue notifies. > >> > > > > >> > > > The result of this change is improved performance for userspac= e virtio devices. > >> > > > Virtio-blk throughput increases especially for multithreaded s= cenarios and > >> > > > virtio-net transmit throughput increases substantially. > >> > > > >> > > Interestingly, I see decreased throughput for small message > >> > > host to get netperf runs. > >> > > > >> > > The command that I used was: > >> > > netperf -H $vguest -- -m 200 > >> > > > >> > > And the results are: > >> > > - with ioeventfd=3Doff > >> > > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 11.0.0.= 104 (11.0.0.104) port 0 AF_INET : demo > >> > > Recv =A0 Send =A0 =A0Send =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0Utilization =A0 =A0 =A0 Service Demand > >> > > Socket Socket =A0Message =A0Elapsed =A0 =A0 =A0 =A0 =A0 =A0 =A0S= end =A0 =A0 Recv =A0 =A0 Send =A0 =A0Recv > >> > > Size =A0 Size =A0 =A0Size =A0 =A0 Time =A0 =A0 Throughput =A0loc= al =A0 =A0remote =A0 local =A0 remote > >> > > bytes =A0bytes =A0 bytes =A0 =A0secs. =A0 =A010^6bits/s =A0% S =A0= =A0 =A0% S =A0 =A0 =A0us/KB =A0 us/KB > >> > > > >> > > =A087380 =A016384 =A0 =A0200 =A0 =A010.00 =A0 =A0 =A03035.48 =A0= 15.50 =A0 =A099.30 =A0 =A06.695 =A0 2.680 > >> > > > >> > > - with ioeventfd=3Don > >> > > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 11.0.0.= 104 (11.0.0.104) port 0 AF_INET : demo > >> > > Recv =A0 Send =A0 =A0Send =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0Utilization =A0 =A0 =A0 Service Demand > >> > > Socket Socket =A0Message =A0Elapsed =A0 =A0 =A0 =A0 =A0 =A0 =A0S= end =A0 =A0 Recv =A0 =A0 Send =A0 =A0Recv > >> > > Size =A0 Size =A0 =A0Size =A0 =A0 Time =A0 =A0 Throughput =A0loc= al =A0 =A0remote =A0 local =A0 remote > >> > > bytes =A0bytes =A0 bytes =A0 =A0secs. =A0 =A010^6bits/s =A0% S =A0= =A0 =A0% S =A0 =A0 =A0us/KB =A0 us/KB > >> > > > >> > > =A087380 =A016384 =A0 =A0200 =A0 =A010.00 =A0 =A0 =A01770.95 =A0= 18.16 =A0 =A051.65 =A0 =A013.442 =A02.389 > >> > > > >> > > > >> > > Do you see this behaviour too? > >> > > >> > Just a note: this is with the patchset ported to qemu-kvm. > >> > >> And just another note: the trend is reversed for larged messages, > >> e.g. with 1.5k messages ioeventfd=3Don outputforms ioeventfd=3Doff. > > > > Another datapoint where I see a regression is with 4000 byte messages > > for guest to host traffic. > > > > ioeventfd=3Doff > > set_up_server could not establish a listen endpoint for =A0port 12865= with family AF_UNSPEC > > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 11.0.0.4 (11= .0.0.4) port 0 AF_INET : demo > > Recv =A0 Send =A0 =A0Send =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0Utilization =A0 =A0 =A0 Service Demand > > Socket Socket =A0Message =A0Elapsed =A0 =A0 =A0 =A0 =A0 =A0 =A0Send =A0= =A0 Recv =A0 =A0 Send =A0 =A0Recv > > Size =A0 Size =A0 =A0Size =A0 =A0 Time =A0 =A0 Throughput =A0local =A0= =A0remote =A0 local =A0 remote > > bytes =A0bytes =A0 bytes =A0 =A0secs. =A0 =A010^6bits/s =A0% S =A0 =A0= =A0% S =A0 =A0 =A0us/KB =A0 us/KB > > > > =A087380 =A016384 =A0 4000 =A0 =A010.00 =A0 =A0 =A07717.56 =A0 98.80 = =A0 =A015.11 =A0 =A01.049 =A0 2.566 > > > > ioeventfd=3Don > > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 11.0.0.4 (11= .0.0.4) port 0 AF_INET : demo > > Recv =A0 Send =A0 =A0Send =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0Utilization =A0 =A0 =A0 Service Demand > > Socket Socket =A0Message =A0Elapsed =A0 =A0 =A0 =A0 =A0 =A0 =A0Send =A0= =A0 Recv =A0 =A0 Send =A0 =A0Recv > > Size =A0 Size =A0 =A0Size =A0 =A0 Time =A0 =A0 Throughput =A0local =A0= =A0remote =A0 local =A0 remote > > bytes =A0bytes =A0 bytes =A0 =A0secs. =A0 =A010^6bits/s =A0% S =A0 =A0= =A0% S =A0 =A0 =A0us/KB =A0 us/KB > > > > =A087380 =A016384 =A0 4000 =A0 =A010.00 =A0 =A0 =A03965.86 =A0 87.69 = =A0 =A015.29 =A0 =A01.811 =A0 5.055 >=20 > Interesting. I posted the following results in an earlier version of > this patch: >=20 > "Sridhar Samudrala collected the following data for > virtio-net with 2.6.36-rc1 on the host and 2.6.34 on the guest. >=20 > Guest to Host TCP_STREAM throughput(Mb/sec) > ------------------------------------------- > Msg Size vhost-net virtio-net virtio-net/ioeventfd > 65536 12755 6430 7590 > 16384 8499 3084 5764 > 4096 4723 1578 3659" >=20 > Here we got a throughput improvement where you got a regression. Your > virtio-net ioeventfd=3Doff throughput is much higher than what we got > (different hardware and configuration, but still I didn't know that > virtio-net reaches 7 Gbit/s!). Which qemu are you running? Mine is upstream qemu-kvm + your patches v4 + my patch to port to qemu-kvm. Are you testing qemu.git? My cpu is Intel(R) Xeon(R) CPU X5560 @ 2.80GHz, I am running without any special flags, so IIRC kvm64 cpu type is emulated. Should really try +x2apic. > I have focussed on the block side of things. Any thoughts about the > virtio-net performance we're seeing? >=20 > " 1024 1827 981 2060 I tried 1.5k, I am getting about 3000 guest to host, but in my testing I get about 2000 without ioeventfd as well. > Host to Guest TCP_STREAM throughput(Mb/sec) > ------------------------------------------- > Msg Size vhost-net virtio-net virtio-net/ioeventfd > 65536 11156 5790 5853 > 16384 10787 5575 5691 > 4096 10452 5556 4277 > 1024 4437 3671 5277 >=20 > Guest to Host TCP_RR latency(transactions/sec) > ---------------------------------------------- >=20 > Msg Size vhost-net virtio-net virtio-net/ioeventfd > 1 9903 3459 3425 > 4096 7185 1931 1899 > 16384 6108 2102 1923 > 65536 3161 1610 1744" >=20 > I'll also run the netperf tests you posted to check what I get. >=20 > Stefan