From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Subject: Re: Performance test result between per-vhost kthread disable and enable Date: Tue, 23 Nov 2010 15:41:17 +0200 Message-ID: <20101123134117.GA30256@redhat.com> References: <20101123111251.GA26350@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: tj@kernel.org, sri@us.ibm.com, Avi Kivity , kvm@vger.kernel.org To: lidong chen Return-path: Received: from mx1.redhat.com ([209.132.183.28]:13752 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753949Ab0KWNlj (ORCPT ); Tue, 23 Nov 2010 08:41:39 -0500 Content-Disposition: inline In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: On Tue, Nov 23, 2010 at 09:23:41PM +0800, lidong chen wrote: > At this point, I'd suggest testing vhost-net on the upstream kernel, > not on rhel kernels. The change that introduced per-device threads is= : > c23f3445e68e1db0e74099f264bc5ff5d55ebdeb > i will try this tomorrow. >=20 > Is CONFIG_SCHED_DEBUG set? > yes. CONFIG_SCHED_DEBUG=3Dy. Disable it. Either debug scheduler or perf-test it :) > 2010/11/23 Michael S. Tsirkin : > > On Tue, Nov 23, 2010 at 10:13:43AM +0800, lidong chen wrote: > >> I test the performance between per-vhost kthread disable and enabl= e. > >> > >> Test method: > >> Send the same traffic load between per-vhost kthread disable and > >> enable, and compare the cpu rate of host os. > >> I run five vm on kvm, each of them have five nic. > >> the vhost version which per-vhost kthread disable we used is rhel6 > >> beta 2(2.6.32.60). > >> the vhost version which per-vhost kthread enable we used is rhel6 = (2.6.32-71). > > > > At this point, I'd suggest testing vhost-net on the upstream kernel= , > > not on rhel kernels. The change that introduced per-device threads = is: > > c23f3445e68e1db0e74099f264bc5ff5d55ebdeb > > > >> Test result: > >> with per-vhost kthread disable, the cpu rate of host os is 110%. > >> with per-vhost kthread enable, the cpu rate of host os is 130%. > > > > Is CONFIG_SCHED_DEBUG set? We are stressing the scheduler a lot wit= h > > vhost-net. > > > >> In 2.6.32.60,the whole system only have a kthread. > >> [root@rhel6-kvm1 ~]# ps -ef | grep vhost > >> root =A0 =A0 =A0 973 =A0 =A0 2 =A00 Nov22 ? =A0 =A0 =A0 =A000:00:0= 0 [vhost] > >> > >> In 2.6.32.71,the whole system have 25 kthread. > >> [root@kvm-4slot ~]# ps -ef | grep vhost- > >> root =A0 =A0 12896 =A0 =A0 2 =A00 10:26 ? =A0 =A0 =A0 =A000:00:00 = [vhost-12842] > >> root =A0 =A0 12897 =A0 =A0 2 =A00 10:26 ? =A0 =A0 =A0 =A000:00:00 = [vhost-12842] > >> root =A0 =A0 12898 =A0 =A0 2 =A00 10:26 ? =A0 =A0 =A0 =A000:00:00 = [vhost-12842] > >> root =A0 =A0 12899 =A0 =A0 2 =A00 10:26 ? =A0 =A0 =A0 =A000:00:00 = [vhost-12842] > >> root =A0 =A0 12900 =A0 =A0 2 =A00 10:26 ? =A0 =A0 =A0 =A000:00:00 = [vhost-12842] > >> > >> root =A0 =A0 13022 =A0 =A0 2 =A00 10:26 ? =A0 =A0 =A0 =A000:00:00 = [vhost-12981] > >> root =A0 =A0 13023 =A0 =A0 2 =A00 10:26 ? =A0 =A0 =A0 =A000:00:00 = [vhost-12981] > >> root =A0 =A0 13024 =A0 =A0 2 =A00 10:26 ? =A0 =A0 =A0 =A000:00:00 = [vhost-12981] > >> root =A0 =A0 13025 =A0 =A0 2 =A00 10:26 ? =A0 =A0 =A0 =A000:00:00 = [vhost-12981] > >> root =A0 =A0 13026 =A0 =A0 2 =A00 10:26 ? =A0 =A0 =A0 =A000:00:00 = [vhost-12981] > >> > >> root =A0 =A0 13146 =A0 =A0 2 =A00 10:26 ? =A0 =A0 =A0 =A000:00:00 = [vhost-13088] > >> root =A0 =A0 13147 =A0 =A0 2 =A00 10:26 ? =A0 =A0 =A0 =A000:00:00 = [vhost-13088] > >> root =A0 =A0 13148 =A0 =A0 2 =A00 10:26 ? =A0 =A0 =A0 =A000:00:00 = [vhost-13088] > >> root =A0 =A0 13149 =A0 =A0 2 =A00 10:26 ? =A0 =A0 =A0 =A000:00:00 = [vhost-13088] > >> root =A0 =A0 13150 =A0 =A0 2 =A00 10:26 ? =A0 =A0 =A0 =A000:00:00 = [vhost-13088] > >> ... > >> > >> Code difference: > >> In 2.6.32.60,in function vhost_init, create the kthread for vhost. > >> vhost_workqueue =3D create_singlethread_workqueue("vhost"); > >> > >> In 2.6.32.71,in function vhost_dev_set_owner, create the kthread f= or > >> each nic interface. > >> dev->wq =3D create_singlethread_workqueue(vhost_name); > >> > >> Conclusion: > >> with per-vhost kthread enable, the system can more throughput. > >> but deal the same traffic load with per-vhost kthread enable, it w= aste > >> more cpu resource. > >> > >> In my application scene, the cpu resource is more important, and o= ne > >> kthread for deal with traffic load is enough. > >> > >> So i think we should add a param to control this. > >> for the CPU-bound system, this param disable per-vhost kthread. > >> for the I/O-bound system, this param enable per-vhost kthread. > >> the default value of this param is enable. > >> > >> If my opinion is right, i will give a patch for this. > > > > Let's try to figure out what the issue is, first. > > > > -- > > MST > >