From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sridhar Samudrala Subject: Re: Performance test result between per-vhost kthread disable and enable Date: Tue, 23 Nov 2010 08:00:09 -0800 Message-ID: <4CEBE509.8020704@us.ibm.com> References: <20101123111251.GA26350@redhat.com> <20101123134117.GA30256@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: lidong chen , tj@kernel.org, Avi Kivity , kvm@vger.kernel.org To: "Michael S. Tsirkin" Return-path: Received: from e6.ny.us.ibm.com ([32.97.182.146]:50703 "EHLO e6.ny.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752138Ab0KWQAy (ORCPT ); Tue, 23 Nov 2010 11:00:54 -0500 Received: from d01dlp01.pok.ibm.com (d01dlp01.pok.ibm.com [9.56.224.56]) by e6.ny.us.ibm.com (8.14.4/8.13.1) with ESMTP id oANG07kY022368 for ; Tue, 23 Nov 2010 11:02:00 -0500 Received: from d01relay04.pok.ibm.com (d01relay04.pok.ibm.com [9.56.227.236]) by d01dlp01.pok.ibm.com (Postfix) with ESMTP id 985AD728047 for ; Tue, 23 Nov 2010 11:00:19 -0500 (EST) Received: from d03av01.boulder.ibm.com (d03av01.boulder.ibm.com [9.17.195.167]) by d01relay04.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id oANG0JZW183422 for ; Tue, 23 Nov 2010 11:00:19 -0500 Received: from d03av01.boulder.ibm.com (loopback [127.0.0.1]) by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id oANG0GIm006530 for ; Tue, 23 Nov 2010 09:00:18 -0700 In-Reply-To: <20101123134117.GA30256@redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: On 11/23/2010 5:41 AM, Michael S. Tsirkin wrote: > On Tue, Nov 23, 2010 at 09:23:41PM +0800, lidong chen wrote: >> At this point, I'd suggest testing vhost-net on the upstream kernel, >> not on rhel kernels. The change that introduced per-device threads is: >> c23f3445e68e1db0e74099f264bc5ff5d55ebdeb >> i will try this tomorrow. >> >> Is CONFIG_SCHED_DEBUG set? >> yes. CONFIG_SCHED_DEBUG=y. > Disable it. Either debug scheduler or perf-test it :) Another debug option to disable is CONFIG_WORKQUEUE_TRACER if it is set when using old rhel6 kernels. -Sridhar >> 2010/11/23 Michael S. Tsirkin: >>> On Tue, Nov 23, 2010 at 10:13:43AM +0800, lidong chen wrote: >>>> I test the performance between per-vhost kthread disable and enable. >>>> >>>> Test method: >>>> Send the same traffic load between per-vhost kthread disable and >>>> enable, and compare the cpu rate of host os. >>>> I run five vm on kvm, each of them have five nic. >>>> the vhost version which per-vhost kthread disable we used is rhel6 >>>> beta 2(2.6.32.60). >>>> the vhost version which per-vhost kthread enable we used is rhel6 (2.6.32-71). >>> At this point, I'd suggest testing vhost-net on the upstream kernel, >>> not on rhel kernels. The change that introduced per-device threads is: >>> c23f3445e68e1db0e74099f264bc5ff5d55ebdeb >>> >>>> Test result: >>>> with per-vhost kthread disable, the cpu rate of host os is 110%. >>>> with per-vhost kthread enable, the cpu rate of host os is 130%. >>> Is CONFIG_SCHED_DEBUG set? We are stressing the scheduler a lot with >>> vhost-net. >>> >>>> In 2.6.32.60,the whole system only have a kthread. >>>> [root@rhel6-kvm1 ~]# ps -ef | grep vhost >>>> root 973 2 0 Nov22 ? 00:00:00 [vhost] >>>> >>>> In 2.6.32.71,the whole system have 25 kthread. >>>> [root@kvm-4slot ~]# ps -ef | grep vhost- >>>> root 12896 2 0 10:26 ? 00:00:00 [vhost-12842] >>>> root 12897 2 0 10:26 ? 00:00:00 [vhost-12842] >>>> root 12898 2 0 10:26 ? 00:00:00 [vhost-12842] >>>> root 12899 2 0 10:26 ? 00:00:00 [vhost-12842] >>>> root 12900 2 0 10:26 ? 00:00:00 [vhost-12842] >>>> >>>> root 13022 2 0 10:26 ? 00:00:00 [vhost-12981] >>>> root 13023 2 0 10:26 ? 00:00:00 [vhost-12981] >>>> root 13024 2 0 10:26 ? 00:00:00 [vhost-12981] >>>> root 13025 2 0 10:26 ? 00:00:00 [vhost-12981] >>>> root 13026 2 0 10:26 ? 00:00:00 [vhost-12981] >>>> >>>> root 13146 2 0 10:26 ? 00:00:00 [vhost-13088] >>>> root 13147 2 0 10:26 ? 00:00:00 [vhost-13088] >>>> root 13148 2 0 10:26 ? 00:00:00 [vhost-13088] >>>> root 13149 2 0 10:26 ? 00:00:00 [vhost-13088] >>>> root 13150 2 0 10:26 ? 00:00:00 [vhost-13088] >>>> ... >>>> >>>> Code difference: >>>> In 2.6.32.60,in function vhost_init, create the kthread for vhost. >>>> vhost_workqueue = create_singlethread_workqueue("vhost"); >>>> >>>> In 2.6.32.71,in function vhost_dev_set_owner, create the kthread for >>>> each nic interface. >>>> dev->wq = create_singlethread_workqueue(vhost_name); >>>> >>>> Conclusion: >>>> with per-vhost kthread enable, the system can more throughput. >>>> but deal the same traffic load with per-vhost kthread enable, it waste >>>> more cpu resource. >>>> >>>> In my application scene, the cpu resource is more important, and one >>>> kthread for deal with traffic load is enough. >>>> >>>> So i think we should add a param to control this. >>>> for the CPU-bound system, this param disable per-vhost kthread. >>>> for the I/O-bound system, this param enable per-vhost kthread. >>>> the default value of this param is enable. >>>> >>>> If my opinion is right, i will give a patch for this. >>> Let's try to figure out what the issue is, first. >>> >>> -- >>> MST >>>