From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Ahern Subject: Re: performance of virtual functions compared to virtio Date: Mon, 02 May 2011 12:58:56 -0600 Message-ID: <4DBEFEF0.6020604@gmail.com> References: <4DAF8EF0.8010203@gmail.com> <1303353349.3110.181.camel@x201> <4DB5B1C4.4000602@gmail.com> <1303755193.3431.50.camel@x201> <4DB5C65C.20306@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: KVM mailing list To: Alex Williamson Return-path: Received: from mail-pz0-f46.google.com ([209.85.210.46]:43038 "EHLO mail-pz0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753410Ab1EBS7B (ORCPT ); Mon, 2 May 2011 14:59:01 -0400 Received: by pzk9 with SMTP id 9so3159477pzk.19 for ; Mon, 02 May 2011 11:59:00 -0700 (PDT) In-Reply-To: <4DB5C65C.20306@gmail.com> Sender: kvm-owner@vger.kernel.org List-ID: On 04/25/11 13:07, David Ahern wrote: >>> same host, host-VM: >>> A-C, using VF 488 1085 (seen as high as 1280's) >>> A-C, virtio 150 4282 >> >> We know virtio has a shorter path for this test. > > No complaints about the throughput numbers; the latency is the problem. rx-usecs is the magical parameter. It defaults to 3 for both the igb and igbvf drivers which is the 'magic' performance number -- i.e., the drivers dynamically adapt to the packet rate. Setting it to 10 in the *VM only* (lowest limit controlled by IGBVF_MIN_ITR_USECS) dramatically lowers latency with little-to-no impact to throughput (ie., mostly within the +-10% variation I see between netperf runs with system defaults everywhere). Latency in usecs: default rx-usec=10 host-host 97 105 same host, host-VM 488 158 cross host, host-VM 488 181 cross host, VM-VM 488 255 Changing the default in the host for the physical function kills throughput with no impact to latency. I'd still like to know why 100 usec is the baseline for even host-to-host packets. David