Re: performance of virtual functions compared to virtio

[David Ahern <dsahern@gmail.com>]

On 04/25/11 13:07, David Ahern wrote:
>>> same host, host-VM:
>>>   A-C, using VF      488           1085 (seen as high as 1280's)
>>>   A-C, virtio        150           4282
>>
>> We know virtio has a shorter path for this test.
> 
> No complaints about the throughput numbers; the latency is the problem.

rx-usecs is the magical parameter. It defaults to 3 for both the igb and
igbvf drivers which is the 'magic' performance number -- i.e., the
drivers dynamically adapt to the packet rate.

Setting it to 10 in the *VM only* (lowest limit controlled by
IGBVF_MIN_ITR_USECS) dramatically lowers latency with little-to-no
impact to throughput (ie., mostly within the +-10% variation I see
between netperf runs with system defaults everywhere).

Latency in usecs:
                         default    rx-usec=10
host-host                  97          105
same host, host-VM        488          158
cross host, host-VM       488          181
cross host, VM-VM         488          255

Changing the default in the host for the physical function kills
throughput with no impact to latency.

I'd still like to know why 100 usec is the baseline for even
host-to-host packets.

David