Re: SR-IOV and KVM?

[David Ahern <dsahern@gmail.com>]

On 06/27/2011 08:40 AM, Avi Kivity wrote:
> On 06/27/2011 05:15 PM, Chris Friesen wrote:
>>
>> Maybe it's just a documentation thing.  I'm just getting into KVM and
>> trying to figure everything out, and I don't see a step-by-step guide
>> to configuring a host/guest system with either SR-IOV or Intel's VMDq.
> 
> You can use virt-manager; it has a GUI for that.  Once you're
> comfortable with it you can drop down to virsh command line and from
> there to raw qemu command line / monitor.

To some virt-manager is more painful than the command line is not too
painful. I have a set of home grown scripts. This is the code I added
for PCI devices:

function claim_pci_dev
{
        local pciid=$1
        local domain=0000   # TO-DO: determine this automatically
        local id

        [ -z "$pciid" ] && return 1

        local drv=$(lspci -s ${pciid} -k | awk '$0 ~ /Kernel driver in
use:/ {print $NF}')
        [ "$drv" = "pci-stub" ] && return 0

        id=$(lspci -n | awk  '{if ($1 == "'${pciid}'") {print $3}}')
        id=${id/:/ }
        if [ -z "$id" ]
        then
                err "failed to find vendor-product id for PCI id \"$pciid\""
                return 1
        fi

        echo "$id" > /sys/bus/pci/drivers/pci-stub/new_id
        if [ $? -ne 0 ]
        then
                err "Failed to tell pci-stub about id \"$id\""
                return 1
        fi

        echo "${domain}:${pciid}" >
/sys/bus/pci/devices/${domain}:${pciid}/driver/unbind
        if [ $? -ne 0 ]
        then
                err "Failed to unbind PCI device \"${domain}:${pciid}\""
                return 1
        fi

        # pci-stub claims device so it can be assigned guest
        echo "${domain}:${pciid}" > /sys/bus/pci/drivers/pci-stub/bind
        if [ $? -ne 0 ]
        then
                err "Failed to bind pci-stub to device
\"${domain}:${pciid}\""
                return 1
        fi

        return 0
}

e.g., claim_pci_dev "06:10.0"

PF-VF correlations. Example using eth2
INTFC=eth2

# pci id to VF correlation:
DEVPATH=$(readlink -f /sys/class/net/${INTFC})
ls -l ${DEVPATH}/device/virtfn*

# map a virtual function to a netdevice - in this case find the PCI id
for VF 0
VFN=0
VFID=$(readlink  ${DEVPATH}/device/virtfn${VFN} | sed -e 's,^../,,')

--> VFID is the PCI device ID for VF 0 on netdevice eth2

qemu-kvm command line addition is:
-device pci-assign,host=${PCIID}


> 
>>
>> For SR-IOV I assume I need to configure the PF in the host, enable
>> some number of VFs, pass through a VF to each guest and load the vf
>> driver. Is that right?
> 
> I think all that is needed is to prevent the host from binding to the
> guest - not sure.

See above.

> 
>> Do I need to configure a software bridge to allow the guests to talk
>> to each other? 
> 
> S/R-IOV networking is beyond my knowledge - I think it depends on your
> switch, sometimes it will loop back packets to other guests, other times
> the NIC itself will do that.  It may just work, so try it out.

PF for the NIC is attached to the network. VFs within the nic are
attached by extension. VF-VF packet flows (ie., VM to VM on the same
host) work more efficiently than hitting the wire, but not as well as
virtio connected to a bridge:
http://comments.gmane.org/gmane.comp.emulators.kvm.devel/70686

Very good descriptions in the companion guide at
http://www.intel.com/products/ethernet/resource.htm#s1=Gigabit%20Ethernet&s2=82576EB&s3=all

David

> 
>> I assume the passthrough require VT-d support in the hardware and host
>> kernel--is that correct?
>>
> 
> Yes.  Recent kernels have that.
>