Re: Hitting 29 NIC limit

[Markus Armbruster <armbru@redhat.com>]

Anthony Liguori <anthony@codemonkey.ws> writes:

> On 10/14/2010 07:36 AM, Markus Armbruster wrote:
>> Avi Kivity<avi@redhat.com>  writes:
>>
>>    
>>>   On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>>>      
>>>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>>        
>>>>> Hi,
>>>>>
>>>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>>>> after 29
>>>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>>>> limit on a single bus- 3 are already taken up)
>>>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>>>> more, but ofcource that wont work.
>>>>> Is there any way to go beyond 29 NICs the legacy way?  What is the
>>>>> maximum
>>>>> that can be supported by the qdev mothod?
>>>>>          
>>>> I got up to 104 without trying very hard using the following script:
>>>>
>>>> args=""
>>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>>> for fn in 0 1 2 3 4 5 6 7; do
>>>>      args="$args -netdev user,id=eth${slot}_${fn}"
>>>>      args="$args -device
>>>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>>>> done
>>>> done
>>>>
>>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>>> -enable-kvm
>>>>
>>>> The key is to make the virtio-net devices multifunction and to fill
>>>> out all 8 functions for each slot.
>>>>        
>> I'm amazed that works.  Can't see how creating another qdev in the same
>> slot makes a proper multifunction device.
>>    
>
> multifunction=on sets the multifunction bit for the PCI device.  Then
> it's a matter of setting the address to be a specific function.

I missed that bit, or perhaps forgot it already.  Neat!

Unfortunately, it's not yet good enough for hot plug.  We need to plug
the complete device in one go, not function by function.  This is harder
than hot unplug, where we can just add a command "unplug this PCI
device", then device_del its functions at our leisure.  The symmetric
way to plug would be device_add the functions, then plug the device.

Backward compatibility headache: current device_add/del plug/unplug
automatically.

> Our default platform devices are actually multifunction.

Yes, but they're built "by hand".

[...]