Hitting 29 NIC limit

Hitting 29 NIC limit

[Anjali Kulkarni]

 
Hi,
 
Using the legacy way of starting up NICs, I am hitting a limitation after 29
NICs ie no more than 29 are detected (that's because of the 32 PCI slot
limit on a single bus- 3 are already taken up)
I had initially increased the MAX_NICS to 48, just on my tree, to get to
more, but ofcource that wont work.
Is there any way to go beyond 29 NICs the legacy way?  What is the maximum
that can be supported by the qdev mothod?

Thanks
Anjali

Re: [Qemu-devel] Hitting 29 NIC limit

[Anthony Liguori]

On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>
> Hi,
>
> Using the legacy way of starting up NICs, I am hitting a limitation after 29
> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
> limit on a single bus- 3 are already taken up)
> I had initially increased the MAX_NICS to 48, just on my tree, to get to
> more, but ofcource that wont work.
> Is there any way to go beyond 29 NICs the legacy way?  What is the maximum
> that can be supported by the qdev mothod?
>    

I got up to 104 without trying very hard using the following script:

args=""
for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
for fn in 0 1 2 3 4 5 6 7; do
     args="$args -netdev user,id=eth${slot}_${fn}"
     args="$args -device 
virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
done
done

x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args} 
-enable-kvm

The key is to make the virtio-net devices multifunction and to fill out 
all 8 functions for each slot.

Regards,

Anthony Liguori

> Thanks
> Anjali
>
>
>    

Re: [Qemu-devel] Hitting 29 NIC limit

[Avi Kivity]

  On 10/14/2010 12:54 AM, Anthony Liguori wrote:
> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>
>> Hi,
>>
>> Using the legacy way of starting up NICs, I am hitting a limitation 
>> after 29
>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>> limit on a single bus- 3 are already taken up)
>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>> more, but ofcource that wont work.
>> Is there any way to go beyond 29 NICs the legacy way?  What is the 
>> maximum
>> that can be supported by the qdev mothod?
>
> I got up to 104 without trying very hard using the following script:
>
> args=""
> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
> for fn in 0 1 2 3 4 5 6 7; do
>     args="$args -netdev user,id=eth${slot}_${fn}"
>     args="$args -device 
> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
> done
> done
>
> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args} 
> -enable-kvm
>
> The key is to make the virtio-net devices multifunction and to fill 
> out all 8 functions for each slot.

This is unlikely to work right wrt pci hotplug.  If we want to support a 
large number of interfaces, we need true multiport cards.

What's the motivation for such a huge number of interfaces?

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Hitting 29 NIC limit

[Daniel P. Berrange]

On Thu, Oct 14, 2010 at 02:07:17PM +0200, Avi Kivity wrote:
>  On 10/14/2010 12:54 AM, Anthony Liguori wrote:
> >On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
> >>
> >>Hi,
> >>
> >>Using the legacy way of starting up NICs, I am hitting a limitation 
> >>after 29
> >>NICs ie no more than 29 are detected (that's because of the 32 PCI slot
> >>limit on a single bus- 3 are already taken up)
> >>I had initially increased the MAX_NICS to 48, just on my tree, to get to
> >>more, but ofcource that wont work.
> >>Is there any way to go beyond 29 NICs the legacy way?  What is the 
> >>maximum
> >>that can be supported by the qdev mothod?
> >
> >I got up to 104 without trying very hard using the following script:
> >
> >args=""
> >for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
> >for fn in 0 1 2 3 4 5 6 7; do
> >    args="$args -netdev user,id=eth${slot}_${fn}"
> >    args="$args -device 
> >virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
> >done
> >done
> >
> >x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args} 
> >-enable-kvm
> >
> >The key is to make the virtio-net devices multifunction and to fill 
> >out all 8 functions for each slot.
> 
> This is unlikely to work right wrt pci hotplug.  If we want to support a 
> large number of interfaces, we need true multiport cards.

Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
any type of device we emulate.

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Hitting 29 NIC limit

[Markus Armbruster]

Avi Kivity <avi@redhat.com> writes:

>  On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>
>>> Hi,
>>>
>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>> after 29
>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>> limit on a single bus- 3 are already taken up)
>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>> more, but ofcource that wont work.
>>> Is there any way to go beyond 29 NICs the legacy way?  What is the
>>> maximum
>>> that can be supported by the qdev mothod?
>>
>> I got up to 104 without trying very hard using the following script:
>>
>> args=""
>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>> for fn in 0 1 2 3 4 5 6 7; do
>>     args="$args -netdev user,id=eth${slot}_${fn}"
>>     args="$args -device
>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>> done
>> done
>>
>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>> -enable-kvm
>>
>> The key is to make the virtio-net devices multifunction and to fill
>> out all 8 functions for each slot.

I'm amazed that works.  Can't see how creating another qdev in the same
slot makes a proper multifunction device.

> This is unlikely to work right wrt pci hotplug.  If we want to support
> a large number of interfaces, we need true multiport cards.

Indeed.  As far as I know, we can't hot plug multifunction PCI devices.

> What's the motivation for such a huge number of interfaces?

Re: [Qemu-devel] Hitting 29 NIC limit

[Anthony Liguori]

On 10/14/2010 07:07 AM, Avi Kivity wrote:
>  On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>
>>> Hi,
>>>
>>> Using the legacy way of starting up NICs, I am hitting a limitation 
>>> after 29
>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>> limit on a single bus- 3 are already taken up)
>>> I had initially increased the MAX_NICS to 48, just on my tree, to 
>>> get to
>>> more, but ofcource that wont work.
>>> Is there any way to go beyond 29 NICs the legacy way?  What is the 
>>> maximum
>>> that can be supported by the qdev mothod?
>>
>> I got up to 104 without trying very hard using the following script:
>>
>> args=""
>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>> for fn in 0 1 2 3 4 5 6 7; do
>>     args="$args -netdev user,id=eth${slot}_${fn}"
>>     args="$args -device 
>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile=" 
>>
>> done
>> done
>>
>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args} 
>> -enable-kvm
>>
>> The key is to make the virtio-net devices multifunction and to fill 
>> out all 8 functions for each slot.
>
> This is unlikely to work right wrt pci hotplug.

Yes.  Our hotplug design is based on devices..  This is wrong, it should 
be based on bus-level concepts (like PCI slots).

> If we want to support a large number of interfaces, we need true 
> multiport cards.

This magic here creates a multiport virtio-net card so I'm not really 
sure what you're suggesting.  It would certainly be nice to make this 
all more user friendly (and make hotplug work).

Regards,

Anthony Liguori

> What's the motivation for such a huge number of interfaces?

Re: [Qemu-devel] Hitting 29 NIC limit

[Anthony Liguori]

On 10/14/2010 07:10 AM, Daniel P. Berrange wrote:
> On Thu, Oct 14, 2010 at 02:07:17PM +0200, Avi Kivity wrote:
>    
>>   On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>>      
>>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>        
>>>> Hi,
>>>>
>>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>>> after 29
>>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>>> limit on a single bus- 3 are already taken up)
>>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>>> more, but ofcource that wont work.
>>>> Is there any way to go beyond 29 NICs the legacy way?  What is the
>>>> maximum
>>>> that can be supported by the qdev mothod?
>>>>          
>>> I got up to 104 without trying very hard using the following script:
>>>
>>> args=""
>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>> for fn in 0 1 2 3 4 5 6 7; do
>>>     args="$args -netdev user,id=eth${slot}_${fn}"
>>>     args="$args -device
>>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>>> done
>>> done
>>>
>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>> -enable-kvm
>>>
>>> The key is to make the virtio-net devices multifunction and to fill
>>> out all 8 functions for each slot.
>>>        
>> This is unlikely to work right wrt pci hotplug.  If we want to support a
>> large number of interfaces, we need true multiport cards.
>>      
> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
> any type of device we emulate.
>    

I've always been sceptical of this.  When physical systems have a large 
number of NICs, it's via multiple functions, not a bunch of PCI bridges.

With just a handful of 8-port NICs, you can exceed the current 
slot-based limit on physical hardware.  It's not an extremely common 
configuration but it does exist.

BTW, I don't think it's possible to hot-add physical functions.  I 
believe I know of a card that supports dynamic add of physical functions 
(pre-dating SR-IOV).

Regards,

Anthony Liguori

> Daniel
>    

Re: [Qemu-devel] Hitting 29 NIC limit

[Anthony Liguori]

On 10/14/2010 07:36 AM, Markus Armbruster wrote:
> Avi Kivity<avi@redhat.com>  writes:
>
>    
>>   On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>>      
>>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>        
>>>> Hi,
>>>>
>>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>>> after 29
>>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>>> limit on a single bus- 3 are already taken up)
>>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>>> more, but ofcource that wont work.
>>>> Is there any way to go beyond 29 NICs the legacy way?  What is the
>>>> maximum
>>>> that can be supported by the qdev mothod?
>>>>          
>>> I got up to 104 without trying very hard using the following script:
>>>
>>> args=""
>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>> for fn in 0 1 2 3 4 5 6 7; do
>>>      args="$args -netdev user,id=eth${slot}_${fn}"
>>>      args="$args -device
>>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>>> done
>>> done
>>>
>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>> -enable-kvm
>>>
>>> The key is to make the virtio-net devices multifunction and to fill
>>> out all 8 functions for each slot.
>>>        
> I'm amazed that works.  Can't see how creating another qdev in the same
> slot makes a proper multifunction device.
>    

multifunction=on sets the multifunction bit for the PCI device.  Then 
it's a matter of setting the address to be a specific function.

Our default platform devices are actually multifunction.

>> This is unlikely to work right wrt pci hotplug.  If we want to support
>> a large number of interfaces, we need true multiport cards.
>>      
> Indeed.  As far as I know, we can't hot plug multifunction PCI devices.
>    

Yup.

Regards,

Anthony Liguori

>> What's the motivation for such a huge number of interfaces?
>>      
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

Re: [Qemu-devel] Hitting 29 NIC limit

[Avi Kivity]

  On 10/14/2010 02:54 PM, Anthony Liguori wrote:
>>> The key is to make the virtio-net devices multifunction and to fill 
>>> out all 8 functions for each slot.
>>
>> This is unlikely to work right wrt pci hotplug.
>
>
> Yes.  Our hotplug design is based on devices..  This is wrong, it 
> should be based on bus-level concepts (like PCI slots).
>
>> If we want to support a large number of interfaces, we need true 
>> multiport cards.
>
> This magic here creates a multiport virtio-net card so I'm not really 
> sure what you're suggesting.  It would certainly be nice to make this 
> all more user friendly (and make hotplug work).
>

The big issue is to fix hotplug.

I don't see how we can make it user friendly, without making the 
ordinary case even more unfriendly.  Looks like we need yet another 
level of indirection here.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Hitting 29 NIC limit

[Anthony Liguori]

On 10/14/2010 08:23 AM, Avi Kivity wrote:
>  On 10/14/2010 02:54 PM, Anthony Liguori wrote:
>>>> The key is to make the virtio-net devices multifunction and to fill 
>>>> out all 8 functions for each slot.
>>>
>>> This is unlikely to work right wrt pci hotplug.
>>
>>
>> Yes.  Our hotplug design is based on devices..  This is wrong, it 
>> should be based on bus-level concepts (like PCI slots).
>>
>>> If we want to support a large number of interfaces, we need true 
>>> multiport cards.
>>
>> This magic here creates a multiport virtio-net card so I'm not really 
>> sure what you're suggesting.  It would certainly be nice to make this 
>> all more user friendly (and make hotplug work).
>>
>
> The big issue is to fix hotplug.

Yes, but this is entirely independent of multifunction devices.

Today we shoe-horn hot remove into device_del.  Instead, we should have 
explicit bus-level interfaces for hot remove.

Regards,

Anthony Liguori

> I don't see how we can make it user friendly, without making the 
> ordinary case even more unfriendly.  Looks like we need yet another 
> level of indirection here.
>

Re: [Qemu-devel] Hitting 29 NIC limit

[Avi Kivity]

  On 10/14/2010 04:11 PM, Anthony Liguori wrote:
> On 10/14/2010 08:23 AM, Avi Kivity wrote:
>>  On 10/14/2010 02:54 PM, Anthony Liguori wrote:
>>>>> The key is to make the virtio-net devices multifunction and to 
>>>>> fill out all 8 functions for each slot.
>>>>
>>>> This is unlikely to work right wrt pci hotplug.
>>>
>>>
>>> Yes.  Our hotplug design is based on devices..  This is wrong, it 
>>> should be based on bus-level concepts (like PCI slots).
>>>
>>>> If we want to support a large number of interfaces, we need true 
>>>> multiport cards.
>>>
>>> This magic here creates a multiport virtio-net card so I'm not 
>>> really sure what you're suggesting.  It would certainly be nice to 
>>> make this all more user friendly (and make hotplug work).
>>>
>>
>> The big issue is to fix hotplug.
>
> Yes, but this is entirely independent of multifunction devices.
>
> Today we shoe-horn hot remove into device_del.  Instead, we should 
> have explicit bus-level interfaces for hot remove.

I'm not saying multiplug is not the right way to approach this (it is).  
The only concern is to get hotplug right.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Hitting 29 NIC limit

[Richard W.M. Jones]

On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
> any type of device we emulate.

Break the 29/30/31 virtio-blk limit ... please!

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

Re: [Qemu-devel] Hitting 29 NIC limit

[Anthony Liguori]

On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>    
>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>> any type of device we emulate.
>>      
> Break the 29/30/31 virtio-blk limit ... please!
>    

It was broken ages ago:

anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
422 /proc/partitions
  251     1618          1 vdcx2
  251     1621     489951 vdcx5
  251     1632   10485760 vdcy
  251     1633    9992398 vdcy1
  251     1634          1 vdcy2
  251     1637     489951 vdcy5
  251     1648   10485760 vdcz
  251     1649    9992398 vdcz1
  251     1650          1 vdcz2
  251     1653     489951 vdcz5

This is what makes qdev so useful.

args=""
for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
for fn in 0 1 2 3 4 5 6 7; do
     args="$args -drive 
file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}"
     args="$args -device 
virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
done
done

x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args} 
-enable-kvm -serial stdio

Regards,

Anthony Liguori

> Rich.
>
>    

Re: [Qemu-devel] Hitting 29 NIC limit

[Anjali Kulkarni]

Can you send me pointers to the qdev documentation? How can I use it? Will
it allow us to scale above the 32 PCI limit?

Anjali


On 10/14/10 2:57 PM, "Anthony Liguori" <anthony@codemonkey.ws> wrote:

> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
>> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>>    
>>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>>> any type of device we emulate.
>>>      
>> Break the 29/30/31 virtio-blk limit ... please!
>>    
> 
> It was broken ages ago:
> 
> anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
> 422 /proc/partitions
>   251     1618          1 vdcx2
>   251     1621     489951 vdcx5
>   251     1632   10485760 vdcy
>   251     1633    9992398 vdcy1
>   251     1634          1 vdcy2
>   251     1637     489951 vdcy5
>   251     1648   10485760 vdcz
>   251     1649    9992398 vdcz1
>   251     1650          1 vdcz2
>   251     1653     489951 vdcz5
> 
> This is what makes qdev so useful.
> 
> args=""
> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
> for fn in 0 1 2 3 4 5 6 7; do
>      args="$args -drive
> file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}"
>      args="$args -device
> virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
> done
> done
> 
> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
> -enable-kvm -serial stdio
> 
> Regards,
> 
> Anthony Liguori
> 
>> Rich.
>> 
>>    
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Qemu-devel] Hitting 29 NIC limit

[Richard W.M. Jones]

On Thu, Oct 14, 2010 at 04:57:36PM -0500, Anthony Liguori wrote:
> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
> >On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
> >>Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
> >>any type of device we emulate.
> >Break the 29/30/31 virtio-blk limit ... please!
> 
> It was broken ages ago:
[...]

Excellent news indeeed.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

Re: [Qemu-devel] Hitting 29 NIC limit

[Anthony Liguori]

On 10/14/2010 05:00 PM, Anjali Kulkarni wrote:
> Can you send me pointers to the qdev documentation? How can I use it? Will
> it allow us to scale above the 32 PCI limit?
>    

It's all below.  You just have to create a PCI device and mark the 
multifunction flag to on and then assign it a PCI address that includes 
a function number.  Then you can pack 8 virtio PCI devices into a single 
slot.

Regards,

Anthony Liguori

> Anjali
>
>
> On 10/14/10 2:57 PM, "Anthony Liguori"<anthony@codemonkey.ws>  wrote:
>
>    
>> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
>>      
>>> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>>>
>>>        
>>>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>>>> any type of device we emulate.
>>>>
>>>>          
>>> Break the 29/30/31 virtio-blk limit ... please!
>>>
>>>        
>> It was broken ages ago:
>>
>> anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
>> 422 /proc/partitions
>>    251     1618          1 vdcx2
>>    251     1621     489951 vdcx5
>>    251     1632   10485760 vdcy
>>    251     1633    9992398 vdcy1
>>    251     1634          1 vdcy2
>>    251     1637     489951 vdcy5
>>    251     1648   10485760 vdcz
>>    251     1649    9992398 vdcz1
>>    251     1650          1 vdcz2
>>    251     1653     489951 vdcz5
>>
>> This is what makes qdev so useful.
>>
>> args=""
>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>> for fn in 0 1 2 3 4 5 6 7; do
>>       args="$args -drive
>> file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}"
>>       args="$args -device
>> virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
>> done
>> done
>>
>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>> -enable-kvm -serial stdio
>>
>> Regards,
>>
>> Anthony Liguori
>>
>>      
>>> Rich.
>>>
>>>
>>>        
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>      
>    

Re: [Qemu-devel] Hitting 29 NIC limit

[Anjali Kulkarni]

Thanks. Does this work for e1000 as well?
Also, does it support pci hotplug?

Anjali

On 10/14/10 3:09 PM, "Anthony Liguori" <anthony@codemonkey.ws> wrote:

> On 10/14/2010 05:00 PM, Anjali Kulkarni wrote:
>> Can you send me pointers to the qdev documentation? How can I use it? Will
>> it allow us to scale above the 32 PCI limit?
>>    
> 
> It's all below.  You just have to create a PCI device and mark the
> multifunction flag to on and then assign it a PCI address that includes
> a function number.  Then you can pack 8 virtio PCI devices into a single
> slot.
> 
> Regards,
> 
> Anthony Liguori
> 
>> Anjali
>> 
>> 
>> On 10/14/10 2:57 PM, "Anthony Liguori"<anthony@codemonkey.ws>  wrote:
>> 
>>    
>>> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
>>>      
>>>> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>>>> 
>>>>        
>>>>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>>>>> any type of device we emulate.
>>>>> 
>>>>>          
>>>> Break the 29/30/31 virtio-blk limit ... please!
>>>> 
>>>>        
>>> It was broken ages ago:
>>> 
>>> anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
>>> 422 /proc/partitions
>>>    251     1618          1 vdcx2
>>>    251     1621     489951 vdcx5
>>>    251     1632   10485760 vdcy
>>>    251     1633    9992398 vdcy1
>>>    251     1634          1 vdcy2
>>>    251     1637     489951 vdcy5
>>>    251     1648   10485760 vdcz
>>>    251     1649    9992398 vdcz1
>>>    251     1650          1 vdcz2
>>>    251     1653     489951 vdcz5
>>> 
>>> This is what makes qdev so useful.
>>> 
>>> args=""
>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>> for fn in 0 1 2 3 4 5 6 7; do
>>>       args="$args -drive
>>> 
file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}>>>
"
>>>       args="$args -device
>>> virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
>>> done
>>> done
>>> 
>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>> -enable-kvm -serial stdio
>>> 
>>> Regards,
>>> 
>>> Anthony Liguori
>>> 
>>>      
>>>> Rich.
>>>> 
>>>> 
>>>>        
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>      
>>    
> 

Re: [Qemu-devel] Hitting 29 NIC limit

[Anthony Liguori]

On 10/14/2010 05:12 PM, Anjali Kulkarni wrote:
> Thanks. Does this work for e1000 as well?
>    

Haven't tried.  I don't know how various e1000 drivers would react.

> Also, does it support pci hotplug?
>    

No, but that's fixable down the road.

Regards,

Anthony Liguori

> Anjali
>
> On 10/14/10 3:09 PM, "Anthony Liguori"<anthony@codemonkey.ws>  wrote:
>
>    
>> On 10/14/2010 05:00 PM, Anjali Kulkarni wrote:
>>      
>>> Can you send me pointers to the qdev documentation? How can I use it? Will
>>> it allow us to scale above the 32 PCI limit?
>>>
>>>        
>> It's all below.  You just have to create a PCI device and mark the
>> multifunction flag to on and then assign it a PCI address that includes
>> a function number.  Then you can pack 8 virtio PCI devices into a single
>> slot.
>>
>> Regards,
>>
>> Anthony Liguori
>>
>>      
>>> Anjali
>>>
>>>
>>> On 10/14/10 2:57 PM, "Anthony Liguori"<anthony@codemonkey.ws>   wrote:
>>>
>>>
>>>        
>>>> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
>>>>
>>>>          
>>>>> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>>>>>
>>>>>
>>>>>            
>>>>>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>>>>>> any type of device we emulate.
>>>>>>
>>>>>>
>>>>>>              
>>>>> Break the 29/30/31 virtio-blk limit ... please!
>>>>>
>>>>>
>>>>>            
>>>> It was broken ages ago:
>>>>
>>>> anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
>>>> 422 /proc/partitions
>>>>     251     1618          1 vdcx2
>>>>     251     1621     489951 vdcx5
>>>>     251     1632   10485760 vdcy
>>>>     251     1633    9992398 vdcy1
>>>>     251     1634          1 vdcy2
>>>>     251     1637     489951 vdcy5
>>>>     251     1648   10485760 vdcz
>>>>     251     1649    9992398 vdcz1
>>>>     251     1650          1 vdcz2
>>>>     251     1653     489951 vdcz5
>>>>
>>>> This is what makes qdev so useful.
>>>>
>>>> args=""
>>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>>> for fn in 0 1 2 3 4 5 6 7; do
>>>>        args="$args -drive
>>>>
>>>>          
> file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}>>>
> "
>    
>>>>        args="$args -device
>>>> virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
>>>> done
>>>> done
>>>>
>>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>>> -enable-kvm -serial stdio
>>>>
>>>> Regards,
>>>>
>>>> Anthony Liguori
>>>>
>>>>
>>>>          
>>>>> Rich.
>>>>>
>>>>>
>>>>>
>>>>>            
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>>          
>>>
>>>        
>>      
>    

Re: Hitting 29 NIC limit

[Markus Armbruster]

Anthony Liguori <anthony@codemonkey.ws> writes:

> On 10/14/2010 07:36 AM, Markus Armbruster wrote:
>> Avi Kivity<avi@redhat.com>  writes:
>>
>>    
>>>   On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>>>      
>>>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>>        
>>>>> Hi,
>>>>>
>>>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>>>> after 29
>>>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>>>> limit on a single bus- 3 are already taken up)
>>>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>>>> more, but ofcource that wont work.
>>>>> Is there any way to go beyond 29 NICs the legacy way?  What is the
>>>>> maximum
>>>>> that can be supported by the qdev mothod?
>>>>>          
>>>> I got up to 104 without trying very hard using the following script:
>>>>
>>>> args=""
>>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>>> for fn in 0 1 2 3 4 5 6 7; do
>>>>      args="$args -netdev user,id=eth${slot}_${fn}"
>>>>      args="$args -device
>>>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>>>> done
>>>> done
>>>>
>>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>>> -enable-kvm
>>>>
>>>> The key is to make the virtio-net devices multifunction and to fill
>>>> out all 8 functions for each slot.
>>>>        
>> I'm amazed that works.  Can't see how creating another qdev in the same
>> slot makes a proper multifunction device.
>>    
>
> multifunction=on sets the multifunction bit for the PCI device.  Then
> it's a matter of setting the address to be a specific function.

I missed that bit, or perhaps forgot it already.  Neat!

Unfortunately, it's not yet good enough for hot plug.  We need to plug
the complete device in one go, not function by function.  This is harder
than hot unplug, where we can just add a command "unplug this PCI
device", then device_del its functions at our leisure.  The symmetric
way to plug would be device_add the functions, then plug the device.

Backward compatibility headache: current device_add/del plug/unplug
automatically.

> Our default platform devices are actually multifunction.

Yes, but they're built "by hand".

[...]

Passing in additional info to guest OS and e1000 test suite?

[Anjali Kulkarni]

Hi,

- If I want to pass in additional arguments to the guest OS while booting(in
particular which slot I want to map a nic to) - is there any way to do it?
Some kind of configuration file that I can pass in would also be ok for me.
- Is there a KVM/Qemu/e1000 test suite that is already available and I can
us?

Thanks
Anjali

Re: Passing in additional info to guest OS and e1000 test suite?

[Anthony Liguori]

On 10/15/2010 01:49 PM, Anjali Kulkarni wrote:
> Hi,
>
> - If I want to pass in additional arguments to the guest OS while booting(in
> particular which slot I want to map a nic to) - is there any way to do it?
> Some kind of configuration file that I can pass in would also be ok for me.
> - Is there a KVM/Qemu/e1000 test suite that is already available and I can
> us?
>    

I'm a little confused about what you're really asking.  Can you give a 
more illustrated example of what you're asking?  Are you using libvirt?

> Thanks
> Anjali
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

Re: [Qemu-devel] Hitting 29 NIC limit

[H. Peter Anvin]

On 10/14/2010 05:57 AM, Anthony Liguori wrote:
> 
> I've always been sceptical of this.  When physical systems have a large 
> number of NICs, it's via multiple functions, not a bunch of PCI bridges.
> 

Actually a lot of multiport PCI cards are in fact single or dual NICs
behind PCI bridges.

	-hpa

Re: [Qemu-devel] Hitting 29 NIC limit (+Intel VT-c)

[linux_kvm]

On Thu, 14 Oct 2010 14:07 +0200, "Avi Kivity" <avi@redhat.com> wrote:
>   On 10/14/2010 12:54 AM, Anthony Liguori wrote:
> > On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:

> What's the motivation for such a huge number of interfaces?

Ultimately to bring multiple 10Gb bonds into a Vyatta guest.

---

> BTW, I don't think it's possible to hot-add physical functions.  I 
> believe I know of a card that supports dynamic add of physical functions 
> (pre-dating SR-IOV)

I don't know what you're talking about, but it seems you have a better
handle than I on this VT-c stuff, so perhaps misguidedly I'll direct my
next question to you.

Is additional configuration required to make use of SR-IOV & VTq?
I don't immediateley understand how the queueing knows who is who in the
absense of eth.vlan- or if I need to for that matter.

My hope is that this is something like plug n play as long as kernel,
host & driver versions are >foo, but I haven't yet found documentation
to confirm it.

For the sake of future queries, I've come across these references so
far:

http://download.intel.com/design/network/applnots/321211.pdf
http://www.linux-kvm.org/wiki/images/6/6a/KvmForum2008%24kdf2008_7.pdf
http://www.mail-archive.com/kvm@vger.kernel.org/msg27860.html
http://www.mail-archive.com/kvm@vger.kernel.org/msg22721.html
http://thread.gmane.org/gmane.linux.kernel.mm/38508
http://ark.intel.com/Product.aspx?id=36918

Re: Hitting 29 NIC limit

[Michael S. Tsirkin]

On Thu, Oct 14, 2010 at 02:07:17PM +0200, Avi Kivity wrote:
>  On 10/14/2010 12:54 AM, Anthony Liguori wrote:
> >On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
> >>
> >>Hi,
> >>
> >>Using the legacy way of starting up NICs, I am hitting a
> >>limitation after 29
> >>NICs ie no more than 29 are detected (that's because of the 32 PCI slot
> >>limit on a single bus- 3 are already taken up)
> >>I had initially increased the MAX_NICS to 48, just on my tree, to get to
> >>more, but ofcource that wont work.
> >>Is there any way to go beyond 29 NICs the legacy way?  What is
> >>the maximum
> >>that can be supported by the qdev mothod?
> >
> >I got up to 104 without trying very hard using the following script:
> >
> >args=""
> >for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
> >for fn in 0 1 2 3 4 5 6 7; do
> >    args="$args -netdev user,id=eth${slot}_${fn}"
> >    args="$args -device virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
> >done
> >done
> >
> >x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
> >-enable-kvm
> >
> >The key is to make the virtio-net devices multifunction and to
> >fill out all 8 functions for each slot.
> 
> This is unlikely to work right wrt pci hotplug.  If we want to
> support a large number of interfaces, we need true multiport cards.

Bridge support seems to be working mostly fine here.
That will let you go up to 256 devices without multifunction.

-- 
MST

Re: Hitting 29 NIC limit

[Michael S. Tsirkin]

On Thu, Oct 14, 2010 at 05:17:36PM -0500, Anthony Liguori wrote:
> On 10/14/2010 05:12 PM, Anjali Kulkarni wrote:
> >Thanks. Does this work for e1000 as well?
> 
> Haven't tried.  I don't know how various e1000 drivers would react.
> 
> >Also, does it support pci hotplug?
> 
> No, but that's fixable down the road.
> 
> Regards,
> 
> Anthony Liguori

Probably not.

Re: Hitting 29 NIC limit

[Anthony Liguori]

On 11/16/2010 11:23 AM, Michael S. Tsirkin wrote:
> On Thu, Oct 14, 2010 at 05:17:36PM -0500, Anthony Liguori wrote:
>    
>> On 10/14/2010 05:12 PM, Anjali Kulkarni wrote:
>>      
>>> Thanks. Does this work for e1000 as well?
>>>        
>> Haven't tried.  I don't know how various e1000 drivers would react.
>>
>>      
>>> Also, does it support pci hotplug?
>>>        
>> No, but that's fixable down the road.
>>
>> Regards,
>>
>> Anthony Liguori
>>      
> Probably not.
>    

FWIW, by fixable, I meant, we could add the ability to hot plug a 
multi-port NIC device.  For instance, we could have -device 
virtio-net-pci-x8 which would be an 8-port virtio-net card.

Regards,

Anthony Liguori