[Qemu-devel] Re: 8 NIC limit

[Qemu-devel] Re: 8 NIC limit

[Dustin Kirkland]

On Tue, Oct 5, 2010 at 7:48 AM,  <linux_kvm@proinbox.com> wrote:
> Hello list:
>
> I'm working on a project that calls for the creation of a firewall in
> KVM.
> While adding a 20-interface trunk of virtio adapters to bring in a dual
> 10GB bond, I've discovered an 8 NIC limit in QEMU.
>
> I found the following thread in the list archives detailing a similar
> problem:
> http://kerneltrap.org/mailarchive/linux-kvm/2009/1/29/4848304
>
> It includes a patch for the file qemu/net.h to allow 24 NICs:
> https://bugs.launchpad.net/ubuntu/+source/qemu-kvm">qemu-kvm/+bug/595873/+attachment/1429544/+files/max_nics.patch
>
> In my case I want to attach 29, and have simply changed line 8 to 30
> from 24.
>
> This will be the first patch I've ever had to do, and so far my internet
> search yields results that don't seem to apply.
>
> Would someone like to recommend a pertinent tutorial?

Hi there,

I commented on the original bug in Launchpad.  We're willing and able
to carry the patch against qemu-kvm in Ubuntu, I just asked that the
reporter at least submit the patch upstream for discussion.  I don't
see where that has happened yet.  It's a trivial patch to submit.
Please note in that bug a pointer to the mailing list thread, if you
start one.

To your specific question, different communities have different
requirements on patch submission, so you do need to consult each
community.  A good place to start might be the
Documentation/SubmittingPatches how-to in the kernel tree:
 * http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob_plain;f=Documentation/SubmittingPatches;hb=HEAD

In this case, I think you're going to want to send your patch to the
qemu-devel (on CC) mailing list (perhaps in addition to sending it
here, to the kvm list).

:-Dustin

[Qemu-devel] Re: 8 NIC limit - patch - places limit at 32

[linux_kvm]

[-- Attachment #1: Type: text/plain, Size: 99 bytes --]

Attached is a patch that allows qemu to have up to 32 NICs, without
using the qdev -device method.

[-- Attachment #2: max_nics.patch --]
[-- Type: application/octet-stream, Size: 225 bytes --]

--- a/net.h	2010-02-26 03:34:00.000000000 +0100
+++ b/net.h.my	2010-06-10 18:48:53.215317795 +0200
@@ -119,7 +119,7 @@
 
 /* NIC info */
 
-#define MAX_NICS 8
+#define MAX_NICS 32 
 enum {
 	NIC_NVECTORS_UNSPECIFIED = -1
 };

[Qemu-devel] Re: 8 NIC limit - patch - places limit at 32

[Anthony Liguori]

On 10/06/2010 12:46 AM, linux_kvm@proinbox.com wrote:
> Attached is a patch that allows qemu to have up to 32 NICs, without
> using the qdev -device method.
>    

I'd rather there be no fixed limit and we validate that when add fails 
because there isn't a TCP slot available, we do the right thing.

BTW, using -device, it should be possible to add a very high number of 
nics because you can specify the PCI address including a function.  If 
this doesn't Just Work today, we should make it work.

Regards,

Anthony Liguori

[Qemu-devel] Hitting 29 NIC limit (was Re: 8 NIC limit - patch - places limit at 32)

[Anjali Kulkarni]

Hi,

Using the legacy way of starting up NICs, I am hitting a limitation after 29
NICs ie no more than 29 are detected (that's because of the 32 PCI slot
limit on a single bus- 3 are already taken up)
I had initially increased the MAX_NICS to 48, just on my tree, to get to
more, but ofcource that wont work.
Is there any way to go beyond 29 NICs the legacy way?  What is the maximum
that can be supported by the qdev mothod?

Thanks
Anjali


On 10/5/10 10:46 PM, "linux_kvm@proinbox.com" <linux_kvm@proinbox.com>
wrote:

> Attached is a patch that allows qemu to have up to 32 NICs, without
> using the qdev -device method.

[Qemu-devel] Hitting 29 NIC limit

[Anjali Kulkarni]

 
Hi,
 
Using the legacy way of starting up NICs, I am hitting a limitation after 29
NICs ie no more than 29 are detected (that's because of the 32 PCI slot
limit on a single bus- 3 are already taken up)
I had initially increased the MAX_NICS to 48, just on my tree, to get to
more, but ofcource that wont work.
Is there any way to go beyond 29 NICs the legacy way?  What is the maximum
that can be supported by the qdev mothod?

Thanks
Anjali

Re: [Qemu-devel] Hitting 29 NIC limit

[Anthony Liguori]

On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>
> Hi,
>
> Using the legacy way of starting up NICs, I am hitting a limitation after 29
> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
> limit on a single bus- 3 are already taken up)
> I had initially increased the MAX_NICS to 48, just on my tree, to get to
> more, but ofcource that wont work.
> Is there any way to go beyond 29 NICs the legacy way?  What is the maximum
> that can be supported by the qdev mothod?
>    

I got up to 104 without trying very hard using the following script:

args=""
for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
for fn in 0 1 2 3 4 5 6 7; do
     args="$args -netdev user,id=eth${slot}_${fn}"
     args="$args -device 
virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
done
done

x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args} 
-enable-kvm

The key is to make the virtio-net devices multifunction and to fill out 
all 8 functions for each slot.

Regards,

Anthony Liguori

> Thanks
> Anjali
>
>
>    

Re: [Qemu-devel] Hitting 29 NIC limit

[Avi Kivity]

  On 10/14/2010 12:54 AM, Anthony Liguori wrote:
> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>
>> Hi,
>>
>> Using the legacy way of starting up NICs, I am hitting a limitation 
>> after 29
>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>> limit on a single bus- 3 are already taken up)
>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>> more, but ofcource that wont work.
>> Is there any way to go beyond 29 NICs the legacy way?  What is the 
>> maximum
>> that can be supported by the qdev mothod?
>
> I got up to 104 without trying very hard using the following script:
>
> args=""
> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
> for fn in 0 1 2 3 4 5 6 7; do
>     args="$args -netdev user,id=eth${slot}_${fn}"
>     args="$args -device 
> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
> done
> done
>
> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args} 
> -enable-kvm
>
> The key is to make the virtio-net devices multifunction and to fill 
> out all 8 functions for each slot.

This is unlikely to work right wrt pci hotplug.  If we want to support a 
large number of interfaces, we need true multiport cards.

What's the motivation for such a huge number of interfaces?

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Hitting 29 NIC limit

[Daniel P. Berrange]

On Thu, Oct 14, 2010 at 02:07:17PM +0200, Avi Kivity wrote:
>  On 10/14/2010 12:54 AM, Anthony Liguori wrote:
> >On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
> >>
> >>Hi,
> >>
> >>Using the legacy way of starting up NICs, I am hitting a limitation 
> >>after 29
> >>NICs ie no more than 29 are detected (that's because of the 32 PCI slot
> >>limit on a single bus- 3 are already taken up)
> >>I had initially increased the MAX_NICS to 48, just on my tree, to get to
> >>more, but ofcource that wont work.
> >>Is there any way to go beyond 29 NICs the legacy way?  What is the 
> >>maximum
> >>that can be supported by the qdev mothod?
> >
> >I got up to 104 without trying very hard using the following script:
> >
> >args=""
> >for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
> >for fn in 0 1 2 3 4 5 6 7; do
> >    args="$args -netdev user,id=eth${slot}_${fn}"
> >    args="$args -device 
> >virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
> >done
> >done
> >
> >x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args} 
> >-enable-kvm
> >
> >The key is to make the virtio-net devices multifunction and to fill 
> >out all 8 functions for each slot.
> 
> This is unlikely to work right wrt pci hotplug.  If we want to support a 
> large number of interfaces, we need true multiport cards.

Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
any type of device we emulate.

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Hitting 29 NIC limit

[Anthony Liguori]

On 10/14/2010 07:10 AM, Daniel P. Berrange wrote:
> On Thu, Oct 14, 2010 at 02:07:17PM +0200, Avi Kivity wrote:
>    
>>   On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>>      
>>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>        
>>>> Hi,
>>>>
>>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>>> after 29
>>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>>> limit on a single bus- 3 are already taken up)
>>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>>> more, but ofcource that wont work.
>>>> Is there any way to go beyond 29 NICs the legacy way?  What is the
>>>> maximum
>>>> that can be supported by the qdev mothod?
>>>>          
>>> I got up to 104 without trying very hard using the following script:
>>>
>>> args=""
>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>> for fn in 0 1 2 3 4 5 6 7; do
>>>     args="$args -netdev user,id=eth${slot}_${fn}"
>>>     args="$args -device
>>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>>> done
>>> done
>>>
>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>> -enable-kvm
>>>
>>> The key is to make the virtio-net devices multifunction and to fill
>>> out all 8 functions for each slot.
>>>        
>> This is unlikely to work right wrt pci hotplug.  If we want to support a
>> large number of interfaces, we need true multiport cards.
>>      
> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
> any type of device we emulate.
>    

I've always been sceptical of this.  When physical systems have a large 
number of NICs, it's via multiple functions, not a bunch of PCI bridges.

With just a handful of 8-port NICs, you can exceed the current 
slot-based limit on physical hardware.  It's not an extremely common 
configuration but it does exist.

BTW, I don't think it's possible to hot-add physical functions.  I 
believe I know of a card that supports dynamic add of physical functions 
(pre-dating SR-IOV).

Regards,

Anthony Liguori

> Daniel
>    

Re: [Qemu-devel] Hitting 29 NIC limit

[H. Peter Anvin]

On 10/14/2010 05:57 AM, Anthony Liguori wrote:
> 
> I've always been sceptical of this.  When physical systems have a large 
> number of NICs, it's via multiple functions, not a bunch of PCI bridges.
> 

Actually a lot of multiport PCI cards are in fact single or dual NICs
behind PCI bridges.

	-hpa

Re: [Qemu-devel] Hitting 29 NIC limit

[Richard W.M. Jones]

On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
> any type of device we emulate.

Break the 29/30/31 virtio-blk limit ... please!

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

Re: [Qemu-devel] Hitting 29 NIC limit

[Anthony Liguori]

On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>    
>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>> any type of device we emulate.
>>      
> Break the 29/30/31 virtio-blk limit ... please!
>    

It was broken ages ago:

anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
422 /proc/partitions
  251     1618          1 vdcx2
  251     1621     489951 vdcx5
  251     1632   10485760 vdcy
  251     1633    9992398 vdcy1
  251     1634          1 vdcy2
  251     1637     489951 vdcy5
  251     1648   10485760 vdcz
  251     1649    9992398 vdcz1
  251     1650          1 vdcz2
  251     1653     489951 vdcz5

This is what makes qdev so useful.

args=""
for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
for fn in 0 1 2 3 4 5 6 7; do
     args="$args -drive 
file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}"
     args="$args -device 
virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
done
done

x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args} 
-enable-kvm -serial stdio

Regards,

Anthony Liguori

> Rich.
>
>    

Re: [Qemu-devel] Hitting 29 NIC limit

[Anjali Kulkarni]

Can you send me pointers to the qdev documentation? How can I use it? Will
it allow us to scale above the 32 PCI limit?

Anjali


On 10/14/10 2:57 PM, "Anthony Liguori" <anthony@codemonkey.ws> wrote:

> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
>> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>>    
>>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>>> any type of device we emulate.
>>>      
>> Break the 29/30/31 virtio-blk limit ... please!
>>    
> 
> It was broken ages ago:
> 
> anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
> 422 /proc/partitions
>   251     1618          1 vdcx2
>   251     1621     489951 vdcx5
>   251     1632   10485760 vdcy
>   251     1633    9992398 vdcy1
>   251     1634          1 vdcy2
>   251     1637     489951 vdcy5
>   251     1648   10485760 vdcz
>   251     1649    9992398 vdcz1
>   251     1650          1 vdcz2
>   251     1653     489951 vdcz5
> 
> This is what makes qdev so useful.
> 
> args=""
> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
> for fn in 0 1 2 3 4 5 6 7; do
>      args="$args -drive
> file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}"
>      args="$args -device
> virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
> done
> done
> 
> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
> -enable-kvm -serial stdio
> 
> Regards,
> 
> Anthony Liguori
> 
>> Rich.
>> 
>>    
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Qemu-devel] Hitting 29 NIC limit

[Anthony Liguori]

On 10/14/2010 05:00 PM, Anjali Kulkarni wrote:
> Can you send me pointers to the qdev documentation? How can I use it? Will
> it allow us to scale above the 32 PCI limit?
>    

It's all below.  You just have to create a PCI device and mark the 
multifunction flag to on and then assign it a PCI address that includes 
a function number.  Then you can pack 8 virtio PCI devices into a single 
slot.

Regards,

Anthony Liguori

> Anjali
>
>
> On 10/14/10 2:57 PM, "Anthony Liguori"<anthony@codemonkey.ws>  wrote:
>
>    
>> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
>>      
>>> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>>>
>>>        
>>>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>>>> any type of device we emulate.
>>>>
>>>>          
>>> Break the 29/30/31 virtio-blk limit ... please!
>>>
>>>        
>> It was broken ages ago:
>>
>> anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
>> 422 /proc/partitions
>>    251     1618          1 vdcx2
>>    251     1621     489951 vdcx5
>>    251     1632   10485760 vdcy
>>    251     1633    9992398 vdcy1
>>    251     1634          1 vdcy2
>>    251     1637     489951 vdcy5
>>    251     1648   10485760 vdcz
>>    251     1649    9992398 vdcz1
>>    251     1650          1 vdcz2
>>    251     1653     489951 vdcz5
>>
>> This is what makes qdev so useful.
>>
>> args=""
>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>> for fn in 0 1 2 3 4 5 6 7; do
>>       args="$args -drive
>> file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}"
>>       args="$args -device
>> virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
>> done
>> done
>>
>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>> -enable-kvm -serial stdio
>>
>> Regards,
>>
>> Anthony Liguori
>>
>>      
>>> Rich.
>>>
>>>
>>>        
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>      
>    

Re: [Qemu-devel] Hitting 29 NIC limit

[Anjali Kulkarni]

Thanks. Does this work for e1000 as well?
Also, does it support pci hotplug?

Anjali

On 10/14/10 3:09 PM, "Anthony Liguori" <anthony@codemonkey.ws> wrote:

> On 10/14/2010 05:00 PM, Anjali Kulkarni wrote:
>> Can you send me pointers to the qdev documentation? How can I use it? Will
>> it allow us to scale above the 32 PCI limit?
>>    
> 
> It's all below.  You just have to create a PCI device and mark the
> multifunction flag to on and then assign it a PCI address that includes
> a function number.  Then you can pack 8 virtio PCI devices into a single
> slot.
> 
> Regards,
> 
> Anthony Liguori
> 
>> Anjali
>> 
>> 
>> On 10/14/10 2:57 PM, "Anthony Liguori"<anthony@codemonkey.ws>  wrote:
>> 
>>    
>>> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
>>>      
>>>> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>>>> 
>>>>        
>>>>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>>>>> any type of device we emulate.
>>>>> 
>>>>>          
>>>> Break the 29/30/31 virtio-blk limit ... please!
>>>> 
>>>>        
>>> It was broken ages ago:
>>> 
>>> anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
>>> 422 /proc/partitions
>>>    251     1618          1 vdcx2
>>>    251     1621     489951 vdcx5
>>>    251     1632   10485760 vdcy
>>>    251     1633    9992398 vdcy1
>>>    251     1634          1 vdcy2
>>>    251     1637     489951 vdcy5
>>>    251     1648   10485760 vdcz
>>>    251     1649    9992398 vdcz1
>>>    251     1650          1 vdcz2
>>>    251     1653     489951 vdcz5
>>> 
>>> This is what makes qdev so useful.
>>> 
>>> args=""
>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>> for fn in 0 1 2 3 4 5 6 7; do
>>>       args="$args -drive
>>> 
file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}>>>
"
>>>       args="$args -device
>>> virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
>>> done
>>> done
>>> 
>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>> -enable-kvm -serial stdio
>>> 
>>> Regards,
>>> 
>>> Anthony Liguori
>>> 
>>>      
>>>> Rich.
>>>> 
>>>> 
>>>>        
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>      
>>    
> 

Re: [Qemu-devel] Hitting 29 NIC limit

[Anthony Liguori]

On 10/14/2010 05:12 PM, Anjali Kulkarni wrote:
> Thanks. Does this work for e1000 as well?
>    

Haven't tried.  I don't know how various e1000 drivers would react.

> Also, does it support pci hotplug?
>    

No, but that's fixable down the road.

Regards,

Anthony Liguori

> Anjali
>
> On 10/14/10 3:09 PM, "Anthony Liguori"<anthony@codemonkey.ws>  wrote:
>
>    
>> On 10/14/2010 05:00 PM, Anjali Kulkarni wrote:
>>      
>>> Can you send me pointers to the qdev documentation? How can I use it? Will
>>> it allow us to scale above the 32 PCI limit?
>>>
>>>        
>> It's all below.  You just have to create a PCI device and mark the
>> multifunction flag to on and then assign it a PCI address that includes
>> a function number.  Then you can pack 8 virtio PCI devices into a single
>> slot.
>>
>> Regards,
>>
>> Anthony Liguori
>>
>>      
>>> Anjali
>>>
>>>
>>> On 10/14/10 2:57 PM, "Anthony Liguori"<anthony@codemonkey.ws>   wrote:
>>>
>>>
>>>        
>>>> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
>>>>
>>>>          
>>>>> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>>>>>
>>>>>
>>>>>            
>>>>>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>>>>>> any type of device we emulate.
>>>>>>
>>>>>>
>>>>>>              
>>>>> Break the 29/30/31 virtio-blk limit ... please!
>>>>>
>>>>>
>>>>>            
>>>> It was broken ages ago:
>>>>
>>>> anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
>>>> 422 /proc/partitions
>>>>     251     1618          1 vdcx2
>>>>     251     1621     489951 vdcx5
>>>>     251     1632   10485760 vdcy
>>>>     251     1633    9992398 vdcy1
>>>>     251     1634          1 vdcy2
>>>>     251     1637     489951 vdcy5
>>>>     251     1648   10485760 vdcz
>>>>     251     1649    9992398 vdcz1
>>>>     251     1650          1 vdcz2
>>>>     251     1653     489951 vdcz5
>>>>
>>>> This is what makes qdev so useful.
>>>>
>>>> args=""
>>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>>> for fn in 0 1 2 3 4 5 6 7; do
>>>>        args="$args -drive
>>>>
>>>>          
> file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}>>>
> "
>    
>>>>        args="$args -device
>>>> virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
>>>> done
>>>> done
>>>>
>>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>>> -enable-kvm -serial stdio
>>>>
>>>> Regards,
>>>>
>>>> Anthony Liguori
>>>>
>>>>
>>>>          
>>>>> Rich.
>>>>>
>>>>>
>>>>>
>>>>>            
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>>          
>>>
>>>        
>>      
>    

[Qemu-devel] Re: Hitting 29 NIC limit

[Michael S. Tsirkin]

On Thu, Oct 14, 2010 at 05:17:36PM -0500, Anthony Liguori wrote:
> On 10/14/2010 05:12 PM, Anjali Kulkarni wrote:
> >Thanks. Does this work for e1000 as well?
> 
> Haven't tried.  I don't know how various e1000 drivers would react.
> 
> >Also, does it support pci hotplug?
> 
> No, but that's fixable down the road.
> 
> Regards,
> 
> Anthony Liguori

Probably not.

[Qemu-devel] Re: Hitting 29 NIC limit

[Anthony Liguori]

On 11/16/2010 11:23 AM, Michael S. Tsirkin wrote:
> On Thu, Oct 14, 2010 at 05:17:36PM -0500, Anthony Liguori wrote:
>    
>> On 10/14/2010 05:12 PM, Anjali Kulkarni wrote:
>>      
>>> Thanks. Does this work for e1000 as well?
>>>        
>> Haven't tried.  I don't know how various e1000 drivers would react.
>>
>>      
>>> Also, does it support pci hotplug?
>>>        
>> No, but that's fixable down the road.
>>
>> Regards,
>>
>> Anthony Liguori
>>      
> Probably not.
>    

FWIW, by fixable, I meant, we could add the ability to hot plug a 
multi-port NIC device.  For instance, we could have -device 
virtio-net-pci-x8 which would be an 8-port virtio-net card.

Regards,

Anthony Liguori

Re: [Qemu-devel] Hitting 29 NIC limit

[Richard W.M. Jones]

On Thu, Oct 14, 2010 at 04:57:36PM -0500, Anthony Liguori wrote:
> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
> >On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
> >>Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
> >>any type of device we emulate.
> >Break the 29/30/31 virtio-blk limit ... please!
> 
> It was broken ages ago:
[...]

Excellent news indeeed.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

Re: [Qemu-devel] Hitting 29 NIC limit

[Markus Armbruster]

Avi Kivity <avi@redhat.com> writes:

>  On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>
>>> Hi,
>>>
>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>> after 29
>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>> limit on a single bus- 3 are already taken up)
>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>> more, but ofcource that wont work.
>>> Is there any way to go beyond 29 NICs the legacy way?  What is the
>>> maximum
>>> that can be supported by the qdev mothod?
>>
>> I got up to 104 without trying very hard using the following script:
>>
>> args=""
>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>> for fn in 0 1 2 3 4 5 6 7; do
>>     args="$args -netdev user,id=eth${slot}_${fn}"
>>     args="$args -device
>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>> done
>> done
>>
>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>> -enable-kvm
>>
>> The key is to make the virtio-net devices multifunction and to fill
>> out all 8 functions for each slot.

I'm amazed that works.  Can't see how creating another qdev in the same
slot makes a proper multifunction device.

> This is unlikely to work right wrt pci hotplug.  If we want to support
> a large number of interfaces, we need true multiport cards.

Indeed.  As far as I know, we can't hot plug multifunction PCI devices.

> What's the motivation for such a huge number of interfaces?

Re: [Qemu-devel] Hitting 29 NIC limit

[Anthony Liguori]

On 10/14/2010 07:36 AM, Markus Armbruster wrote:
> Avi Kivity<avi@redhat.com>  writes:
>
>    
>>   On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>>      
>>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>        
>>>> Hi,
>>>>
>>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>>> after 29
>>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>>> limit on a single bus- 3 are already taken up)
>>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>>> more, but ofcource that wont work.
>>>> Is there any way to go beyond 29 NICs the legacy way?  What is the
>>>> maximum
>>>> that can be supported by the qdev mothod?
>>>>          
>>> I got up to 104 without trying very hard using the following script:
>>>
>>> args=""
>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>> for fn in 0 1 2 3 4 5 6 7; do
>>>      args="$args -netdev user,id=eth${slot}_${fn}"
>>>      args="$args -device
>>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>>> done
>>> done
>>>
>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>> -enable-kvm
>>>
>>> The key is to make the virtio-net devices multifunction and to fill
>>> out all 8 functions for each slot.
>>>        
> I'm amazed that works.  Can't see how creating another qdev in the same
> slot makes a proper multifunction device.
>    

multifunction=on sets the multifunction bit for the PCI device.  Then 
it's a matter of setting the address to be a specific function.

Our default platform devices are actually multifunction.

>> This is unlikely to work right wrt pci hotplug.  If we want to support
>> a large number of interfaces, we need true multiport cards.
>>      
> Indeed.  As far as I know, we can't hot plug multifunction PCI devices.
>    

Yup.

Regards,

Anthony Liguori

>> What's the motivation for such a huge number of interfaces?
>>      
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

Re: [Qemu-devel] Hitting 29 NIC limit

[Markus Armbruster]

Anthony Liguori <anthony@codemonkey.ws> writes:

> On 10/14/2010 07:36 AM, Markus Armbruster wrote:
>> Avi Kivity<avi@redhat.com>  writes:
>>
>>    
>>>   On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>>>      
>>>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>>        
>>>>> Hi,
>>>>>
>>>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>>>> after 29
>>>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>>>> limit on a single bus- 3 are already taken up)
>>>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>>>> more, but ofcource that wont work.
>>>>> Is there any way to go beyond 29 NICs the legacy way?  What is the
>>>>> maximum
>>>>> that can be supported by the qdev mothod?
>>>>>          
>>>> I got up to 104 without trying very hard using the following script:
>>>>
>>>> args=""
>>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>>> for fn in 0 1 2 3 4 5 6 7; do
>>>>      args="$args -netdev user,id=eth${slot}_${fn}"
>>>>      args="$args -device
>>>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>>>> done
>>>> done
>>>>
>>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>>> -enable-kvm
>>>>
>>>> The key is to make the virtio-net devices multifunction and to fill
>>>> out all 8 functions for each slot.
>>>>        
>> I'm amazed that works.  Can't see how creating another qdev in the same
>> slot makes a proper multifunction device.
>>    
>
> multifunction=on sets the multifunction bit for the PCI device.  Then
> it's a matter of setting the address to be a specific function.

I missed that bit, or perhaps forgot it already.  Neat!

Unfortunately, it's not yet good enough for hot plug.  We need to plug
the complete device in one go, not function by function.  This is harder
than hot unplug, where we can just add a command "unplug this PCI
device", then device_del its functions at our leisure.  The symmetric
way to plug would be device_add the functions, then plug the device.

Backward compatibility headache: current device_add/del plug/unplug
automatically.

> Our default platform devices are actually multifunction.

Yes, but they're built "by hand".

[...]

[Qemu-devel] Passing in additional info to guest OS and e1000 test suite?

[Anjali Kulkarni]

Hi,

- If I want to pass in additional arguments to the guest OS while booting(in
particular which slot I want to map a nic to) - is there any way to do it?
Some kind of configuration file that I can pass in would also be ok for me.
- Is there a KVM/Qemu/e1000 test suite that is already available and I can
us?

Thanks
Anjali

[Qemu-devel] Re: Passing in additional info to guest OS and e1000 test suite?

[Anthony Liguori]

On 10/15/2010 01:49 PM, Anjali Kulkarni wrote:
> Hi,
>
> - If I want to pass in additional arguments to the guest OS while booting(in
> particular which slot I want to map a nic to) - is there any way to do it?
> Some kind of configuration file that I can pass in would also be ok for me.
> - Is there a KVM/Qemu/e1000 test suite that is already available and I can
> us?
>    

I'm a little confused about what you're really asking.  Can you give a 
more illustrated example of what you're asking?  Are you using libvirt?

> Thanks
> Anjali
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

Re: [Qemu-devel] Hitting 29 NIC limit

[Anthony Liguori]

On 10/14/2010 07:07 AM, Avi Kivity wrote:
>  On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>
>>> Hi,
>>>
>>> Using the legacy way of starting up NICs, I am hitting a limitation 
>>> after 29
>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>> limit on a single bus- 3 are already taken up)
>>> I had initially increased the MAX_NICS to 48, just on my tree, to 
>>> get to
>>> more, but ofcource that wont work.
>>> Is there any way to go beyond 29 NICs the legacy way?  What is the 
>>> maximum
>>> that can be supported by the qdev mothod?
>>
>> I got up to 104 without trying very hard using the following script:
>>
>> args=""
>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>> for fn in 0 1 2 3 4 5 6 7; do
>>     args="$args -netdev user,id=eth${slot}_${fn}"
>>     args="$args -device 
>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile=" 
>>
>> done
>> done
>>
>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args} 
>> -enable-kvm
>>
>> The key is to make the virtio-net devices multifunction and to fill 
>> out all 8 functions for each slot.
>
> This is unlikely to work right wrt pci hotplug.

Yes.  Our hotplug design is based on devices..  This is wrong, it should 
be based on bus-level concepts (like PCI slots).

> If we want to support a large number of interfaces, we need true 
> multiport cards.

This magic here creates a multiport virtio-net card so I'm not really 
sure what you're suggesting.  It would certainly be nice to make this 
all more user friendly (and make hotplug work).

Regards,

Anthony Liguori

> What's the motivation for such a huge number of interfaces?

Re: [Qemu-devel] Hitting 29 NIC limit

[Avi Kivity]

  On 10/14/2010 02:54 PM, Anthony Liguori wrote:
>>> The key is to make the virtio-net devices multifunction and to fill 
>>> out all 8 functions for each slot.
>>
>> This is unlikely to work right wrt pci hotplug.
>
>
> Yes.  Our hotplug design is based on devices..  This is wrong, it 
> should be based on bus-level concepts (like PCI slots).
>
>> If we want to support a large number of interfaces, we need true 
>> multiport cards.
>
> This magic here creates a multiport virtio-net card so I'm not really 
> sure what you're suggesting.  It would certainly be nice to make this 
> all more user friendly (and make hotplug work).
>

The big issue is to fix hotplug.

I don't see how we can make it user friendly, without making the 
ordinary case even more unfriendly.  Looks like we need yet another 
level of indirection here.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Hitting 29 NIC limit

[Anthony Liguori]

On 10/14/2010 08:23 AM, Avi Kivity wrote:
>  On 10/14/2010 02:54 PM, Anthony Liguori wrote:
>>>> The key is to make the virtio-net devices multifunction and to fill 
>>>> out all 8 functions for each slot.
>>>
>>> This is unlikely to work right wrt pci hotplug.
>>
>>
>> Yes.  Our hotplug design is based on devices..  This is wrong, it 
>> should be based on bus-level concepts (like PCI slots).
>>
>>> If we want to support a large number of interfaces, we need true 
>>> multiport cards.
>>
>> This magic here creates a multiport virtio-net card so I'm not really 
>> sure what you're suggesting.  It would certainly be nice to make this 
>> all more user friendly (and make hotplug work).
>>
>
> The big issue is to fix hotplug.

Yes, but this is entirely independent of multifunction devices.

Today we shoe-horn hot remove into device_del.  Instead, we should have 
explicit bus-level interfaces for hot remove.

Regards,

Anthony Liguori

> I don't see how we can make it user friendly, without making the 
> ordinary case even more unfriendly.  Looks like we need yet another 
> level of indirection here.
>

Re: [Qemu-devel] Hitting 29 NIC limit

[Avi Kivity]

  On 10/14/2010 04:11 PM, Anthony Liguori wrote:
> On 10/14/2010 08:23 AM, Avi Kivity wrote:
>>  On 10/14/2010 02:54 PM, Anthony Liguori wrote:
>>>>> The key is to make the virtio-net devices multifunction and to 
>>>>> fill out all 8 functions for each slot.
>>>>
>>>> This is unlikely to work right wrt pci hotplug.
>>>
>>>
>>> Yes.  Our hotplug design is based on devices..  This is wrong, it 
>>> should be based on bus-level concepts (like PCI slots).
>>>
>>>> If we want to support a large number of interfaces, we need true 
>>>> multiport cards.
>>>
>>> This magic here creates a multiport virtio-net card so I'm not 
>>> really sure what you're suggesting.  It would certainly be nice to 
>>> make this all more user friendly (and make hotplug work).
>>>
>>
>> The big issue is to fix hotplug.
>
> Yes, but this is entirely independent of multifunction devices.
>
> Today we shoe-horn hot remove into device_del.  Instead, we should 
> have explicit bus-level interfaces for hot remove.

I'm not saying multiplug is not the right way to approach this (it is).  
The only concern is to get hotplug right.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: Hitting 29 NIC limit

[Michael S. Tsirkin]

On Thu, Oct 14, 2010 at 02:07:17PM +0200, Avi Kivity wrote:
>  On 10/14/2010 12:54 AM, Anthony Liguori wrote:
> >On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
> >>
> >>Hi,
> >>
> >>Using the legacy way of starting up NICs, I am hitting a
> >>limitation after 29
> >>NICs ie no more than 29 are detected (that's because of the 32 PCI slot
> >>limit on a single bus- 3 are already taken up)
> >>I had initially increased the MAX_NICS to 48, just on my tree, to get to
> >>more, but ofcource that wont work.
> >>Is there any way to go beyond 29 NICs the legacy way?  What is
> >>the maximum
> >>that can be supported by the qdev mothod?
> >
> >I got up to 104 without trying very hard using the following script:
> >
> >args=""
> >for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
> >for fn in 0 1 2 3 4 5 6 7; do
> >    args="$args -netdev user,id=eth${slot}_${fn}"
> >    args="$args -device virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
> >done
> >done
> >
> >x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
> >-enable-kvm
> >
> >The key is to make the virtio-net devices multifunction and to
> >fill out all 8 functions for each slot.
> 
> This is unlikely to work right wrt pci hotplug.  If we want to
> support a large number of interfaces, we need true multiport cards.

Bridge support seems to be working mostly fine here.
That will let you go up to 256 devices without multifunction.

-- 
MST

[Qemu-devel] Re: Hitting 29 NIC limit

[Isaku Yamahata]

Right now, qemu is internally ready to handle pci-to-pci bridge emulators
and it's quite easy to add a new simple p2p bridge emulator.
Just a single function to initialize vid/did and a single PCIDevinceInfo.

But it's difficult to use multi pci buses at the moment
because other part isn't ready. It's only for developers
who are interested in this area.

BTW, how many pci buses/slots/functions do you want at most?


What we have in the upstream repository is
- various fixes for p2p bridge support
- the library for p2p bridge support
- pcie root/upstream/downstream port emulators which is p2p bridges
  ioh3420/xio3130_upstream/xio3130_downstream

What is missing for multiple pci buses is
- a new PC machine description which has multiple pci buses
  My Q35 chipset patch will provide this.
- a simple P2P bridge emulator for pc architecture
- seabios support
  - DSDT for a new machine which has multi pci buses(IRQ routing)
  - pv pci bus numbering
    this isn't a must, I suppose
  I have those patches locally.

thanks,

On Wed, Nov 17, 2010 at 12:58:43PM +0200, Michael S. Tsirkin wrote:
> On Tue, Nov 16, 2010 at 11:20:52AM -0800, Anjali Kulkarni wrote:
> > Hi,
> > 
> > Can you elaborate what you mean by bridge support? I would really like to go
> > to 256 interfaces.
> > 
> > Anjali
> 
> Isaku Yamahata wrote most of it, I'll let him comment.
> 
> -- 
> MST
> 

-- 
yamahata

Re: [Qemu-devel] Re: Hitting 29 NIC limit

[Wei Xu]

Isaku,

Could you explain more on irq routing in dsdt and related code in qemu? I
encountered a problem: emulated user networking (-net nic -net user) cannot
work in my qemu-kvm+q35+pcie code. Your original q35 (pure qemu) works fine.

Thanks!
Wei


On 11/17/10 7:09 PM, "Isaku Yamahata" <yamahata@valinux.co.jp> wrote:

> Right now, qemu is internally ready to handle pci-to-pci bridge emulators
> and it's quite easy to add a new simple p2p bridge emulator.
> Just a single function to initialize vid/did and a single PCIDevinceInfo.
> 
> But it's difficult to use multi pci buses at the moment
> because other part isn't ready. It's only for developers
> who are interested in this area.
> 
> BTW, how many pci buses/slots/functions do you want at most?
> 
> 
> What we have in the upstream repository is
> - various fixes for p2p bridge support
> - the library for p2p bridge support
> - pcie root/upstream/downstream port emulators which is p2p bridges
>   ioh3420/xio3130_upstream/xio3130_downstream
> 
> What is missing for multiple pci buses is
> - a new PC machine description which has multiple pci buses
>   My Q35 chipset patch will provide this.
> - a simple P2P bridge emulator for pc architecture
> - seabios support
>   - DSDT for a new machine which has multi pci buses(IRQ routing)
>   - pv pci bus numbering
>     this isn't a must, I suppose
>   I have those patches locally.
> 
> thanks,
> 
> On Wed, Nov 17, 2010 at 12:58:43PM +0200, Michael S. Tsirkin wrote:
>> On Tue, Nov 16, 2010 at 11:20:52AM -0800, Anjali Kulkarni wrote:
>>> Hi,
>>> 
>>> Can you elaborate what you mean by bridge support? I would really like to go
>>> to 256 interfaces.
>>> 
>>> Anjali
>> 
>> Isaku Yamahata wrote most of it, I'll let him comment.
>> 
>> -- 
>> MST
>> 

Re: [Qemu-devel] Re: Hitting 29 NIC limit

[Isaku Yamahata]

On Wed, Nov 17, 2010 at 09:41:03PM -0800, Wei Xu wrote:
> Could you explain more on irq routing in dsdt and related code in qemu? I
> encountered a problem: emulated user networking (-net nic -net user) cannot
> work in my qemu-kvm+q35+pcie code. Your original q35 (pure qemu) works fine.

The related functions are ich9_lpc_set_irq_pic() and ich9_lpc_set_irq()
in q35.c.
I'm aware that there is something wrong.
I'll fix it when I rebase the patches, but I haven't found time yet.
-- 
yamahata

[Qemu-devel] Re: Hitting 29 NIC limit

[Michael S. Tsirkin]

On Thu, Nov 18, 2010 at 12:09:18PM +0900, Isaku Yamahata wrote:
> Right now, qemu is internally ready to handle pci-to-pci bridge emulators
> and it's quite easy to add a new simple p2p bridge emulator.
> Just a single function to initialize vid/did and a single PCIDevinceInfo.
> 
> But it's difficult to use multi pci buses at the moment
> because other part isn't ready. It's only for developers
> who are interested in this area.
> 
> BTW, how many pci buses/slots/functions do you want at most?
> 
> 
> What we have in the upstream repository is
> - various fixes for p2p bridge support
> - the library for p2p bridge support
> - pcie root/upstream/downstream port emulators which is p2p bridges
>   ioh3420/xio3130_upstream/xio3130_downstream
> 
> What is missing for multiple pci buses is
> - a new PC machine description which has multiple pci buses
>   My Q35 chipset patch will provide this.

IMHO, what we really need is a way to specify bridges
through qdev. Not a machine with a hard-coded buses.
The firmware paths used by Gleb's code might supply
a good syntax to specify topology.

> - a simple P2P bridge emulator for pc architecture
> - seabios support
>   - DSDT for a new machine which has multi pci buses(IRQ routing)
>   - pv pci bus numbering

Interesting. What's the motivation for this?

>     this isn't a must, I suppose
>   I have those patches locally.
> 
> thanks,
> 
> On Wed, Nov 17, 2010 at 12:58:43PM +0200, Michael S. Tsirkin wrote:
> > On Tue, Nov 16, 2010 at 11:20:52AM -0800, Anjali Kulkarni wrote:
> > > Hi,
> > > 
> > > Can you elaborate what you mean by bridge support? I would really like to go
> > > to 256 interfaces.
> > > 
> > > Anjali
> > 
> > Isaku Yamahata wrote most of it, I'll let him comment.
> > 
> > -- 
> > MST
> > 
> 
> -- 
> yamahata

[Qemu-devel] Re: Hitting 29 NIC limit

[Isaku Yamahata]

On Thu, Nov 18, 2010 at 08:46:43AM +0200, Michael S. Tsirkin wrote:
> > What is missing for multiple pci buses is
> > - a new PC machine description which has multiple pci buses
> >   My Q35 chipset patch will provide this.
> 
> IMHO, what we really need is a way to specify bridges
> through qdev. Not a machine with a hard-coded buses.

Yes, it's long-term goal.


> The firmware paths used by Gleb's code might supply
> a good syntax to specify topology.
> 
> > - a simple P2P bridge emulator for pc architecture
> > - seabios support
> >   - DSDT for a new machine which has multi pci buses(IRQ routing)
> >   - pv pci bus numbering
> 
> Interesting. What's the motivation for this?

For hot-plugging pci-to-pci bridge. It requires sparse pci bus numbering.
In fact, it's not so strong requirement.
Possibly I would have to dig into dsdt.
-- 
yamahata

[Qemu-devel] Re: Hitting 29 NIC limit

[Michael S. Tsirkin]

On Thu, Nov 18, 2010 at 04:09:34PM +0900, Isaku Yamahata wrote:
> On Thu, Nov 18, 2010 at 08:46:43AM +0200, Michael S. Tsirkin wrote:
> > > What is missing for multiple pci buses is
> > > - a new PC machine description which has multiple pci buses
> > >   My Q35 chipset patch will provide this.
> > 
> > IMHO, what we really need is a way to specify bridges
> > through qdev. Not a machine with a hard-coded buses.
> 
> Yes, it's long-term goal.
> 
> 
> > The firmware paths used by Gleb's code might supply
> > a good syntax to specify topology.
> > 
> > > - a simple P2P bridge emulator for pc architecture
> > > - seabios support
> > >   - DSDT for a new machine which has multi pci buses(IRQ routing)
> > >   - pv pci bus numbering
> > 
> > Interesting. What's the motivation for this?
> 
> For hot-plugging pci-to-pci bridge. It requires sparse pci bus numbering.

Not sure I understand. pci bus numbering should be
completely up to the guest.

> In fact, it's not so strong requirement.
> Possibly I would have to dig into dsdt.

Yes, some guest-side technology like acpi might be needed.

> -- 
> yamahata

[Qemu-devel] Re: Hitting 29 NIC limit

[Isaku Yamahata]

On Thu, Nov 18, 2010 at 09:15:19AM +0200, Michael S. Tsirkin wrote:
> On Thu, Nov 18, 2010 at 04:09:34PM +0900, Isaku Yamahata wrote:
> > On Thu, Nov 18, 2010 at 08:46:43AM +0200, Michael S. Tsirkin wrote:
> > > > What is missing for multiple pci buses is
> > > > - a new PC machine description which has multiple pci buses
> > > >   My Q35 chipset patch will provide this.
> > > 
> > > IMHO, what we really need is a way to specify bridges
> > > through qdev. Not a machine with a hard-coded buses.
> > 
> > Yes, it's long-term goal.
> > 
> > 
> > > The firmware paths used by Gleb's code might supply
> > > a good syntax to specify topology.
> > > 
> > > > - a simple P2P bridge emulator for pc architecture
> > > > - seabios support
> > > >   - DSDT for a new machine which has multi pci buses(IRQ routing)
> > > >   - pv pci bus numbering
> > > 
> > > Interesting. What's the motivation for this?
> > 
> > For hot-plugging pci-to-pci bridge. It requires sparse pci bus numbering.
> 
> Not sure I understand. pci bus numbering should be
> completely up to the guest.

Yes.


> > In fact, it's not so strong requirement.
> > Possibly I would have to dig into dsdt.
> 
> Yes, some guest-side technology like acpi might be needed.

OSes like Linux tries to keep bios settings as much as possible.
It means seabios needs to understand dsdt and initialize pci buses
properly according to it.
I think pv numbering is much easier than teaching seabios dsdt parsing.
-- 
yamahata

Re: [Qemu-devel] Re: Hitting 29 NIC limit

[Gerd Hoffmann]

   Hi,

> What is missing for multiple pci buses is

> - a simple P2P bridge emulator for pc architecture
> - seabios support
>    - DSDT for a new machine which has multi pci buses(IRQ routing)

Is this really needed?

IIRC there are (used to be?) multiport nic pci cards which are simply 
multiple ethernet card pci chips behind a pci-to-pci bridge.  Such a 
pci-to-pci bridge should be able to operate without bios support, no?

So one cold just do ...

   -device pci-bridge,id=br
   -device e1000,bus=br.0
   -device e1000,bus=br.0

to create a bridge with two nics behind it?

cheers,
   Gerd

Re: [Qemu-devel] Re: Hitting 29 NIC limit

[Isaku Yamahata]

On Thu, Nov 18, 2010 at 12:54:29PM +0100, Gerd Hoffmann wrote:
>   Hi,
>
>> What is missing for multiple pci buses is
>
>> - a simple P2P bridge emulator for pc architecture
>> - seabios support
>>    - DSDT for a new machine which has multi pci buses(IRQ routing)
>
> Is this really needed?

Q35 IRQ routing is different from piix4's. So new one is necessary.
Possibly it won't depend on the number of pci buses,
I haven't confirmed it.
-- 
yamahata