Unable to create more than 1 guest virtio-net device using vhost-net backend

Unable to create more than 1 guest virtio-net device using vhost-net backend

[Sridhar Samudrala]

When creating a guest with 2 virtio-net interfaces, i am running
into a issue causing the 2nd i/f falling back to userpace virtio
even when vhost is enabled.

After some debugging, it turned out that KVM_IOEVENTFD ioctl() 
call in qemu is failing with ENOSPC.
This is because of the NR_IOBUS_DEVS(6) limit in kvm_io_bus_register_dev()
routine in the host kernel.

I think we need to increase this limit if we want to support multiple
network interfaces using vhost-net.
Is there an alternate solution?

Thanks
Sridhar

Re: Unable to create more than 1 guest virtio-net device using vhost-net backend

[Michael S. Tsirkin]

On Fri, Mar 19, 2010 at 03:19:27PM -0700, Sridhar Samudrala wrote:
> When creating a guest with 2 virtio-net interfaces, i am running
> into a issue causing the 2nd i/f falling back to userpace virtio
> even when vhost is enabled.
> 
> After some debugging, it turned out that KVM_IOEVENTFD ioctl() 
> call in qemu is failing with ENOSPC.
> This is because of the NR_IOBUS_DEVS(6) limit in kvm_io_bus_register_dev()
> routine in the host kernel.
> 
> I think we need to increase this limit if we want to support multiple
> network interfaces using vhost-net.
> Is there an alternate solution?
> 
> Thanks
> Sridhar

Nothing easy that I can see. Each device needs 2 of these.  Avi, Gleb,
any objections to increasing the limit to say 16?  That would give us
5 more devices to the limit of 6 per guest.

-- 
MST

Re: Unable to create more than 1 guest virtio-net device using vhost-net backend

[Avi Kivity]

On 03/21/2010 11:55 AM, Michael S. Tsirkin wrote:
> On Fri, Mar 19, 2010 at 03:19:27PM -0700, Sridhar Samudrala wrote:
>    
>> When creating a guest with 2 virtio-net interfaces, i am running
>> into a issue causing the 2nd i/f falling back to userpace virtio
>> even when vhost is enabled.
>>
>> After some debugging, it turned out that KVM_IOEVENTFD ioctl()
>> call in qemu is failing with ENOSPC.
>> This is because of the NR_IOBUS_DEVS(6) limit in kvm_io_bus_register_dev()
>> routine in the host kernel.
>>
>> I think we need to increase this limit if we want to support multiple
>> network interfaces using vhost-net.
>> Is there an alternate solution?
>>
>> Thanks
>> Sridhar
>>      
> Nothing easy that I can see. Each device needs 2 of these.  Avi, Gleb,
> any objections to increasing the limit to say 16?  That would give us
> 5 more devices to the limit of 6 per guest.
>    

Increase it to 200, then.

Is the limit visible to userspace?  If not, we need to expose it.

-- 
error compiling committee.c: too many arguments to function

Re: Unable to create more than 1 guest virtio-net device using vhost-net backend

[Michael S. Tsirkin]

On Sun, Mar 21, 2010 at 12:11:33PM +0200, Avi Kivity wrote:
> On 03/21/2010 11:55 AM, Michael S. Tsirkin wrote:
>> On Fri, Mar 19, 2010 at 03:19:27PM -0700, Sridhar Samudrala wrote:
>>    
>>> When creating a guest with 2 virtio-net interfaces, i am running
>>> into a issue causing the 2nd i/f falling back to userpace virtio
>>> even when vhost is enabled.
>>>
>>> After some debugging, it turned out that KVM_IOEVENTFD ioctl()
>>> call in qemu is failing with ENOSPC.
>>> This is because of the NR_IOBUS_DEVS(6) limit in kvm_io_bus_register_dev()
>>> routine in the host kernel.
>>>
>>> I think we need to increase this limit if we want to support multiple
>>> network interfaces using vhost-net.
>>> Is there an alternate solution?
>>>
>>> Thanks
>>> Sridhar
>>>      
>> Nothing easy that I can see. Each device needs 2 of these.  Avi, Gleb,
>> any objections to increasing the limit to say 16?  That would give us
>> 5 more devices to the limit of 6 per guest.
>>    
>
> Increase it to 200, then.

OK. I think we'll also need a smarter allocator
than bus->dev_count++ than we now have. Right?

> Is the limit visible to userspace?  If not, we need to expose it.

I don't think it's visible: it seems to be used in a single
place in kvm. Let's add an ioctl? Note that qemu doesn't
need it now ...

> -- 
> error compiling committee.c: too many arguments to function

Re: Unable to create more than 1 guest virtio-net device using vhost-net backend

[Gleb Natapov]

On Sun, Mar 21, 2010 at 12:11:33PM +0200, Avi Kivity wrote:
> On 03/21/2010 11:55 AM, Michael S. Tsirkin wrote:
> >On Fri, Mar 19, 2010 at 03:19:27PM -0700, Sridhar Samudrala wrote:
> >>When creating a guest with 2 virtio-net interfaces, i am running
> >>into a issue causing the 2nd i/f falling back to userpace virtio
> >>even when vhost is enabled.
> >>
> >>After some debugging, it turned out that KVM_IOEVENTFD ioctl()
> >>call in qemu is failing with ENOSPC.
> >>This is because of the NR_IOBUS_DEVS(6) limit in kvm_io_bus_register_dev()
> >>routine in the host kernel.
> >>
> >>I think we need to increase this limit if we want to support multiple
> >>network interfaces using vhost-net.
> >>Is there an alternate solution?
> >>
> >>Thanks
> >>Sridhar
> >Nothing easy that I can see. Each device needs 2 of these.  Avi, Gleb,
> >any objections to increasing the limit to say 16?  That would give us
> >5 more devices to the limit of 6 per guest.
> 
> Increase it to 200, then.
> 
Currently on each device read/write we iterate over all registered
devices. This is not scalable.

> Is the limit visible to userspace?  If not, we need to expose it.
> 
> -- 
> error compiling committee.c: too many arguments to function

--
			Gleb.

Re: Unable to create more than 1 guest virtio-net device using vhost-net backend

[Avi Kivity]

On 03/21/2010 12:15 PM, Michael S. Tsirkin wrote:
>>> Nothing easy that I can see. Each device needs 2 of these.  Avi, Gleb,
>>> any objections to increasing the limit to say 16?  That would give us
>>> 5 more devices to the limit of 6 per guest.
>>>
>>>        
>> Increase it to 200, then.
>>      
> OK. I think we'll also need a smarter allocator
> than bus->dev_count++ than we now have. Right?
>    

No, why?

Eventually we'll want faster scanning than the linear search we employ 
now, though.

>> Is the limit visible to userspace?  If not, we need to expose it.
>>      
> I don't think it's visible: it seems to be used in a single
> place in kvm. Let's add an ioctl? Note that qemu doesn't
> need it now ...
>    

We usually expose limits via KVM_CHECK_EXTENSION(KVM_CAP_BLAH).  We can 
expose it via KVM_CAP_IOEVENTFD (and need to reserve iodev entries for 
those).

-- 
error compiling committee.c: too many arguments to function

Re: Unable to create more than 1 guest virtio-net device using vhost-net backend

[Avi Kivity]

On 03/21/2010 12:21 PM, Gleb Natapov wrote:
> On Sun, Mar 21, 2010 at 12:11:33PM +0200, Avi Kivity wrote:
>    
>> On 03/21/2010 11:55 AM, Michael S. Tsirkin wrote:
>>      
>>> On Fri, Mar 19, 2010 at 03:19:27PM -0700, Sridhar Samudrala wrote:
>>>        
>>>> When creating a guest with 2 virtio-net interfaces, i am running
>>>> into a issue causing the 2nd i/f falling back to userpace virtio
>>>> even when vhost is enabled.
>>>>
>>>> After some debugging, it turned out that KVM_IOEVENTFD ioctl()
>>>> call in qemu is failing with ENOSPC.
>>>> This is because of the NR_IOBUS_DEVS(6) limit in kvm_io_bus_register_dev()
>>>> routine in the host kernel.
>>>>
>>>> I think we need to increase this limit if we want to support multiple
>>>> network interfaces using vhost-net.
>>>> Is there an alternate solution?
>>>>
>>>> Thanks
>>>> Sridhar
>>>>          
>>> Nothing easy that I can see. Each device needs 2 of these.  Avi, Gleb,
>>> any objections to increasing the limit to say 16?  That would give us
>>> 5 more devices to the limit of 6 per guest.
>>>        
>> Increase it to 200, then.
>>
>>      
> Currently on each device read/write we iterate over all registered
> devices. This is not scalable.
>    

Yeah.  We need first to drop the callback based matching and replace it 
with explicit ranges, then to replace the search with a hash table for 
small ranges (keeping a linear search for large ranges, can happen for 
coalesced mmio).

-- 
error compiling committee.c: too many arguments to function

Re: Unable to create more than 1 guest virtio-net device using vhost-net backend

[Michael S. Tsirkin]

On Sun, Mar 21, 2010 at 12:29:31PM +0200, Avi Kivity wrote:
> On 03/21/2010 12:15 PM, Michael S. Tsirkin wrote:
>>>> Nothing easy that I can see. Each device needs 2 of these.  Avi, Gleb,
>>>> any objections to increasing the limit to say 16?  That would give us
>>>> 5 more devices to the limit of 6 per guest.
>>>>
>>>>        
>>> Increase it to 200, then.
>>>      
>> OK. I think we'll also need a smarter allocator
>> than bus->dev_count++ than we now have. Right?
>>    
>
> No, why?

We'll run into problems if devices are created/removed in random order,
won't we?

> Eventually we'll want faster scanning than the linear search we employ  
> now, though.

Yes I suspect with 200 entries we will :). Let's just make it 16 for
now?

>>> Is the limit visible to userspace?  If not, we need to expose it.
>>>      
>> I don't think it's visible: it seems to be used in a single
>> place in kvm. Let's add an ioctl? Note that qemu doesn't
>> need it now ...
>>    
>
> We usually expose limits via KVM_CHECK_EXTENSION(KVM_CAP_BLAH).  We can  
> expose it via KVM_CAP_IOEVENTFD (and need to reserve iodev entries for  
> those).
>
> -- 
> error compiling committee.c: too many arguments to function

Re: Unable to create more than 1 guest virtio-net device using vhost-net backend

[Avi Kivity]

On 03/21/2010 01:34 PM, Michael S. Tsirkin wrote:
> On Sun, Mar 21, 2010 at 12:29:31PM +0200, Avi Kivity wrote:
>    
>> On 03/21/2010 12:15 PM, Michael S. Tsirkin wrote:
>>      
>>>>> Nothing easy that I can see. Each device needs 2 of these.  Avi, Gleb,
>>>>> any objections to increasing the limit to say 16?  That would give us
>>>>> 5 more devices to the limit of 6 per guest.
>>>>>
>>>>>
>>>>>            
>>>> Increase it to 200, then.
>>>>
>>>>          
>>> OK. I think we'll also need a smarter allocator
>>> than bus->dev_count++ than we now have. Right?
>>>
>>>        
>> No, why?
>>      
> We'll run into problems if devices are created/removed in random order,
> won't we?
>    

unregister_dev() takes care of it.

>> Eventually we'll want faster scanning than the linear search we employ
>> now, though.
>>      
> Yes I suspect with 200 entries we will :). Let's just make it 16 for
> now?
>    

Let's make it 200 and fix the performance problems later.  Making it 16 
is just asking for trouble.

-- 
error compiling committee.c: too many arguments to function

Re: Unable to create more than 1 guest virtio-net device using vhost-net backend

[Michael S. Tsirkin]

On Sun, Mar 21, 2010 at 01:58:29PM +0200, Avi Kivity wrote:
> On 03/21/2010 01:34 PM, Michael S. Tsirkin wrote:
>> On Sun, Mar 21, 2010 at 12:29:31PM +0200, Avi Kivity wrote:
>>    
>>> On 03/21/2010 12:15 PM, Michael S. Tsirkin wrote:
>>>      
>>>>>> Nothing easy that I can see. Each device needs 2 of these.  Avi, Gleb,
>>>>>> any objections to increasing the limit to say 16?  That would give us
>>>>>> 5 more devices to the limit of 6 per guest.
>>>>>>
>>>>>>
>>>>>>            
>>>>> Increase it to 200, then.
>>>>>
>>>>>          
>>>> OK. I think we'll also need a smarter allocator
>>>> than bus->dev_count++ than we now have. Right?
>>>>
>>>>        
>>> No, why?
>>>      
>> We'll run into problems if devices are created/removed in random order,
>> won't we?
>>    
>
> unregister_dev() takes care of it.
>
>>> Eventually we'll want faster scanning than the linear search we employ
>>> now, though.
>>>      
>> Yes I suspect with 200 entries we will :). Let's just make it 16 for
>> now?
>>    
>
> Let's make it 200 and fix the performance problems later.  Making it 16  
> is just asking for trouble.

I did this and performance with vhost seems to become much more noisy,
and drop by about 10% on average, even though in practice only
a single device is created. Still trying to figure it out ...
Any idea?

> -- 
> error compiling committee.c: too many arguments to function

Re: Unable to create more than 1 guest virtio-net device using vhost-net backend

[Sridhar Samudrala]

On Mon, 2010-03-22 at 20:16 +0200, Michael S. Tsirkin wrote:
> On Sun, Mar 21, 2010 at 01:58:29PM +0200, Avi Kivity wrote:
> > On 03/21/2010 01:34 PM, Michael S. Tsirkin wrote:
> >> On Sun, Mar 21, 2010 at 12:29:31PM +0200, Avi Kivity wrote:
> >>    
> >>> On 03/21/2010 12:15 PM, Michael S. Tsirkin wrote:
> >>>      
> >>>>>> Nothing easy that I can see. Each device needs 2 of these.  Avi, Gleb,
> >>>>>> any objections to increasing the limit to say 16?  That would give us
> >>>>>> 5 more devices to the limit of 6 per guest.
> >>>>>>
> >>>>>>
> >>>>>>            
> >>>>> Increase it to 200, then.
> >>>>>
> >>>>>          
> >>>> OK. I think we'll also need a smarter allocator
> >>>> than bus->dev_count++ than we now have. Right?
> >>>>
> >>>>        
> >>> No, why?
> >>>      
> >> We'll run into problems if devices are created/removed in random order,
> >> won't we?
> >>    
> >
> > unregister_dev() takes care of it.
> >
> >>> Eventually we'll want faster scanning than the linear search we employ
> >>> now, though.
> >>>      
> >> Yes I suspect with 200 entries we will :). Let's just make it 16 for
> >> now?
> >>    
> >
> > Let's make it 200 and fix the performance problems later.  Making it 16  
> > is just asking for trouble.
> 
> I did this and performance with vhost seems to become much more noisy,
> and drop by about 10% on average, even though in practice only
> a single device is created. Still trying to figure it out ...
> Any idea?

I am not sure if this 10% variation is due to the increase of NR_IO_BUS_DEVS.
In our testing, we do see variations of 10% or higher between multiple netperf 
instances with the same setup/configuration when using virtio/vhost.

Thanks
Sridhar