virtio & hypercall interface?

virtio & hypercall interface?

[Rusty Russell]

Hi all,

	I've finally started looking at Dor's git tree, and it struck me that
it conflicts with Anthony's hypercall patches.  FWIW I like Anthony's
patching thing, and don't really care about arg order.  It'd be nice if
we could pull in the same direction tho 8)

Thanks,
Rusty.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Anthony Liguori]

Rusty Russell wrote:
> Hi all,
>
> 	I've finally started looking at Dor's git tree, and it struck me that
> it conflicts with Anthony's hypercall patches.  FWIW I like Anthony's
> patching thing, and don't really care about arg order.

Are you trying to implement cpuid in assembly in the monitor?  
Otherwise, I don't see why arg order would matter.

Regards,

Anthony Liguori

>   It'd be nice if
> we could pull in the same direction tho 8)
>
> Thanks,
> Rusty.
>
>   


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Dor Laor]

>Hi all,
>
>	I've finally started looking at Dor's git tree, and it struck me
that
>it conflicts with Anthony's hypercall patches.  FWIW I like Anthony's
>patching thing, and don't really care about arg order.  It'd be nice if
>we could pull in the same direction tho 8)
>
>Thanks,
>Rusty.

Good news you're looking at my tree, since the forum I didn't do much
since I had to catch up some gazlion other tasks, never the less
starting on Sunday I'm back again.

Actually, I wanted to rebase my hypercalls over Anhtony's too (except
for allowing userspace handling). I paused it since Anhtony's patch
didn't apply. 
Best regards, Dor.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Anthony Liguori]

Dor Laor wrote:
>> Hi all,
>>
>> 	I've finally started looking at Dor's git tree, and it struck me
>>     
> that
>   
>> it conflicts with Anthony's hypercall patches.  FWIW I like Anthony's
>> patching thing, and don't really care about arg order.  It'd be nice if
>> we could pull in the same direction tho 8)
>>
>> Thanks,
>> Rusty.
>>     
>
> Good news you're looking at my tree, since the forum I didn't do much
> since I had to catch up some gazlion other tasks, never the less
> starting on Sunday I'm back again.
>
> Actually, I wanted to rebase my hypercalls over Anhtony's too (except
> for allowing userspace handling).

I thought we discussed just providing a signaling message to userspace 
for virtio?  It's not strictly necessary to expose hypercalls to 
userspace in order to implement a virtio backend in userspace.

Regards,

Anthony Liguori

>  I paused it since Anhtony's patch
> didn't apply. 
> Best regards, Dor.
>   


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Rusty Russell]

On Thu, 2007-09-13 at 14:21 -0500, Anthony Liguori wrote:
> Rusty Russell wrote:
> > Hi all,
> >
> > 	I've finally started looking at Dor's git tree, and it struck me that
> > it conflicts with Anthony's hypercall patches.  FWIW I like Anthony's
> > patching thing, and don't really care about arg order.
> 
> Are you trying to implement cpuid in assembly in the monitor?  
> Otherwise, I don't see why arg order would matter.

I'm using x86_emulate_cpuid().  And I'm lazy.

Rusty.



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Anthony Liguori]

Rusty Russell wrote:
> On Thu, 2007-09-13 at 14:21 -0500, Anthony Liguori wrote:
>   
>> Rusty Russell wrote:
>>     
>>> Hi all,
>>>
>>> 	I've finally started looking at Dor's git tree, and it struck me that
>>> it conflicts with Anthony's hypercall patches.  FWIW I like Anthony's
>>> patching thing, and don't really care about arg order.
>>>       
>> Are you trying to implement cpuid in assembly in the monitor?  
>> Otherwise, I don't see why arg order would matter.
>>     
>
> I'm using x86_emulate_cpuid().  And I'm lazy.
>   

Hrm, how are you managing that?   CPUID can take 4 arguments and return 
4.  The hypercall interface can only return a single argument.  What do 
you do for the signature lookup?

Regards,

Anthony Liguori

> Rusty.
>
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> kvm-devel mailing list
> kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/kvm-devel
>
>   


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Avi Kivity]

Rusty Russell wrote:
> Hi all,
>
> 	I've finally started looking at Dor's git tree, and it struck me that
> it conflicts with Anthony's hypercall patches.  FWIW I like Anthony's
> patching thing, and don't really care about arg order.  It'd be nice if
> we could pull in the same direction tho 8)
>   

I prefer Anthony's hypercalls too.  Anthony, can you resend the
patchset?  Last I looked I was unable to get a coherent set.


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Avi Kivity]

Anthony Liguori wrote:
> Dor Laor wrote:
>   
>>> Hi all,
>>>
>>> 	I've finally started looking at Dor's git tree, and it struck me
>>>     
>>>       
>> that
>>   
>>     
>>> it conflicts with Anthony's hypercall patches.  FWIW I like Anthony's
>>> patching thing, and don't really care about arg order.  It'd be nice if
>>> we could pull in the same direction tho 8)
>>>
>>> Thanks,
>>> Rusty.
>>>     
>>>       
>> Good news you're looking at my tree, since the forum I didn't do much
>> since I had to catch up some gazlion other tasks, never the less
>> starting on Sunday I'm back again.
>>
>> Actually, I wanted to rebase my hypercalls over Anhtony's too (except
>> for allowing userspace handling).
>>     
>
> I thought we discussed just providing a signaling message to userspace 
> for virtio?  It's not strictly necessary to expose hypercalls to 
> userspace in order to implement a virtio backend in userspace.
>
>   

Yes, that's what I'd like to see too.  Signal a channel.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Anthony Liguori]

Avi Kivity wrote:
> Rusty Russell wrote:
>   
>> Hi all,
>>
>> 	I've finally started looking at Dor's git tree, and it struck me that
>> it conflicts with Anthony's hypercall patches.  FWIW I like Anthony's
>> patching thing, and don't really care about arg order.  It'd be nice if
>> we could pull in the same direction tho 8)
>>   
>>     
>
> I prefer Anthony's hypercalls too.  Anthony, can you resend the
> patchset?  Last I looked I was unable to get a coherent set.
>
>   

Yeah, I'll rebase and send out today.

Regards,

Anthony Liguori


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Dor Laor]

Avi Kivity wrote:
> Anthony Liguori wrote:
>   
>> Dor Laor wrote:
>>   
>>     
>>>> Hi all,
>>>>
>>>> 	I've finally started looking at Dor's git tree, and it struck me
>>>>     
>>>>       
>>>>         
>>> that
>>>   
>>>     
>>>       
>>>> it conflicts with Anthony's hypercall patches.  FWIW I like Anthony's
>>>> patching thing, and don't really care about arg order.  It'd be nice if
>>>> we could pull in the same direction tho 8)
>>>>
>>>> Thanks,
>>>> Rusty.
>>>>     
>>>>       
>>>>         
>>> Good news you're looking at my tree, since the forum I didn't do much
>>> since I had to catch up some gazlion other tasks, never the less
>>> starting on Sunday I'm back again.
>>>
>>> Actually, I wanted to rebase my hypercalls over Anhtony's too (except
>>> for allowing userspace handling).
>>>     
>>>       
>> I thought we discussed just providing a signaling message to userspace 
>> for virtio?  It's not strictly necessary to expose hypercalls to 
>> userspace in order to implement a virtio backend in userspace.
>>
>>   
>>     
>
> Yes, that's what I'd like to see too.  Signal a channel.
>
>   
First, I though that this 
http://www.mail-archive.com/kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org/msg06230.html
was your latest opinion.
Second, regardless of the channel signal notification, there are still 
real necessities for userspace hypercall handling:
1. For virtio drivers there is also registration hypercall for passing 
the shared memory pfns.
    Sure there are other possibilities, but why limit ourselves?
2. For other purposes such as a balloon driver, a deflate/inflate 
hypercalls are needed.
    Although for x86 mmio/pio can be used but this is not compatible 
with other architectures.

Regards & thanks for the patch resend,
Dor

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Rusty Russell]

On Sat, 2007-09-15 at 01:31 +0300, Dor Laor wrote:
> Second, regardless of the channel signal notification, there are still 
> real necessities for userspace hypercall handling:
> 1. For virtio drivers there is also registration hypercall for passing 
> the shared memory pfns.
>     Sure there are other possibilities, but why limit ourselves?

I really prefer doing this the more "hardware-like" way and having the
device description say where the pages are.  Surely this is simpler from
the qemu side, too?

I'm working from the lguest side to try to get uniform virtio, and
lguest does it this way.

> 2. For other purposes such as a balloon driver, a deflate/inflate 
> hypercalls are needed.
>     Although for x86 mmio/pio can be used but this is not compatible 
> with other architectures.

We could encode the notification method in the config space  ("execute
this!").

OK, maybe not.  But it'd be generic!

Rusty.



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Avi Kivity]

Dor Laor wrote:
> Avi Kivity wrote:
>   
>> Anthony Liguori wrote:
>>   
>>     
>>> Dor Laor wrote:
>>>   
>>>     
>>>       
>>>>> Hi all,
>>>>>
>>>>> 	I've finally started looking at Dor's git tree, and it struck me
>>>>>     
>>>>>       
>>>>>         
>>>>>           
>>>> that
>>>>   
>>>>     
>>>>       
>>>>         
>>>>> it conflicts with Anthony's hypercall patches.  FWIW I like Anthony's
>>>>> patching thing, and don't really care about arg order.  It'd be nice if
>>>>> we could pull in the same direction tho 8)
>>>>>
>>>>> Thanks,
>>>>> Rusty.
>>>>>     
>>>>>       
>>>>>         
>>>>>           
>>>> Good news you're looking at my tree, since the forum I didn't do much
>>>> since I had to catch up some gazlion other tasks, never the less
>>>> starting on Sunday I'm back again.
>>>>
>>>> Actually, I wanted to rebase my hypercalls over Anhtony's too (except
>>>> for allowing userspace handling).
>>>>     
>>>>       
>>>>         
>>> I thought we discussed just providing a signaling message to userspace 
>>> for virtio?  It's not strictly necessary to expose hypercalls to 
>>> userspace in order to implement a virtio backend in userspace.
>>>
>>>   
>>>     
>>>       
>> Yes, that's what I'd like to see too.  Signal a channel.
>>
>>   
>>     
> First, I though that this 
> http://www.mail-archive.com/kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org/msg06230.html
> was your latest opinion.
>   

The 'signal a channel' method has two sub-options when talking to a
userspace implementation: exit immediately to userspace or signal some
other thread asynchronously.  The second option turned out to be not so
good so we won't be implementing it.  But the first option is still valid.


> Second, regardless of the channel signal notification, there are still 
> real necessities for userspace hypercall handling:
> 1. For virtio drivers there is also registration hypercall for passing 
> the shared memory pfns.
>   

That should be done via the device condig space, either pci or virtbus.

>     Sure there are other possibilities, but why limit ourselves?
> 2. For other purposes such as a balloon driver, a deflate/inflate 
> hypercalls are needed.
>     Although for x86 mmio/pio can be used but this is not compatible 
> with other architectures.
>   

It can be just a signal on the balloon's channel.  This is better than
the current hypercall implementation since it allows the guest to
proceed while the host is allocating memory, a potentially time
consuming operation.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Avi Kivity]

Rusty Russell wrote:
> On Sat, 2007-09-15 at 01:31 +0300, Dor Laor wrote:
>   
>> Second, regardless of the channel signal notification, there are still 
>> real necessities for userspace hypercall handling:
>> 1. For virtio drivers there is also registration hypercall for passing 
>> the shared memory pfns.
>>     Sure there are other possibilities, but why limit ourselves?
>>     
>
> I really prefer doing this the more "hardware-like" way and having the
> device description say where the pages are.  Surely this is simpler from
> the qemu side, too?
>   

This is mmio style (device provides memory).  DMA style (guest provides
memory, device dmas it) is easier for kvm: we don't need to allocate a
new memory slot, and migration is easier.

Most current hardware place the descriptor ring in guest memory, not the
device, since mmio is slower for the cpu than RAM.



-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Rusty Russell]

On Sat, 2007-09-15 at 11:05 +0300, Avi Kivity wrote:
> Rusty Russell wrote:
> > On Sat, 2007-09-15 at 01:31 +0300, Dor Laor wrote:
> >   
> >> Second, regardless of the channel signal notification, there are still 
> >> real necessities for userspace hypercall handling:
> >> 1. For virtio drivers there is also registration hypercall for passing 
> >> the shared memory pfns.
> >>     Sure there are other possibilities, but why limit ourselves?
> >>     
> >
> > I really prefer doing this the more "hardware-like" way and having the
> > device description say where the pages are.  Surely this is simpler from
> > the qemu side, too?
> >   
> 
> This is mmio style (device provides memory).  DMA style (guest provides
> memory, device dmas it) is easier for kvm: we don't need to allocate a
> new memory slot, and migration is easier.
> 
> Most current hardware place the descriptor ring in guest memory, not the
> device, since mmio is slower for the cpu than RAM.

But for virtual devices it seems unnecessarily convoluted: we'd have
some mmio space (or PCI config space?) to tell the device where the ring
buffer is.  That's a little nasty for lguest early printk, too.

I'm looking now at getting lguest to present a PCI bus implementation
(probably changing virtbus to a series of PCI id plus bytes for "config
space").  That's easier with a "device mem" model, since no config info
needs to go back to the host.

OTOH, with the guest memory model, guest can potentially choose size of
ring.  I don't know if that's an actual advantage (but I have code now
which makes the ring size variable).

Rusty.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Avi Kivity]

Rusty Russell wrote:
> On Sat, 2007-09-15 at 11:05 +0300, Avi Kivity wrote:
>   
>> Rusty Russell wrote:
>>     
>>> On Sat, 2007-09-15 at 01:31 +0300, Dor Laor wrote:
>>>   
>>>       
>>>> Second, regardless of the channel signal notification, there are still 
>>>> real necessities for userspace hypercall handling:
>>>> 1. For virtio drivers there is also registration hypercall for passing 
>>>> the shared memory pfns.
>>>>     Sure there are other possibilities, but why limit ourselves?
>>>>     
>>>>         
>>> I really prefer doing this the more "hardware-like" way and having the
>>> device description say where the pages are.  Surely this is simpler from
>>> the qemu side, too?
>>>   
>>>       
>> This is mmio style (device provides memory).  DMA style (guest provides
>> memory, device dmas it) is easier for kvm: we don't need to allocate a
>> new memory slot, and migration is easier.
>>
>> Most current hardware place the descriptor ring in guest memory, not the
>> device, since mmio is slower for the cpu than RAM.
>>     
>
> But for virtual devices it seems unnecessarily convoluted: we'd have
> some mmio space (or PCI config space?) to tell the device where the ring
> buffer is.  That's a little nasty for lguest early printk, too.
>   

I don't see why there is a difference.  With mmio, the host tells the
guest where the ring is.  With dma, the guest tells the host where the
ring is.  In both cases, you need some form of communication (read-only
for mmio, write-only for dma).

For mmio, the mechanism is standardized within pci; for dma it is not,
but it is still just as simple, write to some word in pci config space
and you're done.


If early printk can't handle pci, we can provide a pio port that does
byte-at-a-time output.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Anthony Liguori]

Dor Laor wrote:
> Avi Kivity wrote:
>> Anthony Liguori wrote:
>>  
>>> Dor Laor wrote:
>>>      
>>>>> Hi all,
>>>>>
>>>>>     I've finally started looking at Dor's git tree, and it struck me
>>>>>                   
>>>> that
>>>>            
>>>>> it conflicts with Anthony's hypercall patches.  FWIW I like Anthony's
>>>>> patching thing, and don't really care about arg order.  It'd be 
>>>>> nice if
>>>>> we could pull in the same direction tho 8)
>>>>>
>>>>> Thanks,
>>>>> Rusty.
>>>>>                   
>>>> Good news you're looking at my tree, since the forum I didn't do much
>>>> since I had to catch up some gazlion other tasks, never the less
>>>> starting on Sunday I'm back again.
>>>>
>>>> Actually, I wanted to rebase my hypercalls over Anhtony's too (except
>>>> for allowing userspace handling).
>>>>           
>>> I thought we discussed just providing a signaling message to 
>>> userspace for virtio?  It's not strictly necessary to expose 
>>> hypercalls to userspace in order to implement a virtio backend in 
>>> userspace.
>>>
>>>       
>>
>> Yes, that's what I'd like to see too.  Signal a channel.
>>
>>   
> First, I though that this 
> http://www.mail-archive.com/kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org/msg06230.html
> was your latest opinion.
> Second, regardless of the channel signal notification, there are still 
> real necessities for userspace hypercall handling:
> 1. For virtio drivers there is also registration hypercall for passing 
> the shared memory pfns.
>    Sure there are other possibilities, but why limit ourselves?

Can you elaborate here?  Using a PCI discover mechanism, you've got your 
memory already.  Not point in reinventing PCI with hypercalls.

>
> 2. For other purposes such as a balloon driver, a deflate/inflate 
> hypercalls are needed.
>    Although for x86 mmio/pio can be used but this is not compatible 
> with other architectures.

Isn't a balloon driver just another virtio device?  Rather, it might be 
interesting to build a simple RPC mechanism on top of virtio and do 
things like balloon on top of that.

Regards,

Anthony Liguori

> Regards & thanks for the patch resend,
> Dor


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Dor Laor]

Anthony Liguori wrote:
> Dor Laor wrote:
>> Avi Kivity wrote:
>>> Anthony Liguori wrote:
>>>  
>>>> Dor Laor wrote:
>>>>     
>>>>>> Hi all,
>>>>>>
>>>>>>     I've finally started looking at Dor's git tree, and it struck me
>>>>>>                   
>>>>> that
>>>>>           
>>>>>> it conflicts with Anthony's hypercall patches.  FWIW I like 
>>>>>> Anthony's
>>>>>> patching thing, and don't really care about arg order.  It'd be 
>>>>>> nice if
>>>>>> we could pull in the same direction tho 8)
>>>>>>
>>>>>> Thanks,
>>>>>> Rusty.
>>>>>>                   
>>>>> Good news you're looking at my tree, since the forum I didn't do much
>>>>> since I had to catch up some gazlion other tasks, never the less
>>>>> starting on Sunday I'm back again.
>>>>>
>>>>> Actually, I wanted to rebase my hypercalls over Anhtony's too (except
>>>>> for allowing userspace handling).
>>>>>           
>>>> I thought we discussed just providing a signaling message to 
>>>> userspace for virtio?  It's not strictly necessary to expose 
>>>> hypercalls to userspace in order to implement a virtio backend in 
>>>> userspace.
>>>>
>>>>       
>>>
>>> Yes, that's what I'd like to see too.  Signal a channel.
>>>
>>>   
>> First, I though that this 
>> http://www.mail-archive.com/kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org/msg06230.html 
>>
>> was your latest opinion.
>> Second, regardless of the channel signal notification, there are 
>> still real necessities for userspace hypercall handling:
>> 1. For virtio drivers there is also registration hypercall for 
>> passing the shared memory pfns.
>>    Sure there are other possibilities, but why limit ourselves?
>
> Can you elaborate here?  Using a PCI discover mechanism, you've got 
> your memory already.  Not point in reinventing PCI with hypercalls.
In this case I agree that it should be done using pci/other_bus config 
space.
>
>>
>> 2. For other purposes such as a balloon driver, a deflate/inflate 
>> hypercalls are needed.
>>    Although for x86 mmio/pio can be used but this is not compatible 
>> with other architectures.
>
> Isn't a balloon driver just another virtio device?  Rather, it might 
> be interesting to build a simple RPC mechanism on top of virtio and do 
> things like balloon on top of that.
Currently the balloon driver is not a virtio device but it will become 
one, nevertheless, not all devices must be virtio, and we cannot predict 
all sort
of usages. Even if a device can work over virtio it shouldn't be a 
perquisite.

I have another two new points in favor of userspace hypercall handling:
1. Hypercalls needed for pci pass through devices.
    We have an home grown implementation for pci pass through devices 
that will soon be merged.
     It allows redirecting a physical pci device into a guest.
     The guest kernel issues hypercalls to know whether a device is 
physical or not. It's much more easy to let
     userspace catch them since it is aware of all devices, unlike the 
kernel.

2. Vmexit speeds
   Theoretically, vmcall should be faster than pio/mmio for the bare 
hardware.
   When implementing PV driver, the guest implementation is agnostic to 
the host implementation. For maximum performance
   the host side will use kernel modules while for flexibility a 
userspace implementation will do the job.
   So although vmcall efficiency is neglectable comparing to context 
switch to user, there will be occasions were the host has a PV driver 
backend.

Does it help changing your minds?
Dor

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Anthony Liguori]

Dor Laor wrote:
>> Can you elaborate here?  Using a PCI discover mechanism, you've got 
>> your memory already.  Not point in reinventing PCI with hypercalls.
>>     
> In this case I agree that it should be done using pci/other_bus config 
> space.
>   
>>> 2. For other purposes such as a balloon driver, a deflate/inflate 
>>> hypercalls are needed.
>>>    Although for x86 mmio/pio can be used but this is not compatible 
>>> with other architectures.
>>>       
>> Isn't a balloon driver just another virtio device?  Rather, it might 
>> be interesting to build a simple RPC mechanism on top of virtio and do 
>> things like balloon on top of that.
>>     
> Currently the balloon driver is not a virtio device but it will become 
> one, nevertheless, not all devices must be virtio, and we cannot predict 
> all sort
> of usages. Even if a device can work over virtio it shouldn't be a 
> perquisite.
>
> I have another two new points in favor of userspace hypercall handling:
> 1. Hypercalls needed for pci pass through devices.
>     We have an home grown implementation for pci pass through devices 
> that will soon be merged.
>      It allows redirecting a physical pci device into a guest.
>      The guest kernel issues hypercalls to know whether a device is 
> physical or not. It's much more easy to let
>      userspace catch them since it is aware of all devices, unlike the 
> kernel.
>   

It's really hard to say here without seeing the code but if you really 
need to use a hypercall, then I think the better thing to do is define 
an higher level interface (like an exit reason) and do the translation 
from hypercall to this exit reason.  There's no performance difference 
doing this.

My thinking is that there will be other userspace other than QEMU.  The 
hypercall interface is static and needs to be treated as the host=>guest 
ABI.  By allowing hypercalls to be interpreted by userspace, you are now 
making the host=>guest ABI depend on userspace too instead of just 
kernel space.

The only argument I can see for passing through hypercalls:

1) you may want two separate userspaces to define the same hypercall 
number in two different ways

2) it's easier to just pass through the hypercall by default than it is 
to translate to a higher level exit reason

I think #1 is fundamentally a bad thing to do allow.  I think #2 is not 
justified because you're just making the hypercall interface part of the 
kernel/userspace interface anyway.

> 2. Vmexit speeds
>    Theoretically, vmcall should be faster than pio/mmio for the bare 
> hardware.
>    When implementing PV driver, the guest implementation is agnostic to 
> the host implementation. For maximum performance
>    the host side will use kernel modules while for flexibility a 
> userspace implementation will do the job.
>    So although vmcall efficiency is neglectable comparing to context 
> switch to user, there will be occasions were the host has a PV driver 
> backend.
>   

Whether to use hypercalls vs. PIO is a separate issue from whether 
hypercalls should be handled in userspace.  I think we should always 
handle hypercalls in kernel space and that the hypercall interface ought 
to be defined within the kernel.  Now, this doesn't mean that the result 
of a hypercall can't be dropping down to userspace but I don't think we 
should do it blindly.

> Does it help changing your minds?
>   

No, but I'm hoping that I can change yours :-)

Regards,

Anthony Liguori

> Dor
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> kvm-devel mailing list
> kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/kvm-devel
>
>   


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Rusty Russell]

On Sat, 2007-09-15 at 12:38 +0300, Avi Kivity wrote:
> I don't see why there is a difference.  With mmio, the host tells the
> guest where the ring is.  With dma, the guest tells the host where the
> ring is.  In both cases, you need some form of communication (read-only
> for mmio, write-only for dma).
> 
> For mmio, the mechanism is standardized within pci; for dma it is not,
> but it is still just as simple, write to some word in pci config space
> and you're done.

No, you already need a r/o, whatever you use.  That's because you need
to describe the features of the device (eg disk size).

> If early printk can't handle pci, we can provide a pio port that does
> byte-at-a-time output.

It's not that it can't handle PCI, it's that it now needs to find a page
to use.  That's less trivial than using an already-existing page.

As for making suspend/resume more complex, I can't see it.  Make the
guest memory a few pages bigger, and don't tell the guest about those
extra pages (that's waht lguest does today: those mmio pages are just
above top of "normal" RAM).

Now, we might want some mmio space for our "kick", rather than a
hypercall, but that's separate from the ring buffers.

Rusty.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Avi Kivity]

Rusty Russell wrote:
> On Sat, 2007-09-15 at 12:38 +0300, Avi Kivity wrote:
>   
>> I don't see why there is a difference.  With mmio, the host tells the
>> guest where the ring is.  With dma, the guest tells the host where the
>> ring is.  In both cases, you need some form of communication (read-only
>> for mmio, write-only for dma).
>>
>> For mmio, the mechanism is standardized within pci; for dma it is not,
>> but it is still just as simple, write to some word in pci config space
>> and you're done.
>>     
>
> No, you already need a r/o, whatever you use.  That's because you need
> to describe the features of the device (eg disk size).
>
>   

I don't get your point (I agree with everthing you said except the 
"No,", so maybe I'm not understanding something).


>> If early printk can't handle pci, we can provide a pio port that does
>> byte-at-a-time output.
>>     
>
> It's not that it can't handle PCI, it's that it now needs to find a page
> to use.  That's less trivial than using an already-existing page.
>   

static char a_page[PAGE_SIZE] __aligned(PAGE_SIZE);

> As for making suspend/resume more complex, I can't see it.  Make the
> guest memory a few pages bigger, and don't tell the guest about those
> extra pages (that's waht lguest does today: those mmio pages are just
> above top of "normal" RAM).
>   

That's annoying for memory or device hotplug  (though not insurmountable).

The vga framebuffer handles this by allocating a separate memory slot; 
the right way if we go to mmio device memory is to have one slot per 
device.  But I still think that if e1000 can allocate the ring in system 
memory, so can virtio.

> Now, we might want some mmio space for our "kick", rather than a
> hypercall, but that's separate from the ring buffers.
>
>   

Sure.


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Rusty Russell]

On Sun, 2007-09-16 at 10:57 +0200, Avi Kivity wrote:
> Rusty Russell wrote:
> > On Sat, 2007-09-15 at 12:38 +0300, Avi Kivity wrote:
> >   
> >> I don't see why there is a difference.  With mmio, the host tells the
> >> guest where the ring is.  With dma, the guest tells the host where the
> >> ring is.  In both cases, you need some form of communication (read-only
> >> for mmio, write-only for dma).
> >>
> >> For mmio, the mechanism is standardized within pci; for dma it is not,
> >> but it is still just as simple, write to some word in pci config space
> >> and you're done.
> >>     
> >
> > No, you already need a r/o, whatever you use.  That's because you need
> > to describe the features of the device (eg disk size).
> >
> >   
> 
> I don't get your point (I agree with everthing you said except the 
> "No,", so maybe I'm not understanding something).

You said "In both cases, you need some form of communication".  True,
but we already need host -> guest.  dma-style adds guest -> host, which
is new.

It is not the simplest solution, and this is a standard we're creating.

> >> If early printk can't handle pci, we can provide a pio port that does
> >> byte-at-a-time output.
> >>     
> >
> > It's not that it can't handle PCI, it's that it now needs to find a page
> > to use.  That's less trivial than using an already-existing page.
> >   
> 
> static char a_page[PAGE_SIZE] __aligned(PAGE_SIZE);

We don't want to waste two pages in a driver which might not be used at
all.

> > As for making suspend/resume more complex, I can't see it.  Make the
> > guest memory a few pages bigger, and don't tell the guest about those
> > extra pages (that's waht lguest does today: those mmio pages are just
> > above top of "normal" RAM).
> >   
> 
> That's annoying for memory or device hotplug  (though not insurmountable).

Sure, and we can get more sophisticated when those happen.

> The vga framebuffer handles this by allocating a separate memory slot; 
> the right way if we go to mmio device memory is to have one slot per 
> device.  But I still think that if e1000 can allocate the ring in system 
> memory, so can virtio.

We *can*, but adding complexity needs justification.  For the e1000,
it's performance.  For virtio, it's "might simplify modifications to
existing kvm-qemu", which is far weaker IMHO.

Cheers,
Rusty.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: virtio & hypercall interface?

[Avi Kivity]

Rusty Russell wrote:
> On Sun, 2007-09-16 at 10:57 +0200, Avi Kivity wrote:
>   
>> Rusty Russell wrote:
>>     
>>> On Sat, 2007-09-15 at 12:38 +0300, Avi Kivity wrote:
>>>   
>>>       
>>>> I don't see why there is a difference.  With mmio, the host tells the
>>>> guest where the ring is.  With dma, the guest tells the host where the
>>>> ring is.  In both cases, you need some form of communication (read-only
>>>> for mmio, write-only for dma).
>>>>
>>>> For mmio, the mechanism is standardized within pci; for dma it is not,
>>>> but it is still just as simple, write to some word in pci config space
>>>> and you're done.
>>>>     
>>>>         
>>> No, you already need a r/o, whatever you use.  That's because you need
>>> to describe the features of the device (eg disk size).
>>>
>>>   
>>>       
>> I don't get your point (I agree with everthing you said except the 
>> "No,", so maybe I'm not understanding something).
>>     
>
> You said "In both cases, you need some form of communication".  True,
> but we already need host -> guest.  dma-style adds guest -> host, which
> is new.
>
> It is not the simplest solution, and this is a standard we're creating.
>   

We already have guest -> host (at least for pci).  We'll need guest -> 
host config anyway, to configure interrupt mitigation, blink the virtual 
NIC LEDs, and stuff like that.

>>>> If early printk can't handle pci, we can provide a pio port that does
>>>> byte-at-a-time output.
>>>>     
>>>>         
>>> It's not that it can't handle PCI, it's that it now needs to find a page
>>> to use.  That's less trivial than using an already-existing page.
>>>   
>>>       
>> static char a_page[PAGE_SIZE] __aligned(PAGE_SIZE);
>>     
>
> We don't want to waste two pages in a driver which might not be used at
> all.
>
>   

If the host allocates it, it is wasted too.

>>> As for making suspend/resume more complex, I can't see it.  Make the
>>> guest memory a few pages bigger, and don't tell the guest about those
>>> extra pages (that's waht lguest does today: those mmio pages are just
>>> above top of "normal" RAM).
>>>   
>>>       
>> That's annoying for memory or device hotplug  (though not insurmountable).
>>     
>
> Sure, and we can get more sophisticated when those happen.
>
>   

When we aim, we should keep away from our feet, so that when we shoot we 
have a chance to keep them.

>> The vga framebuffer handles this by allocating a separate memory slot; 
>> the right way if we go to mmio device memory is to have one slot per 
>> device.  But I still think that if e1000 can allocate the ring in system 
>> memory, so can virtio.
>>     
>
> We *can*, but adding complexity needs justification.  For the e1000,
> it's performance.  For virtio, it's "might simplify modifications to
> existing kvm-qemu", which is far weaker IMHO.
>   

I don't see read/write pci config as complexity; it is part of the pci 
spec and we can't choose to implement the parts we like.  In any case it 
is implemented already.

A point in favor of host allocation, though, is that you can share that 
memory with other guests.  Not sure there's much point since the host 
will need to be involved in translating descriptors anyway.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/