[Qemu-devel] Planning for 0.13

[Qemu-devel] Planning for 0.13

[Anthony Liguori]

Hi,

I hope everyone had a happy new year!  Now that we've finished the 0.12 
release and most of us have had a nice break, I think it's time to start 
planning for the next release.

0.12 felt a bit rushed to me.  I'd like to take a bit more time with 
0.13 and try to complete features a bit more than we did in 0.12.  So I 
propose that we target 0.13 as a 6-month cycle and target a June 1st 
release.  Based on the -rc process for 0.12, I would like to have a two 
week -rc cycle for 0.13 but this time, absolutely no non-bug fixes past 
freeze.  We got sloppy in 0.12 with this and we had some ugly 
regressions in the final release.

Here are the features that I'm aware of for 0.13.  Please add anything 
you're working on and I'll try to centralize this somewhere.

  - gPXE support for virtio-blk
  - Helper based network setup
  - Balloon driver statistics
  - Fully supported QMP
  - Live migration protocol support for subvendor versions (aka features)

Regards,

Anthony Liguori

[Qemu-devel] Re: Planning for 0.13

[Anthony Liguori]

On 01/05/2010 06:43 AM, Anthony Liguori wrote:
> Hi,
>
> I hope everyone had a happy new year!  Now that we've finished the 
> 0.12 release and most of us have had a nice break, I think it's time 
> to start planning for the next release.
>
> 0.12 felt a bit rushed to me.  I'd like to take a bit more time with 
> 0.13 and try to complete features a bit more than we did in 0.12.  So 
> I propose that we target 0.13 as a 6-month cycle and target a June 1st 
> release.  Based on the -rc process for 0.12, I would like to have a 
> two week -rc cycle for 0.13 but this time, absolutely no non-bug fixes 
> past freeze.  We got sloppy in 0.12 with this and we had some ugly 
> regressions in the final release.
>
> Here are the features that I'm aware of for 0.13.  Please add anything 
> you're working on and I'll try to centralize this somewhere.
>
>  - gPXE support for virtio-blk
>  - Helper based network setup
>  - Balloon driver statistics
>  - Fully supported QMP
>  - Live migration protocol support for subvendor versions (aka features)

  - virtio-console
  - vhost-net

> Regards,
>
> Anthony Liguori

Re: [Qemu-devel] Planning for 0.13

[Stefan Weil]

Anthony Liguori schrieb:
> Hi,
>
> I hope everyone had a happy new year!  Now that we've finished the
> 0.12 release and most of us have had a nice break, I think it's time
> to start planning for the next release.
>
> 0.12 felt a bit rushed to me.  I'd like to take a bit more time with
> 0.13 and try to complete features a bit more than we did in 0.12.  So
> I propose that we target 0.13 as a 6-month cycle and target a June 1st
> release.  Based on the -rc process for 0.12, I would like to have a
> two week -rc cycle for 0.13 but this time, absolutely no non-bug fixes
> past freeze.  We got sloppy in 0.12 with this and we had some ugly
> regressions in the final release.
>
> Here are the features that I'm aware of for 0.13.  Please add anything
> you're working on and I'll try to centralize this somewhere.
>
>  - gPXE support for virtio-blk
>  - Helper based network setup
>  - Balloon driver statistics
>  - Fully supported QMP
>  - Live migration protocol support for subvendor versions (aka features)
>
> Regards,
>
> Anthony Liguori

I hope you enjoyed your break and had a happy new year, too.

Here are some features I had been working on which
could be integrated in 0.13:

- eepro100 fixes (high priority because needed by many users)
- tcg interpreter (medium priority because it extends the range
  of supported hosts)
- installer for windows (I'm afraid that windows priority is low
  because nobody loves it very much)

The code for these features is available here:
http://repo.or.cz/w/qemu/ar7.git.

Regards,

Stefan Weil

[Qemu-devel] Re: Planning for 0.13

[Michael S. Tsirkin]

On Tue, Jan 05, 2010 at 06:43:11AM -0600, Anthony Liguori wrote:
> Hi,
>
> I hope everyone had a happy new year!  Now that we've finished the 0.12  
> release and most of us have had a nice break, I think it's time to start  
> planning for the next release.
>
> 0.12 felt a bit rushed to me.  I'd like to take a bit more time with  
> 0.13 and try to complete features a bit more than we did in 0.12.  So I  
> propose that we target 0.13 as a 6-month cycle and target a June 1st  
> release.  Based on the -rc process for 0.12, I would like to have a two  
> week -rc cycle for 0.13 but this time, absolutely no non-bug fixes past  
> freeze.  We got sloppy in 0.12 with this and we had some ugly  
> regressions in the final release.
>
> Here are the features that I'm aware of for 0.13.  Please add anything  
> you're working on and I'll try to centralize this somewhere.
>
>  - gPXE support for virtio-blk
>  - Helper based network setup
>  - Balloon driver statistics
>  - Fully supported QMP
>  - Live migration protocol support for subvendor versions (aka features)

pci compliance (virtio passing whql)
vepa networking
kvm irqchip

> Regards,
>
> Anthony Liguori
>

Re: [Qemu-devel] Planning for 0.13

[Aurelien Jarno]

On Tue, Jan 05, 2010 at 06:43:11AM -0600, Anthony Liguori wrote:
> Hi,
>
> I hope everyone had a happy new year!  Now that we've finished the 0.12  
> release and most of us have had a nice break, I think it's time to start  
> planning for the next release.
>
> 0.12 felt a bit rushed to me.  I'd like to take a bit more time with  
> 0.13 and try to complete features a bit more than we did in 0.12.  So I  
> propose that we target 0.13 as a 6-month cycle and target a June 1st  
> release.  Based on the -rc process for 0.12, I would like to have a two  
> week -rc cycle for 0.13 but this time, absolutely no non-bug fixes past  
> freeze.  We got sloppy in 0.12 with this and we had some ugly  
> regressions in the final release.
>
> Here are the features that I'm aware of for 0.13.  Please add anything  
> you're working on and I'll try to centralize this somewhere.
>
>  - gPXE support for virtio-blk
>  - Helper based network setup
>  - Balloon driver statistics
>  - Fully supported QMP
>  - Live migration protocol support for subvendor versions (aka features)

   - TCG IA64 support. I have something mostly working in system mode, 
     so it should not be a problem for June.

-- 
Aurelien Jarno	                        GPG: 1024D/F1BCDB73
aurelien@aurel32.net                 http://www.aurel32.net

Re: [Qemu-devel] Re: Planning for 0.13

[Anthony Liguori]

On 01/05/2010 03:33 PM, Michael S. Tsirkin wrote:
> On Tue, Jan 05, 2010 at 06:43:11AM -0600, Anthony Liguori wrote:
>    
>> Hi,
>>
>> I hope everyone had a happy new year!  Now that we've finished the 0.12
>> release and most of us have had a nice break, I think it's time to start
>> planning for the next release.
>>
>> 0.12 felt a bit rushed to me.  I'd like to take a bit more time with
>> 0.13 and try to complete features a bit more than we did in 0.12.  So I
>> propose that we target 0.13 as a 6-month cycle and target a June 1st
>> release.  Based on the -rc process for 0.12, I would like to have a two
>> week -rc cycle for 0.13 but this time, absolutely no non-bug fixes past
>> freeze.  We got sloppy in 0.12 with this and we had some ugly
>> regressions in the final release.
>>
>> Here are the features that I'm aware of for 0.13.  Please add anything
>> you're working on and I'll try to centralize this somewhere.
>>
>>   - gPXE support for virtio-blk
>>   - Helper based network setup
>>   - Balloon driver statistics
>>   - Fully supported QMP
>>   - Live migration protocol support for subvendor versions (aka features)
>>      
> pci compliance (virtio passing whql)
>    

What's the remaining problem?

> vepa networking
>    

To me, this is covered with helpers.  I really want to get qemu out of 
the network setup business specifically because of things like vepa, 
vmtag, and all of the other weird things that can be done.

> kvm irqchip
>    

Yup, that and guest SMP support.

Regards,

Anthony Liguori

Re: [Qemu-devel] Planning for 0.13

[Roy Tam]

2010/1/5 Anthony Liguori <aliguori@linux.vnet.ibm.com>:
> Hi,
>
> I hope everyone had a happy new year!  Now that we've finished the 0.12
> release and most of us have had a nice break, I think it's time to start
> planning for the next release.
>
> 0.12 felt a bit rushed to me.  I'd like to take a bit more time with 0.13
> and try to complete features a bit more than we did in 0.12.  So I propose
> that we target 0.13 as a 6-month cycle and target a June 1st release.  Based
> on the -rc process for 0.12, I would like to have a two week -rc cycle for
> 0.13 but this time, absolutely no non-bug fixes past freeze.  We got sloppy
> in 0.12 with this and we had some ugly regressions in the final release.
>
> Here are the features that I'm aware of for 0.13.  Please add anything
> you're working on and I'll try to centralize this somewhere.
>
>  - gPXE support for virtio-blk
>  - Helper based network setup
>  - Balloon driver statistics
>  - Fully supported QMP
>  - Live migration protocol support for subvendor versions (aka features)

for me,
- NEC PC-9821 support
hope Mr. Takeda can improve his patch set and push it to git head
before 0.13 release.

>
> Regards,
>
> Anthony Liguori
>
>
>

Re: [Qemu-devel] Planning for 0.13

[Gerd Hoffmann]

On 01/05/10 13:43, Anthony Liguori wrote:
> - gPXE support for virtio-blk

likewise for lsi, not sure how hard this is though.
maybe figuring something sane way to extboot is easier.

> - Helper based network setup
> - Balloon driver statistics
> - Fully supported QMP
> - Live migration protocol support for subvendor versions (aka features)

- qdev conversion continued.
- scsi needs some more love, megasas driver.
- p35 + pcie.

cheers,
   Gerd

Re: [Qemu-devel] Re: Planning for 0.13

[Michael S. Tsirkin]

On Tue, Jan 05, 2010 at 06:32:23PM -0600, Anthony Liguori wrote:
> On 01/05/2010 03:33 PM, Michael S. Tsirkin wrote:
>> On Tue, Jan 05, 2010 at 06:43:11AM -0600, Anthony Liguori wrote:
>>    
>>> Hi,
>>>
>>> I hope everyone had a happy new year!  Now that we've finished the 0.12
>>> release and most of us have had a nice break, I think it's time to start
>>> planning for the next release.
>>>
>>> 0.12 felt a bit rushed to me.  I'd like to take a bit more time with
>>> 0.13 and try to complete features a bit more than we did in 0.12.  So I
>>> propose that we target 0.13 as a 6-month cycle and target a June 1st
>>> release.  Based on the -rc process for 0.12, I would like to have a two
>>> week -rc cycle for 0.13 but this time, absolutely no non-bug fixes past
>>> freeze.  We got sloppy in 0.12 with this and we had some ugly
>>> regressions in the final release.
>>>
>>> Here are the features that I'm aware of for 0.13.  Please add anything
>>> you're working on and I'll try to centralize this somewhere.
>>>
>>>   - gPXE support for virtio-blk
>>>   - Helper based network setup
>>>   - Balloon driver statistics
>>>   - Fully supported QMP
>>>   - Live migration protocol support for subvendor versions (aka features)
>>>      
>> pci compliance (virtio passing whql)
>>    
>
> What's the remaining problem?

IIRC, proper memory/IO access filtering (get rid of map functions) and
PCI Express.

>> vepa networking
>>    
>
> To me, this is covered with helpers.  I really want to get qemu out of  
> the network setup business specifically because of things like vepa,  
> vmtag, and all of the other weird things that can be done.

I don't think you can now make vepa work this way.  For existing
kernels, they only way I see is using packet sockets, and that code
already mostly works.  One day, when macvtap is ready - who knows. But
waiting for that would mean we won't have it in 0.13.

>> kvm irqchip
>>    
>
> Yup, that and guest SMP support.
>
> Regards,
>
> Anthony Liguori

Re: [Qemu-devel] Re: Planning for 0.13

[Anthony Liguori]

On 01/06/2010 04:49 AM, Michael S. Tsirkin wrote:
>> What's the remaining problem?
>>      
> IIRC, proper memory/IO access filtering (get rid of map functions) and
> PCI Express.
>
>    
>>> vepa networking
>>>
>>>        
>> To me, this is covered with helpers.  I really want to get qemu out of
>> the network setup business specifically because of things like vepa,
>> vmtag, and all of the other weird things that can be done.
>>      
> I don't think you can now make vepa work this way.  For existing
> kernels, they only way I see is using packet sockets, and that code
> already mostly works.  One day, when macvtap is ready - who knows. But
> waiting for that would mean we won't have it in 0.13.
>    

We can use helpers for more than just tun/tap.  My current thinking for 
helpers is that they would give qemu an fd and then tell qemu how to 
work with it.  Basically, use read/write vs. send/recv, whether to use a 
virtio-net header or not, etc.

That would allow a helper to open a raw socket, configure macvlan, and 
then hand the fd over to qemu and tell qemu how to use it.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: Planning for 0.13

[Michael S. Tsirkin]

On Wed, Jan 06, 2010 at 06:36:26AM -0600, Anthony Liguori wrote:
> On 01/06/2010 04:49 AM, Michael S. Tsirkin wrote:
>>> What's the remaining problem?
>>>      
>> IIRC, proper memory/IO access filtering (get rid of map functions) and
>> PCI Express.
>>
>>    
>>>> vepa networking
>>>>
>>>>        
>>> To me, this is covered with helpers.  I really want to get qemu out of
>>> the network setup business specifically because of things like vepa,
>>> vmtag, and all of the other weird things that can be done.
>>>      
>> I don't think you can now make vepa work this way.  For existing
>> kernels, they only way I see is using packet sockets, and that code
>> already mostly works.  One day, when macvtap is ready - who knows. But
>> waiting for that would mean we won't have it in 0.13.
>>    
>
> We can use helpers for more than just tun/tap.  My current thinking for  
> helpers is that they would give qemu an fd and then tell qemu how to  
> work with it.  Basically, use read/write vs. send/recv, whether to use a  
> virtio-net header or not, etc.

Frankly I think this is too ambitious for 0.13, and I would like to
avoid typing features that users need today to this effort.

Note that management still needs ability to hand fd to qemu, so we can
not require use of helpers for everyone.  If the helpers are part of
qemu itself, we do not gain anything from them besides (limited)
security.  But if not, we also get a protocol qemu<->helpers to
maintain. Ugh.

What I think is reasonable for 0.13, is what you posted: just allow
helper script as an alternative way to get device fd, and have qemu do
all the querying and feature negotiation exactly the way it already
does.  No protocol to maintain, command line users get some extra
security, management is not affected at all. The only risk is that a new
suid binary is installed.

> That would allow a helper to open a raw socket, configure macvlan, and  
> then hand the fd over to qemu and tell qemu how to use it.

Note binding to macvlan in a script buys you zero extra security
as compared to opening socket and binding in qemu.

> Regards,
>
> Anthony Liguori

-- 
MST

Re: [Qemu-devel] Re: Planning for 0.13

[Anthony Liguori]

On 01/06/2010 07:20 AM, Michael S. Tsirkin wrote:
>> We can use helpers for more than just tun/tap.  My current thinking for
>> helpers is that they would give qemu an fd and then tell qemu how to
>> work with it.  Basically, use read/write vs. send/recv, whether to use a
>> virtio-net header or not, etc.
>>      
> Frankly I think this is too ambitious for 0.13, and I would like to
> avoid typing features that users need today to this effort.
>
> Note that management still needs ability to hand fd to qemu, so we can
> not require use of helpers for everyone.

It's the same mechanism, no?

I want to move to a single net backend that would be something like -net 
fd.  Here are some possible invocations:

-net fd,fd=3,type=tun,vnet=off
-net fd,helper="/usr/libexec/qemu-net-vepa --arg-if=eth0"
-net fd,fd=3,type=socket,vnet=off
-net fd,fd=3,type=vhost

It's really a simple thing to do and it means that we can always 
implement any backend outside of qemu.  As part of this, I would like:

-net vepa,if=eth0

To automatically translate to:

-net fd,helper="/usr/libexec/qemu-net-vepa --arg-if=eth0"

I'm also open to the idea of using shared libraries if people really 
think it's a good idea.

>    If the helpers are part of
> qemu itself, we do not gain anything from them besides (limited)
> security.  But if not, we also get a protocol qemu<->helpers to
> maintain. Ugh.
>    

There really isn't much a protocol here.  Helpers get handed a domain 
socket, then connect and send an fd via SCM_RIGHTS.  They pass a string 
as part of that message that just happens to be equivalent to the arg 
string that would normally be passed to -net fd.

> What I think is reasonable for 0.13, is what you posted: just allow
> helper script as an alternative way to get device fd, and have qemu do
> all the querying and feature negotiation exactly the way it already
> does.  No protocol to maintain, command line users get some extra
> security, management is not affected at all. The only risk is that a new
> suid binary is installed.
>    

The whole suid binary thing is someone orthogonal to my goal here.  The 
observation is that 99% of what people want in terms of network backends 
really just boils down to, here's a file descriptor, interact with it in 
one of a very small number of ways.

>> That would allow a helper to open a raw socket, configure macvlan, and
>> then hand the fd over to qemu and tell qemu how to use it.
>>      
> Note binding to macvlan in a script buys you zero extra security
> as compared to opening socket and binding in qemu.
>    

It's not about security, it's about not making qemu the gateway to 
implementing arbitrarily complex network mechanisms.  There's no reason 
qemu should have to know anything about vepa, for instance.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: Planning for 0.13

[Michael S. Tsirkin]

On Wed, Jan 06, 2010 at 07:34:26AM -0600, Anthony Liguori wrote:
> On 01/06/2010 07:20 AM, Michael S. Tsirkin wrote:
>>> We can use helpers for more than just tun/tap.  My current thinking for
>>> helpers is that they would give qemu an fd and then tell qemu how to
>>> work with it.  Basically, use read/write vs. send/recv, whether to use a
>>> virtio-net header or not, etc.
>>>      
>> Frankly I think this is too ambitious for 0.13, and I would like to
>> avoid typing features that users need today to this effort.
>>
>> Note that management still needs ability to hand fd to qemu, so we can
>> not require use of helpers for everyone.
>
> It's the same mechanism, no?
> 
> I want to move to a single net backend that would be something like -net  
> fd.  Here are some possible invocations:
>
> -net fd,fd=3,type=tun,vnet=off
> -net fd,helper="/usr/libexec/qemu-net-vepa --arg-if=eth0"
> -net fd,fd=3,type=socket,vnet=off

We currently don't let users control whether vnet header
is activated in tap and IMO we are better off this way,
let qemu find out support itself.

> -net fd,fd=3,type=vhost
>
> It's really a simple thing to do and it means that we can always  
> implement any backend outside of qemu.

Look at existing backends, each of them has some quirks in qemu. It's
not realistic to expect that future backends won't need any more.

> As part of this, I would like:
>
> -net vepa,if=eth0
>
> To automatically translate to:
>
> -net fd,helper="/usr/libexec/qemu-net-vepa --arg-if=eth0"
>
> I'm also open to the idea of using shared libraries if people really  
> think it's a good idea.

What does all of this buy us?  The helpers will still need to be shipped
with qemu.

>>    If the helpers are part of
>> qemu itself, we do not gain anything from them besides (limited)
>> security.  But if not, we also get a protocol qemu<->helpers to
>> maintain. Ugh.
>>    
>
> There really isn't much a protocol here.  Helpers get handed a domain  
> socket, then connect and send an fd via SCM_RIGHTS.  They pass a string  
> as part of that message that just happens to be equivalent to the arg  
> string that would normally be passed to -net fd.

How do helpers know which arguments are legal?  Also, e.g. with
vhost-net you can open it in a helper script but you must do the rest of
the set up in qemu.


>> What I think is reasonable for 0.13, is what you posted: just allow
>> helper script as an alternative way to get device fd, and have qemu do
>> all the querying and feature negotiation exactly the way it already
>> does.  No protocol to maintain, command line users get some extra
>> security, management is not affected at all. The only risk is that a new
>> suid binary is installed.
>>    
>
> The whole suid binary thing is someone orthogonal to my goal here.  The  
> observation is that 99% of what people want in terms of network backends  
> really just boils down to, here's a file descriptor, interact with it in  
> one of a very small number of ways.

Each new backend seems to have its own quirks. Nothing seems
gained by moving handling them around.


>>> That would allow a helper to open a raw socket, configure macvlan, and
>>> then hand the fd over to qemu and tell qemu how to use it.
>>>      
>> Note binding to macvlan in a script buys you zero extra security
>> as compared to opening socket and binding in qemu.
>>    
>
> It's not about security, it's about not making qemu the gateway to  
> implementing arbitrarily complex network mechanisms.  There's no reason  
> qemu should have to know anything about vepa, for instance.
>
> Regards,
>
> Anthony Liguori

We'll still need to write all the scripts and bundle them
with qemu. So ... I fail to see 

-- 
MST

Re: [Qemu-devel] Re: Planning for 0.13

[Anthony Liguori]

On 01/06/2010 07:55 AM, Michael S. Tsirkin wrote:
> On Wed, Jan 06, 2010 at 07:34:26AM -0600, Anthony Liguori wrote:
>    
>> On 01/06/2010 07:20 AM, Michael S. Tsirkin wrote:
>>      
>>>> We can use helpers for more than just tun/tap.  My current thinking for
>>>> helpers is that they would give qemu an fd and then tell qemu how to
>>>> work with it.  Basically, use read/write vs. send/recv, whether to use a
>>>> virtio-net header or not, etc.
>>>>
>>>>          
>>> Frankly I think this is too ambitious for 0.13, and I would like to
>>> avoid typing features that users need today to this effort.
>>>
>>> Note that management still needs ability to hand fd to qemu, so we can
>>> not require use of helpers for everyone.
>>>        
>> It's the same mechanism, no?
>>
>> I want to move to a single net backend that would be something like -net
>> fd.  Here are some possible invocations:
>>
>> -net fd,fd=3,type=tun,vnet=off
>> -net fd,helper="/usr/libexec/qemu-net-vepa --arg-if=eth0"
>> -net fd,fd=3,type=socket,vnet=off
>>      
> We currently don't let users control whether vnet header
> is activated in tap and IMO we are better off this way,
> let qemu find out support itself.
>    

-net tap,vnet_hdr=off

>> -net fd,fd=3,type=vhost
>>
>> It's really a simple thing to do and it means that we can always
>> implement any backend outside of qemu.
>>      
> Look at existing backends, each of them has some quirks in qemu. It's
> not realistic to expect that future backends won't need any more.
>    

Quirks in the send/receive paths?  Can you be specific because I don't 
really think there are.

>> As part of this, I would like:
>>
>> -net vepa,if=eth0
>>
>> To automatically translate to:
>>
>> -net fd,helper="/usr/libexec/qemu-net-vepa --arg-if=eth0"
>>
>> I'm also open to the idea of using shared libraries if people really
>> think it's a good idea.
>>      
> What does all of this buy us?  The helpers will still need to be shipped
> with qemu.
>    

Why?  They can be separate packages that are maintained on their own and 
at their own pace.

>> There really isn't much a protocol here.  Helpers get handed a domain
>> socket, then connect and send an fd via SCM_RIGHTS.  They pass a string
>> as part of that message that just happens to be equivalent to the arg
>> string that would normally be passed to -net fd.
>>      
> How do helpers know which arguments are legal?

There is a set number of arguments that we support in qemu.. I think 
this is the sort of thing that is easier to discuss with code in hand.

>    Also, e.g. with
> vhost-net you can open it in a helper script but you must do the rest of
> the set up in qemu.
>    

AFAICT, we have a hand full of types of fds.  We have ones that require 
read/write, ones that require send/recv, and ones that require vhost 
interaction.  Really, the first two are the same but the distinction is 
necessary for Windows.
>>>> That would allow a helper to open a raw socket, configure macvlan, and
>>>> then hand the fd over to qemu and tell qemu how to use it.
>>>>
>>>>          
>>> Note binding to macvlan in a script buys you zero extra security
>>> as compared to opening socket and binding in qemu.
>>>
>>>        
>> It's not about security, it's about not making qemu the gateway to
>> implementing arbitrarily complex network mechanisms.  There's no reason
>> qemu should have to know anything about vepa, for instance.
>>
>> Regards,
>>
>> Anthony Liguori
>>      
> We'll still need to write all the scripts and bundle them
> with qemu. So ... I fail to see
>    

Again, I don't see why it needs to be bundled with qemu.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: Planning for 0.13

[Michael S. Tsirkin]

On Wed, Jan 06, 2010 at 09:10:30AM -0600, Anthony Liguori wrote:
> On 01/06/2010 07:55 AM, Michael S. Tsirkin wrote:
>> On Wed, Jan 06, 2010 at 07:34:26AM -0600, Anthony Liguori wrote:
>>    
>>> On 01/06/2010 07:20 AM, Michael S. Tsirkin wrote:
>>>      
>>>>> We can use helpers for more than just tun/tap.  My current thinking for
>>>>> helpers is that they would give qemu an fd and then tell qemu how to
>>>>> work with it.  Basically, use read/write vs. send/recv, whether to use a
>>>>> virtio-net header or not, etc.
>>>>>
>>>>>          
>>>> Frankly I think this is too ambitious for 0.13, and I would like to
>>>> avoid typing features that users need today to this effort.
>>>>
>>>> Note that management still needs ability to hand fd to qemu, so we can
>>>> not require use of helpers for everyone.
>>>>        
>>> It's the same mechanism, no?
>>>
>>> I want to move to a single net backend that would be something like -net
>>> fd.  Here are some possible invocations:
>>>
>>> -net fd,fd=3,type=tun,vnet=off
>>> -net fd,helper="/usr/libexec/qemu-net-vepa --arg-if=eth0"
>>> -net fd,fd=3,type=socket,vnet=off
>>>      
>> We currently don't let users control whether vnet header
>> is activated in tap and IMO we are better off this way,
>> let qemu find out support itself.
>>    
>
> -net tap,vnet_hdr=off
>
>>> -net fd,fd=3,type=vhost
>>>
>>> It's really a simple thing to do and it means that we can always
>>> implement any backend outside of qemu.
>>>      
>> Look at existing backends, each of them has some quirks in qemu. It's
>> not realistic to expect that future backends won't need any more.
>>    
>
> Quirks in the send/receive paths?  Can you be specific because I don't  
> really think there are.

You listed them yourself. read/write versus send/recv,
vnet header.

>>> As part of this, I would like:
>>>
>>> -net vepa,if=eth0
>>>
>>> To automatically translate to:
>>>
>>> -net fd,helper="/usr/libexec/qemu-net-vepa --arg-if=eth0"
>>>
>>> I'm also open to the idea of using shared libraries if people really
>>> think it's a good idea.
>>>      
>> What does all of this buy us?  The helpers will still need to be shipped
>> with qemu.
>>    
>
> Why?  They can be separate packages that are maintained on their own and  
> at their own pace.

Then they need to support more than one qemu version,
it gets messy real fast.

>>> There really isn't much a protocol here.  Helpers get handed a domain
>>> socket, then connect and send an fd via SCM_RIGHTS.  They pass a string
>>> as part of that message that just happens to be equivalent to the arg
>>> string that would normally be passed to -net fd.
>>>      
>> How do helpers know which arguments are legal?
>
> There is a set number of arguments that we support in qemu.. I think  
> this is the sort of thing that is easier to discuss with code in hand.
>
>>    Also, e.g. with
>> vhost-net you can open it in a helper script but you must do the rest of
>> the set up in qemu.
>>    
>
> AFAICT, we have a hand full of types of fds.  We have ones that require  
> read/write, ones that require send/recv, and ones that require vhost  
> interaction.  Really, the first two are the same but the distinction is  
> necessary for Windows.

Not only for windows. With packet sockets you must use send/recv to
supply socket protocol. You must query tap to find out about UFO
support. There will be more.

>>>>> That would allow a helper to open a raw socket, configure macvlan, and
>>>>> then hand the fd over to qemu and tell qemu how to use it.
>>>>>
>>>>>          
>>>> Note binding to macvlan in a script buys you zero extra security
>>>> as compared to opening socket and binding in qemu.
>>>>
>>>>        
>>> It's not about security, it's about not making qemu the gateway to
>>> implementing arbitrarily complex network mechanisms.  There's no reason
>>> qemu should have to know anything about vepa, for instance.
>>>
>>> Regards,
>>>
>>> Anthony Liguori
>>>      
>> We'll still need to write all the scripts and bundle them
>> with qemu. So ... I fail to see
>>    
>
> Again, I don't see why it needs to be bundled with qemu.
>
> Regards,
>
> Anthony Liguori


How otherwise would scripts know how to talk to qemu?
Just just happens to match command line format you say?
And the way to discover what that format is ... how exactly?

Look, yes we could split this stuff out but this is just maintainance
headache, each change in backend will now need to be done in multiple
places, we'll have to care about old scripts, new scripts, it's just a
mess and at the end we will get existing functionality back and codebase
which is harder to debug and develop.

-- 
MST

Re: [Qemu-devel] Re: Planning for 0.13

[Anthony Liguori]

On 01/06/2010 09:16 AM, Michael S. Tsirkin wrote:
> How otherwise would scripts know how to talk to qemu?
> Just just happens to match command line format you say?
> And the way to discover what that format is ... how exactly?
>
> Look, yes we could split this stuff out but this is just maintainance
> headache, each change in backend will now need to be done in multiple
> places, we'll have to care about old scripts, new scripts, it's just a
> mess and at the end we will get existing functionality back and codebase
> which is harder to debug and develop.
>    

A helper is semantics equivalent to passing an fd from a management 
tool.  All of the problems you describe are equally applicable to that 
model.

The question is, should we take in code in qemu to support any possible 
mechanism of creation of networking or should we just make sure their 
all possible by passing in an appropriate fd.

Having helpers does not mean that we would have no backends built into 
qemu.  It just means that's it's possible to create backends outside of 
qemu.

Of course, we need to evalute whether a new backend should be in qemu or 
outside of qemu but that's something to handle on a case-by-case basis.

Regards,

Anthony Liguori

Re: [Qemu-devel] Planning for 0.13

[Adam Litke]

On Tue, 2010-01-05 at 06:43 -0600, Anthony Liguori wrote:
> Hi,
> 
> I hope everyone had a happy new year!  Now that we've finished the 0.12 
> release and most of us have had a nice break, I think it's time to start 
> planning for the next release.
> 
> 0.12 felt a bit rushed to me.  I'd like to take a bit more time with 
> 0.13 and try to complete features a bit more than we did in 0.12.  So I 
> propose that we target 0.13 as a 6-month cycle and target a June 1st 
> release.  Based on the -rc process for 0.12, I would like to have a two 
> week -rc cycle for 0.13 but this time, absolutely no non-bug fixes past 
> freeze.  We got sloppy in 0.12 with this and we had some ugly 
> regressions in the final release.
> 
> Here are the features that I'm aware of for 0.13.  Please add anything 
> you're working on and I'll try to centralize this somewhere.

Port the -mem-path and -mem-prealloc options from qemu-kvm.
virtio-balloon enhancements for -mem-path enabled VMs.

-- 
Thanks,
Adam

Re: [Qemu-devel] Re: Planning for 0.13

[Michael S. Tsirkin]

On Wed, Jan 06, 2010 at 09:24:45AM -0600, Anthony Liguori wrote:
> On 01/06/2010 09:16 AM, Michael S. Tsirkin wrote:
>> How otherwise would scripts know how to talk to qemu?
>> Just just happens to match command line format you say?
>> And the way to discover what that format is ... how exactly?
>>
>> Look, yes we could split this stuff out but this is just maintainance
>> headache, each change in backend will now need to be done in multiple
>> places, we'll have to care about old scripts, new scripts, it's just a
>> mess and at the end we will get existing functionality back and codebase
>> which is harder to debug and develop.
>>    
>
> A helper is semantics equivalent to passing an fd from a management  
> tool.  All of the problems you describe are equally applicable to that  
> model.

No, because management calls qemu and parses qemu help output.  Yes it
is not ideal but it works today.

> The question is, should we take in code in qemu to support any possible  
> mechanism of creation of networking or should we just make sure their  
> all possible by passing in an appropriate fd.

We already do this. What will not work generally is *returning* fd from
helper. And IMO we are better off not pretending it's possible.

> Having helpers does not mean that we would have no backends built into  
> qemu.  It just means that's it's possible to create backends outside of  
> qemu.
> 
> Of course, we need to evalute whether a new backend should be in qemu or  
> outside of qemu but that's something to handle on a case-by-case basis.
>
> Regards,
>
> Anthony Liguori


To the point, I think we are better off with packet socket (vepa)
backend in qemu than as a helper script.

-- 
MST

Re: [Qemu-devel] Re: Planning for 0.13

[Jamie Lokier]

Michael S. Tsirkin wrote:
> On Wed, Jan 06, 2010 at 09:24:45AM -0600, Anthony Liguori wrote:
> > A helper is semantics equivalent to passing an fd from a management  
> > tool.  All of the problems you describe are equally applicable to that  
> > model.
> 
> No, because management calls qemu and parses qemu help output.  Yes it
> is not ideal but it works today.

I don't understand.  What do you think would not work with
helper="..."  where ... is specified on the qemu command line by the
management script, versus the management script doing the helper
operations itself first and then calling qemu with fd=?

If you are thinking that management scripts will tailor the -net
arguments according to qemu version, you're right for some
configurations (but not well established simple ones).

Presumably management can do the same capability when specifying "..."
- the difference being it would query the helper tool to get _it's_
features in some cases, e.g. for arguments to a helper which uses SSH
to provide an encrypted tunnel.

> > The question is, should we take in code in qemu to support any possible  
> > mechanism of creation of networking or should we just make sure their  
> > all possible by passing in an appropriate fd.
> 
> We already do this. What will not work generally is *returning* fd from
> helper. And IMO we are better off not pretending it's possible.

What about it will not work?  Even on Windows, I don't see why -net
this,that,other,helper="..." cannot be a direct equivalent for -net
this,that,other,fd=N, for any combination of this,that,other options -
with the added bonus that the helper would be allowed to provide
additional options to QEMU if wanted.

> > Having helpers does not mean that we would have no backends built into  
> > qemu.  It just means that's it's possible to create backends outside of  
> > qemu.
> > 
> > Of course, we need to evalute whether a new backend should be in qemu or  
> > outside of qemu but that's something to handle on a case-by-case basis.
> >
> > Regards,
> >
> > Anthony Liguori
> 
> To the point, I think we are better off with packet socket (vepa)
> backend in qemu than as a helper script.

That one, yes, but with the helper= option being more or less
equivalent to fd= with the added ability to tell qemu how it wants
qemu to talk to the fd, it's a bit easier to have user-supplied
helpers such as:

    - Build an encrypted tunnel with SSH
    - Log all packets
    - Fake packets with a Perl script for repeatable tests
    - Send packets through a network simulator
    - Site-specific bridge + iptables setup

You don't want code for those sort of things in qemu itself.

Same, really, could be imagined with -monitor, -serial etc. -
providing a generic "helper" backend in the same way we support
connecting to serial ports, telnet sockets etc.

Btw, as of right now, I have not found a management tool which sets up
bridges correctly for my sites...  There is always something extra
needed with iptables, so it has to be done with hand-holding, or with
the script= and downscript= options - which are annoyingly fragile
because downscript isn't run if qemu has to be killed.

A helper which communicates its result back to qemu, and then *keeps
the unix socket open* would be a nice way to reliably detect when the
helper should destroy whatever it created - more reliable than downscript=.

I agree many backends are better implemented in qemu proper, but
Anthony's idea sounds simple and versatile to me, and I would
certainly use it for site-specific things.

-- Jamie

Re: [Qemu-devel] Re: Planning for 0.13

[Michael S. Tsirkin]

On Wed, Jan 06, 2010 at 05:19:45PM +0000, Jamie Lokier wrote:
> Michael S. Tsirkin wrote:
> > On Wed, Jan 06, 2010 at 09:24:45AM -0600, Anthony Liguori wrote:
> > > A helper is semantics equivalent to passing an fd from a management  
> > > tool.  All of the problems you describe are equally applicable to that  
> > > model.
> > 
> > No, because management calls qemu and parses qemu help output.  Yes it
> > is not ideal but it works today.
> 
> I don't understand.  What do you think would not work with
> helper="..."  where ... is specified on the qemu command line by the
> management script, versus the management script doing the helper
> operations itself first and then calling qemu with fd=?
> 
> If you are thinking that management scripts will tailor the -net
> arguments according to qemu version, you're right for some
> configurations (but not well established simple ones).
> 
> Presumably management can do the same capability when specifying "..."
> - the difference being it would query the helper tool to get _it's_
> features in some cases, e.g. for arguments to a helper which uses SSH
> to provide an encrypted tunnel.

What won't work is that besides fd= management specifies many other
arguments to the -net flag.

> > > The question is, should we take in code in qemu to support any possible  
> > > mechanism of creation of networking or should we just make sure their  
> > > all possible by passing in an appropriate fd.
> > 
> > We already do this. What will not work generally is *returning* fd from
> > helper. And IMO we are better off not pretending it's possible.
> 
> What about it will not work?  Even on Windows, I don't see why -net
> this,that,other,helper="..." cannot be a direct equivalent for -net
> this,that,other,fd=N, for any combination of this,that,other options -
> with the added bonus that the helper would be allowed to provide
> additional options to QEMU if wanted.
> 
> > > Having helpers does not mean that we would have no backends built into  
> > > qemu.  It just means that's it's possible to create backends outside of  
> > > qemu.
> > > 
> > > Of course, we need to evalute whether a new backend should be in qemu or  
> > > outside of qemu but that's something to handle on a case-by-case basis.
> > >
> > > Regards,
> > >
> > > Anthony Liguori
> > 
> > To the point, I think we are better off with packet socket (vepa)
> > backend in qemu than as a helper script.
> 
> That one, yes, but with the helper= option being more or less
> equivalent to fd= with the added ability to tell qemu how it wants
> qemu to talk to the fd, it's a bit easier to have user-supplied
> helpers such as:
> 
>     - Build an encrypted tunnel with SSH
>     - Log all packets
>     - Fake packets with a Perl script for repeatable tests
>     - Send packets through a network simulator
>     - Site-specific bridge + iptables setup
> 
> You don't want code for those sort of things in qemu itself.
> 
> Same, really, could be imagined with -monitor, -serial etc. -
> providing a generic "helper" backend in the same way we support
> connecting to serial ports, telnet sockets etc.
> 
> Btw, as of right now, I have not found a management tool which sets up
> bridges correctly for my sites...  There is always something extra
> needed with iptables, so it has to be done with hand-holding, or with
> the script= and downscript= options - which are annoyingly fragile
> because downscript isn't run if qemu has to be killed.
> 
> A helper which communicates its result back to qemu, and then *keeps
> the unix socket open* would be a nice way to reliably detect when the
> helper should destroy whatever it created - more reliable than downscript=.
> 
> I agree many backends are better implemented in qemu proper, but
> Anthony's idea sounds simple and versatile to me, and I would
> certainly use it for site-specific things.
> 
> -- Jamie

What you describe: helper= instead of fd= will work and I have no issue
with this. And we won't need any protocol between helper and qemu: just
the fd returned.  But of course this means that it can't be used to
create new backends such as vepa, which often need to setup the fd
in complex ways.

-- 
MST

[Qemu-devel] Re: Planning for 0.13

[Paolo Bonzini]

On 01/06/2010 04:10 PM, Anthony Liguori wrote:
> We have ones that require read/write, ones that require send/recv, and
> ones that require vhost interaction.  Really, the first two are the same
> but the distinction is necessary for Windows.

Not necessarily, you can open sockets on Windows so that they support 
read/write.  Just create it with

   fh = WSASocket (domain, type, protocol, NULL, 0, 0);

instead of socket.  Since Windows already has enough problems passing 
file descriptors to processes, imposing the above on an external 
management interface is not a huge chore.

Paolo

[Qemu-devel] Re: Planning for 0.13

[Michael S. Tsirkin]

On Wed, Jan 06, 2010 at 08:48:16PM +0100, Paolo Bonzini wrote:
> On 01/06/2010 04:10 PM, Anthony Liguori wrote:
>> We have ones that require read/write, ones that require send/recv, and
>> ones that require vhost interaction.  Really, the first two are the same
>> but the distinction is necessary for Windows.
>
> Not necessarily, you can open sockets on Windows so that they support  
> read/write.  Just create it with
>
>   fh = WSASocket (domain, type, protocol, NULL, 0, 0);
>
> instead of socket.  Since Windows already has enough problems passing  
> file descriptors to processes, imposing the above on an external  
> management interface is not a huge chore.
>
> Paolo

For linux read/write often isn't a good idea :)
E.g. for packet sockets you really need to use sendmsg and set msg_name
with the proper protocol.  You also must use recvmsg and set MSG_TRUNC
otherwise packets can get truncatred silently.

-- 
MST

[Qemu-devel] Re: Planning for 0.13

[Paolo Bonzini]

>>> We have ones that require read/write, ones that require send/recv, and
>>> ones that require vhost interaction.  Really, the first two are the same
>>> but the distinction is necessary for Windows.
>>
>> Not necessarily, you can open sockets on Windows so that they support
>> read/write.  Just create it with
>>
>>    fh = WSASocket (domain, type, protocol, NULL, 0, 0);
>>
>> instead of socket.  Since Windows already has enough problems passing
>> file descriptors to processes, imposing the above on an external
>> management interface is not a huge chore.
>>
>> Paolo
>
> For linux read/write often isn't a good idea :)

Yeah, only for "normal" raw sockets.

> E.g. for packet sockets you really need to use sendmsg and set msg_name
> with the proper protocol.  You also must use recvmsg and set MSG_TRUNC
> otherwise packets can get truncatred silently.

The situation for packet sockets is more similar to vnet headers no 
(just learning all this stuff)?  They require code in qemu anyway, so 
the helper would do only the set up/tear down.

Paolo

[Qemu-devel] Re: Planning for 0.13

[Michael S. Tsirkin]

On Wed, Jan 06, 2010 at 08:59:37PM +0100, Paolo Bonzini wrote:
>
>>>> We have ones that require read/write, ones that require send/recv, and
>>>> ones that require vhost interaction.  Really, the first two are the same
>>>> but the distinction is necessary for Windows.
>>>
>>> Not necessarily, you can open sockets on Windows so that they support
>>> read/write.  Just create it with
>>>
>>>    fh = WSASocket (domain, type, protocol, NULL, 0, 0);
>>>
>>> instead of socket.  Since Windows already has enough problems passing
>>> file descriptors to processes, imposing the above on an external
>>> management interface is not a huge chore.
>>>
>>> Paolo
>>
>> For linux read/write often isn't a good idea :)
>
> Yeah, only for "normal" raw sockets.

You mean TCP? We really need a UDP option BTW, and that
also needs special handling.

>> E.g. for packet sockets you really need to use sendmsg and set msg_name
>> with the proper protocol.  You also must use recvmsg and set MSG_TRUNC
>> otherwise packets can get truncatred silently.
>
> The situation for packet sockets is more similar to vnet headers no  
> (just learning all this stuff)?  They require code in qemu anyway, so  
> the helper would do only the set up/tear down.
>
> Paolo

Yes. And in that case, it does not make sense to force helper usage
where option to call 'socket' is just as easy.

-- 
MST

Re: [Qemu-devel] Re: Planning for 0.13

[Jamie Lokier]

Michael S. Tsirkin wrote:
> On Wed, Jan 06, 2010 at 05:19:45PM +0000, Jamie Lokier wrote:
> > Michael S. Tsirkin wrote:
> > > On Wed, Jan 06, 2010 at 09:24:45AM -0600, Anthony Liguori wrote:
> > > > A helper is semantics equivalent to passing an fd from a management  
> > > > tool.  All of the problems you describe are equally applicable to that  
> > > > model.
> > > 
> > > No, because management calls qemu and parses qemu help output.  Yes it
> > > is not ideal but it works today.
> > 
> > I don't understand.  What do you think would not work with
> > helper="..."  where ... is specified on the qemu command line by the
> > management script, versus the management script doing the helper
> > operations itself first and then calling qemu with fd=?
> > 
> > If you are thinking that management scripts will tailor the -net
> > arguments according to qemu version, you're right for some
> > configurations (but not well established simple ones).
> > 
> > Presumably management can do the same capability when specifying "..."
> > - the difference being it would query the helper tool to get _it's_
> > features in some cases, e.g. for arguments to a helper which uses SSH
> > to provide an encrypted tunnel.
> 
> What won't work is that besides fd= management specifies many other
> arguments to the -net flag.

I still don't understand.  What prevents management from specifying
the other arguments with helper=, either in the helper argument (for
arguments to the helper itself), or to qemu (for things qemu needs be told)?

> What you describe: helper= instead of fd= will work and I have no issue
> with this. And we won't need any protocol between helper and qemu: just
> the fd returned.

You may still need qemu to know whether to use send/recv (datagram
socket) or read/write with framing (pipe or something), but if it
comes to that, qemu can use fstat() to decide what it needs.  There is
probably no need for options passed back, I agree :-)

I think there is a lot to be said for letting the helper keep the
initial unix socket open, so it can reliably detect when the parent
qemu dies and then clean up.

> But of course this means that it can't be used to create new
> backends such as vepa, which often need to setup the fd in complex
> ways.

For vepa, I don't know if you mean the fd needs complex setup
initially, or complex options with each packet.  (Web browser
unavailable right now).

For complex initial setup, that's exactly what a helper would be good
for.  Any options needed for that setup would be part of the
helper="/path/to/helper opts..." string.  If there are options which
depend on helper version, management can query the helper itself with
/path/to/helper --help; that doesn't seem likely to be needed much.

If individual packet transmission requires "complex ways" then of
course that would have to be inside qemu.  Is that the case with vepa
and other things under discussion?  I got the impression there isn't
much variation in the way packets are tranmitted and received.

-- Jamie

Re: [Qemu-devel] Re: Planning for 0.13

[Jamie Lokier]

Michael S. Tsirkin wrote:
> On Wed, Jan 06, 2010 at 08:48:16PM +0100, Paolo Bonzini wrote:
> > On 01/06/2010 04:10 PM, Anthony Liguori wrote:
> >> We have ones that require read/write, ones that require send/recv, and
> >> ones that require vhost interaction.  Really, the first two are the same
> >> but the distinction is necessary for Windows.
> >
> > Not necessarily, you can open sockets on Windows so that they support  
> > read/write.  Just create it with
> >
> >   fh = WSASocket (domain, type, protocol, NULL, 0, 0);
> >
> > instead of socket.  Since Windows already has enough problems passing  
> > file descriptors to processes, imposing the above on an external  
> > management interface is not a huge chore.

I think socket() is equivalent to that WSASocket() on all modern
Windows versions (since Windows 2000 or so; actually the Windows NT
line and Winsock 2), so there isn't any need for WSASocket.

Passing socket file descriptors isn't very hard (again, the NT line),
but it's different from unix obviously.

On unix you can use a unix domain datagram socket if you're just
communicating with a local process, so recv/send can always be used.

On Windows, you have to use something else for local process
communications (loopback TCP sockets can be used but are sometimes
blocked by annoying software).  A smart thing to do would be detect
the type of handle received, and use the appropriate way of talking to
it, but since you need to receive a string to receive a handle anyway,
it's trivial to include an option with the string saying how to talk to it.

> For linux read/write often isn't a good idea :) E.g. for packet
> sockets you really need to use sendmsg and set msg_name with the
> proper protocol.  You also must use recvmsg and set MSG_TRUNC
> otherwise packets can get truncatred silently.

On packet sockets, connect() + write() should be equivalent to sendmsg
to the connected addresses.  If not it's a bit of a kernel bug.  Is it?
Agreed about recvmsg and MSG_TRUNC though.

But anyway, in what situation would you not use recv/send (except
Windows local IPC)?  And if there is one, is there ever a situation
where fstat() cannot be used by qemu to decide what to use?  If packet
sockets needs sendmsg with msg_name, it could call getpeername and
always use that :-)

-- Jamie

Re: [Qemu-devel] Re: Planning for 0.13

[Michael S. Tsirkin]

On Wed, Jan 06, 2010 at 10:49:54PM +0000, Jamie Lokier wrote:
> Michael S. Tsirkin wrote:
> > On Wed, Jan 06, 2010 at 05:19:45PM +0000, Jamie Lokier wrote:
> > > Michael S. Tsirkin wrote:
> > > > On Wed, Jan 06, 2010 at 09:24:45AM -0600, Anthony Liguori wrote:
> > > > > A helper is semantics equivalent to passing an fd from a management  
> > > > > tool.  All of the problems you describe are equally applicable to that  
> > > > > model.
> > > > 
> > > > No, because management calls qemu and parses qemu help output.  Yes it
> > > > is not ideal but it works today.
> > > 
> > > I don't understand.  What do you think would not work with
> > > helper="..."  where ... is specified on the qemu command line by the
> > > management script, versus the management script doing the helper
> > > operations itself first and then calling qemu with fd=?
> > > 
> > > If you are thinking that management scripts will tailor the -net
> > > arguments according to qemu version, you're right for some
> > > configurations (but not well established simple ones).
> > > 
> > > Presumably management can do the same capability when specifying "..."
> > > - the difference being it would query the helper tool to get _it's_
> > > features in some cases, e.g. for arguments to a helper which uses SSH
> > > to provide an encrypted tunnel.
> > 
> > What won't work is that besides fd= management specifies many other
> > arguments to the -net flag.
> 
> I still don't understand.  What prevents management from specifying
> the other arguments with helper=, either in the helper argument (for
> arguments to the helper itself), or to qemu (for things qemu needs be told)?
> 
> > What you describe: helper= instead of fd= will work and I have no issue
> > with this. And we won't need any protocol between helper and qemu: just
> > the fd returned.
> 
> You may still need qemu to know whether to use send/recv (datagram
> socket) or read/write with framing (pipe or something), but if it
> comes to that, qemu can use fstat() to decide what it needs.  There is
> probably no need for options passed back, I agree :-)

Well, if we special-case some fd type, let's just put all handling in
qemu. There's no advantage in spreading it around. It only makes sense
to move stuff out to helper script if it can be done generically IMO.

> I think there is a lot to be said for letting the helper keep the
> initial unix socket open, so it can reliably detect when the parent
> qemu dies and then clean up.
> 
> > But of course this means that it can't be used to create new
> > backends such as vepa, which often need to setup the fd in complex
> > ways.
> 
> For vepa, I don't know if you mean the fd needs complex setup
> initially, or complex options with each packet.  (Web browser
> unavailable right now).
> 
> For complex initial setup, that's exactly what a helper would be good
> for.  Any options needed for that setup would be part of the
> helper="/path/to/helper opts..." string.  If there are options which
> depend on helper version, management can query the helper itself with
> /path/to/helper --help; that doesn't seem likely to be needed much.
> 
> If individual packet transmission requires "complex ways" then of
> course that would have to be inside qemu.  Is that the case with vepa
> and other things under discussion?

As I said, stuff like vhost needs physical addresses as part of its
setup, so that definitely needs to be in qemu too.


>  I got the impression there isn't much variation in the way packets
>  are tranmitted and received.
> 
> -- Jamie

No. E.g. for TX/RX, with packet socket (which is what you'll use for vepa)
you need to do it differently from e.g. TCP socket.

It is true that TCP sockets really do look very much like pipes,
so you can write code that handles these generically.

But other devices don't, and there's a lot of minor differences there,
which we *could* abstract away - but in the end we won't get any new
features users need, and code will be more complex and hard to debug.

-- 
MST

Re: [Qemu-devel] Re: Planning for 0.13

[Michael S. Tsirkin]

On Wed, Jan 06, 2010 at 11:00:11PM +0000, Jamie Lokier wrote:
> Michael S. Tsirkin wrote:
> > On Wed, Jan 06, 2010 at 08:48:16PM +0100, Paolo Bonzini wrote:
> > > On 01/06/2010 04:10 PM, Anthony Liguori wrote:
> > >> We have ones that require read/write, ones that require send/recv, and
> > >> ones that require vhost interaction.  Really, the first two are the same
> > >> but the distinction is necessary for Windows.
> > >
> > > Not necessarily, you can open sockets on Windows so that they support  
> > > read/write.  Just create it with
> > >
> > >   fh = WSASocket (domain, type, protocol, NULL, 0, 0);
> > >
> > > instead of socket.  Since Windows already has enough problems passing  
> > > file descriptors to processes, imposing the above on an external  
> > > management interface is not a huge chore.
> 
> I think socket() is equivalent to that WSASocket() on all modern
> Windows versions (since Windows 2000 or so; actually the Windows NT
> line and Winsock 2), so there isn't any need for WSASocket.
> 
> Passing socket file descriptors isn't very hard (again, the NT line),
> but it's different from unix obviously.
> 
> On unix you can use a unix domain datagram socket if you're just
> communicating with a local process, so recv/send can always be used.
> 
> On Windows, you have to use something else for local process
> communications (loopback TCP sockets can be used but are sometimes
> blocked by annoying software).  A smart thing to do would be detect
> the type of handle received, and use the appropriate way of talking to
> it, but since you need to receive a string to receive a handle anyway,
> it's trivial to include an option with the string saying how to talk to it.
> 
> > For linux read/write often isn't a good idea :) E.g. for packet
> > sockets you really need to use sendmsg and set msg_name with the
> > proper protocol.  You also must use recvmsg and set MSG_TRUNC
> > otherwise packets can get truncatred silently.
> 
> On packet sockets, connect() + write() should be equivalent to sendmsg
> to the connected addresses.  If not it's a bit of a kernel bug.  Is it?

No. One of the fields in the address is protocol.  We need to look at
the packet to figure it out, so it can not be set per socket.

> Agreed about recvmsg and MSG_TRUNC though.
> 
> But anyway, in what situation would you not use recv/send (except
> Windows local IPC)?  And if there is one, is there ever a situation
> where fstat() cannot be used by qemu to decide what to use?  If packet
> sockets needs sendmsg with msg_name, it could call getpeername and
> always use that :-)
> 
> -- Jamie

If qemu has special code for e.g. packet sockets, I do not see any
advantage in moving setup out of qemu.  It's only generic stuff that
makes sense to use helpers for.

-- 
MST

Re: [Qemu-devel] Planning for 0.13

[Pasi Kärkkäinen]

On Tue, Jan 05, 2010 at 06:43:11AM -0600, Anthony Liguori wrote:
> Hi,
> 
> I hope everyone had a happy new year!  Now that we've finished the 0.12 
> release and most of us have had a nice break, I think it's time to start 
> planning for the next release.
> 
> 0.12 felt a bit rushed to me.  I'd like to take a bit more time with 
> 0.13 and try to complete features a bit more than we did in 0.12.  So I 
> propose that we target 0.13 as a 6-month cycle and target a June 1st 
> release.  Based on the -rc process for 0.12, I would like to have a two 
> week -rc cycle for 0.13 but this time, absolutely no non-bug fixes past 
> freeze.  We got sloppy in 0.12 with this and we had some ugly 
> regressions in the final release.
> 
> Here are the features that I'm aware of for 0.13.  Please add anything 
> you're working on and I'll try to centralize this somewhere.
> 
>  - gPXE support for virtio-blk
>  - Helper based network setup
>  - Balloon driver statistics
>  - Fully supported QMP
>  - Live migration protocol support for subvendor versions (aka features)
> 

Hello,

I think Xen folks were planning to push (some) Xen qemu-dm patches for upstream
Qemu 0.13.

Discussion here:
http://lists.xensource.com/archives/html/xen-devel/2010-01/msg00200.html

-- Pasi

Re: [Qemu-devel] Planning for 0.13

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1533 bytes --]

On Tue, 12 Jan 2010, Pasi Kärkkäinen wrote:
> On Tue, Jan 05, 2010 at 06:43:11AM -0600, Anthony Liguori wrote:
> > Hi,
> > 
> > I hope everyone had a happy new year!  Now that we've finished the 0.12 
> > release and most of us have had a nice break, I think it's time to start 
> > planning for the next release.
> > 
> > 0.12 felt a bit rushed to me.  I'd like to take a bit more time with 
> > 0.13 and try to complete features a bit more than we did in 0.12.  So I 
> > propose that we target 0.13 as a 6-month cycle and target a June 1st 
> > release.  Based on the -rc process for 0.12, I would like to have a two 
> > week -rc cycle for 0.13 but this time, absolutely no non-bug fixes past 
> > freeze.  We got sloppy in 0.12 with this and we had some ugly 
> > regressions in the final release.
> > 
> > Here are the features that I'm aware of for 0.13.  Please add anything 
> > you're working on and I'll try to centralize this somewhere.
> > 
> >  - gPXE support for virtio-blk
> >  - Helper based network setup
> >  - Balloon driver statistics
> >  - Fully supported QMP
> >  - Live migration protocol support for subvendor versions (aka features)
> > 
> 
> Hello,
> 
> I think Xen folks were planning to push (some) Xen qemu-dm patches for upstream
> Qemu 0.13.
> 
> Discussion here:
> http://lists.xensource.com/archives/html/xen-devel/2010-01/msg00200.html
> 

Yes, we are going to try to upstream Xen support in Qemu, hopefully
we'll finish in time for 0.13, but we just started the work so I don't
know yet for sure.

Re: [Qemu-devel] Planning for 0.13

[Artyom Tarasenko]

Hi Anthony,

> Here are the features that I'm aware of for 0.13.  Please add anything
> you're working on and I'll try to centralize this somewhere.
>
>  - gPXE support for virtio-blk
>  - Helper based network setup
>  - Balloon driver statistics
>  - Fully supported QMP
>  - Live migration protocol support for subvendor versions (aka features)

- Support for Solaris/sparc guest
Shouldn't be too hard. After my recent irq and mmu fixes Solaris
2.4-2.6 is already booting under the git/master. There are only few
issues with spurious interrupts (Slavio/Macio), going to track them
down.

Once it works under OBP, the missing functionality can be added to
OpenBIOS. There is already a good test case: NetBSD 1.3.3 fails under
OpenBIOS the same way as Solars does. But OpenBIOS release plan is
independent from qemu release plan anyway, right?

-- 
Regards,
Artyom Tarasenko

solaris/sparc under qemu blog: http://tyom.blogspot.com/