Release plan for 0.12.0

Release plan for 0.12.0

[Anthony Liguori]

Hi,

Now that 0.11.0 is behind us, it's time to start thinking about 0.12.0.

I'd like to do a few things different this time around.  I don't think 
the -rc process went very well as I don't think we got more testing out 
of it.  I'd like to shorten the timeline for 0.12.0 a good bit.  The 
0.10 stable tree got pretty difficult to maintain toward the end of the 
cycle.  We also had a pretty huge amount of change between 0.10 and 0.11 
so I think a shorter cycle is warranted.

I think aiming for early to mid-December would give us roughly a 3 month 
cycle and would align well with some of the Linux distribution cycles.  
I'd like to limit things to a single -rc that lasted only for about a 
week.  This is enough time to fix most of the obvious issues I think.

I'd also like to try to enumerate some features for this release.  
Here's a short list of things I expect to see for this release 
(target-i386 centric).  Please add or comment on items that you'd either 
like to see in the release or are planning on working on.

 o VMState conversion -- I expect most of the pc target to be completed
 o qdev conversion -- I hope that we'll get most of the pc target 
completely converted to qdev
 o storage live migration
 o switch to SeaBIOS (need to finish porting features from Bochs)
 o switch to gPXE (need to resolve slirp tftp server issue)
 o KSM integration
 o in-kernel APIC support for KVM
 o guest SMP support for KVM
 o updates to the default pc machine type

Please add to this list and I'll collect it all and post it somewhere.

Thanks!

-- 
Regards,

Anthony Liguori

Re: Release plan for 0.12.0

[Dustin Kirkland]

On Tue, Sep 29, 2009 at 6:54 PM, Anthony Liguori <aliguori@us.ibm.com> wrote:
> Now that 0.11.0 is behind us, it's time to start thinking about 0.12.0.
>
> I'd like to do a few things different this time around.  I don't think the
> -rc process went very well as I don't think we got more testing out of it.
>  I'd like to shorten the timeline for 0.12.0 a good bit.  The 0.10 stable
> tree got pretty difficult to maintain toward the end of the cycle.  We also
> had a pretty huge amount of change between 0.10 and 0.11 so I think a
> shorter cycle is warranted.
>
> I think aiming for early to mid-December would give us roughly a 3 month
> cycle and would align well with some of the Linux distribution cycles.  I'd
> like to limit things to a single -rc that lasted only for about a week.
>  This is enough time to fix most of the obvious issues I think.

As a downstream packager of qemu-kvm, I thought I'd mention that the
next Ubuntu cycle is now public:
 * https://wiki.ubuntu.com/LucidReleaseSchedule

The key date here is Feature Freeze, which is February 25, 2010.
That's the point by which we'd need to have a new qemu-kvm (which of
course is downstream of qemu) package in Ubuntu for the LTS 10.04
release in April 2010.

I'll gladly track the release candidate(s) in the Lucid development
tree, and hopefully pull 0.12 as soon as its available.

And we also provide daily snapshots of qemu builds at:
 * https://edge.launchpad.net/~ubuntu-virt/+archive/virt-daily-upstream

:-Dustin

Re: Release plan for 0.12.0

[Anthony Liguori]

Dustin Kirkland wrote:
> On Tue, Sep 29, 2009 at 6:54 PM, Anthony Liguori <aliguori@us.ibm.com> wrote:
>   
>> Now that 0.11.0 is behind us, it's time to start thinking about 0.12.0.
>>
>> I'd like to do a few things different this time around.  I don't think the
>> -rc process went very well as I don't think we got more testing out of it.
>>  I'd like to shorten the timeline for 0.12.0 a good bit.  The 0.10 stable
>> tree got pretty difficult to maintain toward the end of the cycle.  We also
>> had a pretty huge amount of change between 0.10 and 0.11 so I think a
>> shorter cycle is warranted.
>>
>> I think aiming for early to mid-December would give us roughly a 3 month
>> cycle and would align well with some of the Linux distribution cycles.  I'd
>> like to limit things to a single -rc that lasted only for about a week.
>>  This is enough time to fix most of the obvious issues I think.
>>     
>
> As a downstream packager of qemu-kvm, I thought I'd mention that the
> next Ubuntu cycle is now public:
>  * https://wiki.ubuntu.com/LucidReleaseSchedule
>
> The key date here is Feature Freeze, which is February 25, 2010.
> That's the point by which we'd need to have a new qemu-kvm (which of
> course is downstream of qemu) package in Ubuntu for the LTS 10.04
> release in April 2010.
>   

If we did a December release, then the 0.13 release would probably be in 
the April time frame.  Not really ideal for Lucid so I'd recommend that 
you ship 0.12.  The good news would be that 0.12 should be very stable 
by Feb 25th and since Lucid is an LTS, that's probably a Good Thing.

-- 
Regards,

Anthony Liguori

Re: [Qemu-devel] Release plan for 0.12.0

[Isaku Yamahata]

On Tue, Sep 29, 2009 at 06:54:53PM -0500, Anthony Liguori wrote:
> Hi,
> 
> Now that 0.11.0 is behind us, it's time to start thinking about 0.12.0.
> 
> I'd like to do a few things different this time around.  I don't think 
> the -rc process went very well as I don't think we got more testing out 
> of it.  I'd like to shorten the timeline for 0.12.0 a good bit.  The 
> 0.10 stable tree got pretty difficult to maintain toward the end of the 
> cycle.  We also had a pretty huge amount of change between 0.10 and 0.11 
> so I think a shorter cycle is warranted.
> 
> I think aiming for early to mid-December would give us roughly a 3 month 
> cycle and would align well with some of the Linux distribution cycles.  
> I'd like to limit things to a single -rc that lasted only for about a 
> week.  This is enough time to fix most of the obvious issues I think.
> 
> I'd also like to try to enumerate some features for this release.  
> Here's a short list of things I expect to see for this release 
> (target-i386 centric).  Please add or comment on items that you'd either 
> like to see in the release or are planning on working on.
> 
> o VMState conversion -- I expect most of the pc target to be completed
> o qdev conversion -- I hope that we'll get most of the pc target 
> completely converted to qdev
> o storage live migration
> o switch to SeaBIOS (need to finish porting features from Bochs)
> o switch to gPXE (need to resolve slirp tftp server issue)
> o KSM integration
> o in-kernel APIC support for KVM
> o guest SMP support for KVM
> o updates to the default pc machine type
> 
> Please add to this list and I'll collect it all and post it somewhere.

 o newer chipset (which is based on Q35 chipset)
 o multiple pci bus 
 o PCI express (MMCONFIG)
 o PCI express hot plug (not acpi based)
 o PCI express switch emulator

Although there is no PCIe emulated device at the moment, 
this will be a fundamental infrastructure for PCI express native
direct attach.

thanks,
-- 
yamahata

Re: [Qemu-devel] Release plan for 0.12.0

[Anthony Liguori]

Hi Isaku,

Isaku Yamahata wrote:
>  o newer chipset (which is based on Q35 chipset)
>  o multiple pci bus 
>  o PCI express (MMCONFIG)
>  o PCI express hot plug (not acpi based)
>  o PCI express switch emulator
>
> Although there is no PCIe emulated device at the moment, 
> this will be a fundamental infrastructure for PCI express native
> direct attach.
>   

Your patches definitely deserve review/commit.  I'll make sure that 
happens for the 0.12 time frame.

Michael, could you help review some of the PCI patches?

Thanks,

-- 
Regards,

Anthony Liguori

Re: [Qemu-devel] Release plan for 0.12.0

[Michael S. Tsirkin]

On Wed, Sep 30, 2009 at 08:03:20AM -0500, Anthony Liguori wrote:
> Hi Isaku,
>
> Isaku Yamahata wrote:
>>  o newer chipset (which is based on Q35 chipset)
>>  o multiple pci bus  o PCI express (MMCONFIG)
>>  o PCI express hot plug (not acpi based)
>>  o PCI express switch emulator
>>
>> Although there is no PCIe emulated device at the moment, this will be a 
>> fundamental infrastructure for PCI express native
>> direct attach.
>>   
>
> Your patches definitely deserve review/commit.  I'll make sure that  
> happens for the 0.12 time frame.
>
> Michael, could you help review some of the PCI patches?

Yes, I am doing this sent comments already.
The only thing I have not looked at yet is the new express file.

> Thanks,
>
> -- 
> Regards,
>
> Anthony Liguori

Re: Release plan for 0.12.0

[Amit Shah]

On (Tue) Sep 29 2009 [18:54:53], Anthony Liguori wrote:
> Hi,
>
> Now that 0.11.0 is behind us, it's time to start thinking about 0.12.0.
>
> I'd like to do a few things different this time around.  I don't think  
> the -rc process went very well as I don't think we got more testing out  
> of it.  I'd like to shorten the timeline for 0.12.0 a good bit.  The  
> 0.10 stable tree got pretty difficult to maintain toward the end of the  
> cycle.  We also had a pretty huge amount of change between 0.10 and 0.11  
> so I think a shorter cycle is warranted.
>
> I think aiming for early to mid-December would give us roughly a 3 month  
> cycle and would align well with some of the Linux distribution cycles.   
> I'd like to limit things to a single -rc that lasted only for about a  
> week.  This is enough time to fix most of the obvious issues I think.
>
> I'd also like to try to enumerate some features for this release.   
> Here's a short list of things I expect to see for this release  
> (target-i386 centric).  Please add or comment on items that you'd either  
> like to see in the release or are planning on working on.
>
> o VMState conversion -- I expect most of the pc target to be completed
> o qdev conversion -- I hope that we'll get most of the pc target  
> completely converted to qdev
> o storage live migration
> o switch to SeaBIOS (need to finish porting features from Bochs)
> o switch to gPXE (need to resolve slirp tftp server issue)
> o KSM integration
> o in-kernel APIC support for KVM
> o guest SMP support for KVM
> o updates to the default pc machine type

  o multiport virtio-console support

> Please add to this list and I'll collect it all and post it somewhere.

Thanks,
		Amit

Re: Release plan for 0.12.0

[Anthony Liguori]

Amit Shah wrote:
> On (Tue) Sep 29 2009 [18:54:53], Anthony Liguori wrote:
>   
>   o multiport virtio-console support
>   

Assuming we can get the kernel drivers straightened out, I think it's 
certainly reasonable for 0.12.

-- 
Regards,

Anthony Liguori

Re: Release plan for 0.12.0

[Amit Shah]

On (Wed) Sep 30 2009 [08:04:17], Anthony Liguori wrote:
> Amit Shah wrote:
>> On (Tue) Sep 29 2009 [18:54:53], Anthony Liguori wrote:
>>     o multiport virtio-console support
>>   
>
> Assuming we can get the kernel drivers straightened out, I think it's  
> certainly reasonable for 0.12.

The kernel drivers are in fine shape.

		Amit

Re: Release plan for 0.12.0

[Anthony Liguori]

Amit Shah wrote:
> On (Wed) Sep 30 2009 [08:04:17], Anthony Liguori wrote:
>   
>> Amit Shah wrote:
>>     
>>> On (Tue) Sep 29 2009 [18:54:53], Anthony Liguori wrote:
>>>     o multiport virtio-console support
>>>   
>>>       
>> Assuming we can get the kernel drivers straightened out, I think it's  
>> certainly reasonable for 0.12.
>>     
>
> The kernel drivers are in fine shape.
>   

I meant on track for including into the appropriate tree.  Looking for 
an Ack/Nack from Rusty.  That's been the general policy for all virtio 
changes btw.  Nothing specific to virtio-console.

> 		Amit
>   


-- 
Regards,

Anthony Liguori

Re: Release plan for 0.12.0

[Amit Shah]

On (Wed) Sep 30 2009 [09:47:22], Anthony Liguori wrote:
> Amit Shah wrote:
>> On (Wed) Sep 30 2009 [08:04:17], Anthony Liguori wrote:
>>   
>>> Amit Shah wrote:
>>>     
>>>> On (Tue) Sep 29 2009 [18:54:53], Anthony Liguori wrote:
>>>>     o multiport virtio-console support
>>>>         
>>> Assuming we can get the kernel drivers straightened out, I think it's 
>>>  certainly reasonable for 0.12.
>>
>> The kernel drivers are in fine shape.
>
> I meant on track for including into the appropriate tree.  Looking for  
> an Ack/Nack from Rusty.  That's been the general policy for all virtio  
> changes btw.  Nothing specific to virtio-console.

That's fine.

		Amit

Re: [Qemu-devel] Release plan for 0.12.0

[Avi Kivity]

On 09/30/2009 01:54 AM, Anthony Liguori wrote:
> Hi,
>
> Now that 0.11.0 is behind us, it's time to start thinking about 0.12.0.
>
> I'd like to do a few things different this time around.  I don't think 
> the -rc process went very well as I don't think we got more testing 
> out of it.  I'd like to shorten the timeline for 0.12.0 a good bit.  
> The 0.10 stable tree got pretty difficult to maintain toward the end 
> of the cycle.  We also had a pretty huge amount of change between 0.10 
> and 0.11 so I think a shorter cycle is warranted.
>
> I think aiming for early to mid-December would give us roughly a 3 
> month cycle and would align well with some of the Linux distribution 
> cycles.  I'd like to limit things to a single -rc that lasted only for 
> about a week.  This is enough time to fix most of the obvious issues I 
> think.
>
> I'd also like to try to enumerate some features for this release.  
> Here's a short list of things I expect to see for this release 
> (target-i386 centric).  Please add or comment on items that you'd 
> either like to see in the release or are planning on working on.
>
> o VMState conversion -- I expect most of the pc target to be completed
> o qdev conversion -- I hope that we'll get most of the pc target 
> completely converted to qdev
> o storage live migration
> o switch to SeaBIOS (need to finish porting features from Bochs)
> o switch to gPXE (need to resolve slirp tftp server issue)
> o KSM integration
> o in-kernel APIC support for KVM
> o guest SMP support for KVM
> o updates to the default pc machine type

Machine monitor protocol.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] Release plan for 0.12.0

[Anthony Liguori]

Avi Kivity wrote:
> On 09/30/2009 01:54 AM, Anthony Liguori wrote:
>> Hi,
>>
>> Now that 0.11.0 is behind us, it's time to start thinking about 0.12.0.
>>
>> I'd like to do a few things different this time around.  I don't 
>> think the -rc process went very well as I don't think we got more 
>> testing out of it.  I'd like to shorten the timeline for 0.12.0 a 
>> good bit.  The 0.10 stable tree got pretty difficult to maintain 
>> toward the end of the cycle.  We also had a pretty huge amount of 
>> change between 0.10 and 0.11 so I think a shorter cycle is warranted.
>>
>> I think aiming for early to mid-December would give us roughly a 3 
>> month cycle and would align well with some of the Linux distribution 
>> cycles.  I'd like to limit things to a single -rc that lasted only 
>> for about a week.  This is enough time to fix most of the obvious 
>> issues I think.
>>
>> I'd also like to try to enumerate some features for this release.  
>> Here's a short list of things I expect to see for this release 
>> (target-i386 centric).  Please add or comment on items that you'd 
>> either like to see in the release or are planning on working on.
>>
>> o VMState conversion -- I expect most of the pc target to be completed
>> o qdev conversion -- I hope that we'll get most of the pc target 
>> completely converted to qdev
>> o storage live migration
>> o switch to SeaBIOS (need to finish porting features from Bochs)
>> o switch to gPXE (need to resolve slirp tftp server issue)
>> o KSM integration
>> o in-kernel APIC support for KVM
>> o guest SMP support for KVM
>> o updates to the default pc machine type
>
> Machine monitor protocol.

If we're going to support the protocol for 0.12, I'd like to most of the 
code merged by the end of October.

-- 
Regards,

Anthony Liguori

Re: [Qemu-devel] Release plan for 0.12.0

[Luiz Capitulino]

On Wed, 30 Sep 2009 08:05:16 -0500
Anthony Liguori <aliguori@us.ibm.com> wrote:

> Avi Kivity wrote:
> > On 09/30/2009 01:54 AM, Anthony Liguori wrote:
> >> Hi,
> >>
> >> Now that 0.11.0 is behind us, it's time to start thinking about 0.12.0.
> >>
> >> I'd like to do a few things different this time around.  I don't 
> >> think the -rc process went very well as I don't think we got more 
> >> testing out of it.  I'd like to shorten the timeline for 0.12.0 a 
> >> good bit.  The 0.10 stable tree got pretty difficult to maintain 
> >> toward the end of the cycle.  We also had a pretty huge amount of 
> >> change between 0.10 and 0.11 so I think a shorter cycle is warranted.
> >>
> >> I think aiming for early to mid-December would give us roughly a 3 
> >> month cycle and would align well with some of the Linux distribution 
> >> cycles.  I'd like to limit things to a single -rc that lasted only 
> >> for about a week.  This is enough time to fix most of the obvious 
> >> issues I think.
> >>
> >> I'd also like to try to enumerate some features for this release.  
> >> Here's a short list of things I expect to see for this release 
> >> (target-i386 centric).  Please add or comment on items that you'd 
> >> either like to see in the release or are planning on working on.
> >>
> >> o VMState conversion -- I expect most of the pc target to be completed
> >> o qdev conversion -- I hope that we'll get most of the pc target 
> >> completely converted to qdev
> >> o storage live migration
> >> o switch to SeaBIOS (need to finish porting features from Bochs)
> >> o switch to gPXE (need to resolve slirp tftp server issue)
> >> o KSM integration
> >> o in-kernel APIC support for KVM
> >> o guest SMP support for KVM
> >> o updates to the default pc machine type
> >
> > Machine monitor protocol.
> 
> If we're going to support the protocol for 0.12, I'd like to most of the 
> code merged by the end of October.

 Four weeks.. Not so much time, but let's try.

 There are two major issues that may delay QMP.

 Firstly, we are still on the infrastructure/design phase, which
is natural to take time. Maybe when handlers start getting converted
en masse things will be faster.

 Secondly: testing. I have a very ugly python script to test the
already converted handlers. The problem is not only the ugliness,
the right way to do this would be to use kvm-autotest. So, I was
planning to take a detailed look at it and perhaps start writing
tests for QMP right when each handler is converted. Right Thing,
but takes time.

Re: [Qemu-devel] Release plan for 0.12.0

[Avi Kivity]

On 10/01/2009 11:13 PM, Luiz Capitulino wrote:
>> If we're going to support the protocol for 0.12, I'd like to most of the
>> code merged by the end of October.
>>      
>   Four weeks.. Not so much time, but let's try.
>
>   There are two major issues that may delay QMP.
>
>   Firstly, we are still on the infrastructure/design phase, which
> is natural to take time. Maybe when handlers start getting converted
> en masse things will be faster.
>    

I sure hope so.  Maybe someone can pitch in if not.

>   Secondly: testing. I have a very ugly python script to test the
> already converted handlers. The problem is not only the ugliness,
> the right way to do this would be to use kvm-autotest. So, I was
> planning to take a detailed look at it and perhaps start writing
> tests for QMP right when each handler is converted. Right Thing,
> but takes time.
>    

I think this could be done by having autotest use two monitors, one with 
the machine protocol and one with the human protocol, trying first the 
machine protocol and falling back if the command is not supported.

Hopefully we can get the autotest people to work on it so we parallelize 
development.  They'll also give user-oriented feedback which can be 
valuable.

Are you using a standard json parser with your test script?  That's an 
additional validation.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] Release plan for 0.12.0

[Luiz Capitulino]

On Sat, 03 Oct 2009 12:04:57 +0200
Avi Kivity <avi@redhat.com> wrote:

> On 10/01/2009 11:13 PM, Luiz Capitulino wrote:
> >> If we're going to support the protocol for 0.12, I'd like to most of the
> >> code merged by the end of October.
> >>      
> >   Four weeks.. Not so much time, but let's try.
> >
> >   There are two major issues that may delay QMP.
> >
> >   Firstly, we are still on the infrastructure/design phase, which
> > is natural to take time. Maybe when handlers start getting converted
> > en masse things will be faster.
> >    
> 
> I sure hope so.  Maybe someone can pitch in if not.

 I've written a TODO list if someone is willing to help:

http://tinyurl.com/ya7l6bo

> >   Secondly: testing. I have a very ugly python script to test the
> > already converted handlers. The problem is not only the ugliness,
> > the right way to do this would be to use kvm-autotest. So, I was
> > planning to take a detailed look at it and perhaps start writing
> > tests for QMP right when each handler is converted. Right Thing,
> > but takes time.
> >    
> 
> I think this could be done by having autotest use two monitors, one with 
> the machine protocol and one with the human protocol, trying first the 
> machine protocol and falling back if the command is not supported.

 Yes, sounds a good idea.

> Hopefully we can get the autotest people to work on it so we parallelize 
> development.  They'll also give user-oriented feedback which can be 
> valuable.

 I will talk to them about that.

> Are you using a standard json parser with your test script?  That's an 
> additional validation.

 I'm using Python's json module, but I could run one of the checkers
listed in the json's page for each test, before the Python's module
kicks in.

Re: [Qemu-devel] Release plan for 0.12.0

[Avi Kivity]

On 10/05/2009 02:43 PM, Luiz Capitulino wrote:
>> Are you using a standard json parser with your test script?  That's an
>> additional validation.
>>      
>   I'm using Python's json module, but I could run one of the checkers
> listed in the json's page for each test, before the Python's module
> kicks in.
>    

python-json should be sufficient, I think.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Release plan for 0.12.0

[Luiz Capitulino]

On Wed, 30 Sep 2009 08:41:23 +0200
Avi Kivity <avi@redhat.com> wrote:

> On 09/30/2009 01:54 AM, Anthony Liguori wrote:
> > Hi,
> >
> > Now that 0.11.0 is behind us, it's time to start thinking about 0.12.0.
> >
> > I'd like to do a few things different this time around.  I don't think 
> > the -rc process went very well as I don't think we got more testing 
> > out of it.  I'd like to shorten the timeline for 0.12.0 a good bit.  
> > The 0.10 stable tree got pretty difficult to maintain toward the end 
> > of the cycle.  We also had a pretty huge amount of change between 0.10 
> > and 0.11 so I think a shorter cycle is warranted.
> >
> > I think aiming for early to mid-December would give us roughly a 3 
> > month cycle and would align well with some of the Linux distribution 
> > cycles.  I'd like to limit things to a single -rc that lasted only for 
> > about a week.  This is enough time to fix most of the obvious issues I 
> > think.
> >
> > I'd also like to try to enumerate some features for this release.  
> > Here's a short list of things I expect to see for this release 
> > (target-i386 centric).  Please add or comment on items that you'd 
> > either like to see in the release or are planning on working on.
> >
> > o VMState conversion -- I expect most of the pc target to be completed
> > o qdev conversion -- I hope that we'll get most of the pc target 
> > completely converted to qdev
> > o storage live migration
> > o switch to SeaBIOS (need to finish porting features from Bochs)
> > o switch to gPXE (need to resolve slirp tftp server issue)
> > o KSM integration
> > o in-kernel APIC support for KVM
> > o guest SMP support for KVM
> > o updates to the default pc machine type
> 
> Machine monitor protocol.

 Yeah, I was going to suggest it as well.

Re: [Qemu-devel] Release plan for 0.12.0

[Michael Tokarev]

Anthony Liguori wrote:
[]
> Here's a short list of things I expect to see for this release 
> (target-i386 centric).  Please add or comment on items that you'd either 
> like to see in the release or are planning on working on.
[..]
> o guest SMP support for KVM

Hmm.  What is this, can you elaborate a bit more please?
-smp nn is already here, no?

Thanks!

/mjt

Re: [Qemu-devel] Release plan for 0.12.0

[Avi Kivity]

On 09/30/2009 10:53 AM, Michael Tokarev wrote:
> Anthony Liguori wrote:
> []
>> Here's a short list of things I expect to see for this release 
>> (target-i386 centric).  Please add or comment on items that you'd 
>> either like to see in the release or are planning on working on.
> [..]
>> o guest SMP support for KVM
>
> Hmm.  What is this, can you elaborate a bit more please?
> -smp nn is already here, no?
>

Only in qemu-kvm.git.  This is about qemu.git (which supports -smp, but 
not with kvm).

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] Release plan for 0.12.0

[Carl-Daniel Hailfinger]

Hi,

On 30.09.2009 01:54, Anthony Liguori wrote:
> Now that 0.11.0 is behind us, it's time to start thinking about 0.12.0.
>
> I'd also like to try to enumerate some features for this release. 
> Here's a short list of things I expect to see for this release
> (target-i386 centric).
>
> o switch to SeaBIOS (need to finish porting features from Bochs)

That switch is much appreciated because it also reduces the testing
matrix of those coreboot developers who boot test every commit with Qemu.

However, to run coreboot on Qemu with the same init sequence as on
simplified real hardware, we need Cache-as-RAM (CAR) support. This is
basically a mode where sizeof(cacheable area) <= sizeof (L2 cache) and
causes the processor to lock the cache and not pass any reads/writes
through to the RAM behind the cached area. The easiest way to implement
this would be to check the cache size criterion upon every MTRR
manipulation and either map a chunk of fresh memory on top of the
existing memory (which may be RAM, ROM or unmapped) for every cacheable
area, and if the cacheable area starts to exceed the L2 cache size,
discard all memory contents of the memory mapped on top.
For additional correctness, the memory shoud not be discarded and
written back to the lower layer of memory if WBINVD (instead of INVD) or
CLFLUSH are called. That one is mostly sugar, though, and coreboot can
do without.

Right now coreboot sets up the MTRRs correctly, but then (conditional on
Qemu) only uses areas which are known to be backed by RAM instead of the
areas designated by CAR.

I'd like to implement CAR support which builds on top of my MTRR code
which was merged some months ago (and I already have code to check for
total cacheable area size), but I need help with the memory mapping
stuff. How do I proceed? Clean up what I have and insert "FIXME"
comments where I don't know how to implement stuff so others can see the
code and comment on it?

Regards,
Carl-Daniel

-- 
http://www.hailfinger.org/

Re: [Qemu-devel] Release plan for 0.12.0

[Anthony Liguori]

Carl-Daniel Hailfinger wrote:
> Hi,
>
> On 30.09.2009 01:54, Anthony Liguori wrote:
>   
>> Now that 0.11.0 is behind us, it's time to start thinking about 0.12.0.
>>
>> I'd also like to try to enumerate some features for this release. 
>> Here's a short list of things I expect to see for this release
>> (target-i386 centric).
>>
>> o switch to SeaBIOS (need to finish porting features from Bochs)
>>     
>
> That switch is much appreciated because it also reduces the testing
> matrix of those coreboot developers who boot test every commit with Qemu.
>
> However, to run coreboot on Qemu with the same init sequence as on
> simplified real hardware, we need Cache-as-RAM (CAR) support. This is
> basically a mode where sizeof(cacheable area) <= sizeof (L2 cache) and
> causes the processor to lock the cache and not pass any reads/writes
> through to the RAM behind the cached area. The easiest way to implement
> this would be to check the cache size criterion upon every MTRR
> manipulation and either map a chunk of fresh memory on top of the
> existing memory (which may be RAM, ROM or unmapped) for every cacheable
> area, and if the cacheable area starts to exceed the L2 cache size,
> discard all memory contents of the memory mapped on top.
> For additional correctness, the memory shoud not be discarded and
> written back to the lower layer of memory if WBINVD (instead of INVD) or
> CLFLUSH are called. That one is mostly sugar, though, and coreboot can
> do without.
>   

Do we really need coreboot to use the same init sequence?   coreboot is 
firmware and we don't necessarily run real firmware under QEMU.  It's a 
short cut that lets us avoid a lot of complexity.

> Right now coreboot sets up the MTRRs correctly, but then (conditional on
> Qemu) only uses areas which are known to be backed by RAM instead of the
> areas designated by CAR.
>
> I'd like to implement CAR support which builds on top of my MTRR code
> which was merged some months ago (and I already have code to check for
> total cacheable area size), but I need help with the memory mapping
> stuff. How do I proceed? Clean up what I have and insert "FIXME"
> comments where I don't know how to implement stuff so others can see the
> code and comment on it?
>   

You could start there.  But from a higher level, I'm not sure I think a 
partial implementation of something like CAR is all that valuable since 
coreboot already runs under QEMU.

> Regards,
> Carl-Daniel
>
>   
-- 

Regards,

Anthony Liguori

Re: [Qemu-devel] Release plan for 0.12.0

[Carl-Daniel Hailfinger]

On 30.09.2009 15:07, Anthony Liguori wrote:
> Carl-Daniel Hailfinger wrote:
>> However, to run coreboot on Qemu with the same init sequence as on
>> simplified real hardware, we need Cache-as-RAM (CAR) support. [...] 
>
> Do we really need coreboot to use the same init sequence?   coreboot
> is firmware and we don't necessarily run real firmware under QEMU. 
> It's a short cut that lets us avoid a lot of complexity.

I know that some people were running 440BX BIOS images for real hardware
on Qemu and they got pretty far.

The complexity would be limited to the MTRR code and unless there were
major architectural changes in mapping RAM to address ranges, no other
code (except VM save and VM restore) should get even a single line changed.

>> Right now coreboot sets up the MTRRs correctly, but then (conditional on
>> Qemu) only uses areas which are known to be backed by RAM instead of the
>> areas designated by CAR.
>>
>> I'd like to implement CAR support which builds on top of my MTRR code
>> which was merged some months ago (and I already have code to check for
>> total cacheable area size), but I need help with the memory mapping
>> stuff. How do I proceed? Clean up what I have and insert "FIXME"
>> comments where I don't know how to implement stuff so others can see the
>> code and comment on it?   
>
> You could start there.  But from a higher level, I'm not sure I think
> a partial implementation of something like CAR is all that valuable
> since coreboot already runs under QEMU.

It only runs if WORKAROUND_QEMU is defined (maybe not exactly that name,
but you get the point). The code in coreboot calculates MTRR settings to
cover the place where the stack will be. To workaround missing CAR in
Qemu, it then has to recalculate the stack location to be able to
actually use the stack. That forces coreboot to keep two stack base
variables and to completely replace the generic logic which switches off
CAR.

I hope the explanation above didn't offend you, I just tried to clarify
why working CAR is such a big deal for coreboot.

If you want either a full CAR implementation or no CAR implementation, I
can write a patch which implements full CAR, but then I need to hook
WBINVD, INVD and CLFLUSH. Neither instruction is executed often enough
to show up in any profile. Besides that, for anything not using CAR
(everything after the firmware), the penalty is a simple test of a
boolean variable per WBINVD/INVD/CLFLUSH.

If you have further questions, please don't hesitate to ask.

Regards,
Carl-Daniel

-- 
http://www.hailfinger.org/

Re: [Qemu-devel] Release plan for 0.12.0

[Blue Swirl]

On Wed, Sep 30, 2009 at 6:59 PM, Carl-Daniel Hailfinger
<c-d.hailfinger.devel.2006@gmx.net> wrote:
> On 30.09.2009 15:07, Anthony Liguori wrote:
>> Carl-Daniel Hailfinger wrote:
>>> However, to run coreboot on Qemu with the same init sequence as on
>>> simplified real hardware, we need Cache-as-RAM (CAR) support. [...]
>>
>> Do we really need coreboot to use the same init sequence?   coreboot
>> is firmware and we don't necessarily run real firmware under QEMU.
>> It's a short cut that lets us avoid a lot of complexity.
>
> I know that some people were running 440BX BIOS images for real hardware
> on Qemu and they got pretty far.
>
> The complexity would be limited to the MTRR code and unless there were
> major architectural changes in mapping RAM to address ranges, no other
> code (except VM save and VM restore) should get even a single line changed.
>
>>> Right now coreboot sets up the MTRRs correctly, but then (conditional on
>>> Qemu) only uses areas which are known to be backed by RAM instead of the
>>> areas designated by CAR.
>>>
>>> I'd like to implement CAR support which builds on top of my MTRR code
>>> which was merged some months ago (and I already have code to check for
>>> total cacheable area size), but I need help with the memory mapping
>>> stuff. How do I proceed? Clean up what I have and insert "FIXME"
>>> comments where I don't know how to implement stuff so others can see the
>>> code and comment on it?
>>
>> You could start there.  But from a higher level, I'm not sure I think
>> a partial implementation of something like CAR is all that valuable
>> since coreboot already runs under QEMU.
>
> It only runs if WORKAROUND_QEMU is defined (maybe not exactly that name,
> but you get the point). The code in coreboot calculates MTRR settings to
> cover the place where the stack will be. To workaround missing CAR in
> Qemu, it then has to recalculate the stack location to be able to
> actually use the stack. That forces coreboot to keep two stack base
> variables and to completely replace the generic logic which switches off
> CAR.
>
> I hope the explanation above didn't offend you, I just tried to clarify
> why working CAR is such a big deal for coreboot.
>
> If you want either a full CAR implementation or no CAR implementation, I
> can write a patch which implements full CAR, but then I need to hook
> WBINVD, INVD and CLFLUSH. Neither instruction is executed often enough
> to show up in any profile. Besides that, for anything not using CAR
> (everything after the firmware), the penalty is a simple test of a
> boolean variable per WBINVD/INVD/CLFLUSH.

The CAR mode could affect only translation so that special CAR
versions of the WBINVD etc. instructions are selected. On switch to
normal mode, the TBs need to be flushed.

Instead of your memory mapping approach (which should work) you could
also try using different memory access functions in CAR mode. It may
be more difficult, though.

Re: [Qemu-devel] Release plan for 0.12.0

[Luiz Capitulino]

On Tue, 29 Sep 2009 18:54:53 -0500
Anthony Liguori <aliguori@us.ibm.com> wrote:

> I think aiming for early to mid-December would give us roughly a 3 month 
> cycle and would align well with some of the Linux distribution cycles.  
> I'd like to limit things to a single -rc that lasted only for about a 
> week.  This is enough time to fix most of the obvious issues I think.

 How do you plan to do it? I mean, are you going to create a separate branch
or make master the -rc?

 Creating a separate branch (which is what we do today, iiuc) makes it
get less attention, freezing master for a certain period is the best
way to stabilize.

 Is this what you had in mind?

Re: [Qemu-devel] Release plan for 0.12.0

[Anthony Liguori]

Luiz Capitulino wrote:
> On Tue, 29 Sep 2009 18:54:53 -0500
> Anthony Liguori <aliguori@us.ibm.com> wrote:
>
>   
>> I think aiming for early to mid-December would give us roughly a 3 month 
>> cycle and would align well with some of the Linux distribution cycles.  
>> I'd like to limit things to a single -rc that lasted only for about a 
>> week.  This is enough time to fix most of the obvious issues I think.
>>     
>
>  How do you plan to do it? I mean, are you going to create a separate branch
> or make master the -rc?
>
>  Creating a separate branch (which is what we do today, iiuc) makes it
> get less attention, freezing master for a certain period is the best
> way to stabilize.
>
>  Is this what you had in mind?
>   
What do people think?

One reason I branch is because some people care a bit less about 
releases so it makes the process non-disruptive to them.  If the other 
maintainers agreed though, I would certainly like to have the master 
branch essentially frozen for the week before the release.

-- 
Regards,

Anthony Liguori

Re: Release plan for 0.12.0

[Fred Leeflang]

[-- Attachment #1: Type: text/plain, Size: 1433 bytes --]

2009/9/30 Anthony Liguori <aliguori@us.ibm.com>

> Luiz Capitulino wrote:
>
>> On Tue, 29 Sep 2009 18:54:53 -0500
>> Anthony Liguori <aliguori@us.ibm.com> wrote:
>>
>>
>>
>>> I think aiming for early to mid-December would give us roughly a 3 month
>>> cycle and would align well with some of the Linux distribution cycles.  I'd
>>> like to limit things to a single -rc that lasted only for about a week.
>>>  This is enough time to fix most of the obvious issues I think.
>>>
>>>
>>
>>  How do you plan to do it? I mean, are you going to create a separate
>> branch
>> or make master the -rc?
>>
>>  Creating a separate branch (which is what we do today, iiuc) makes it
>> get less attention, freezing master for a certain period is the best
>> way to stabilize.
>>
>>  Is this what you had in mind?
>>
>>
> What do people think?
>
> One reason I branch is because some people care a bit less about releases
> so it makes the process non-disruptive to them.  If the other maintainers
> agreed though, I would certainly like to have the master branch essentially
> frozen for the week before the release.
>

freezing is only neccesary if you need time to gather all the patches, build
and test them together etc. If you don't feel you or the developers need to
do that to get a reliable release out I think it only halts developers
without any clear reason to do so. Calling 'attention' to a release is not a
clear reason IMO.

-Fred

[-- Attachment #2: Type: text/html, Size: 2139 bytes --]

Re: [Qemu-devel] Release plan for 0.12.0

[Luiz Capitulino]

On Wed, 30 Sep 2009 17:03:23 +0200
Fred Leeflang <fredl@dutchie.org> wrote:

> 2009/9/30 Anthony Liguori <aliguori@us.ibm.com>
> 
> > Luiz Capitulino wrote:
> >
> >> On Tue, 29 Sep 2009 18:54:53 -0500
> >> Anthony Liguori <aliguori@us.ibm.com> wrote:
> >>
> >>
> >>
> >>> I think aiming for early to mid-December would give us roughly a 3 month
> >>> cycle and would align well with some of the Linux distribution cycles.  I'd
> >>> like to limit things to a single -rc that lasted only for about a week.
> >>>  This is enough time to fix most of the obvious issues I think.
> >>>
> >>>
> >>
> >>  How do you plan to do it? I mean, are you going to create a separate
> >> branch
> >> or make master the -rc?
> >>
> >>  Creating a separate branch (which is what we do today, iiuc) makes it
> >> get less attention, freezing master for a certain period is the best
> >> way to stabilize.
> >>
> >>  Is this what you had in mind?
> >>
> >>
> > What do people think?
> >
> > One reason I branch is because some people care a bit less about releases
> > so it makes the process non-disruptive to them.  If the other maintainers
> > agreed though, I would certainly like to have the master branch essentially
> > frozen for the week before the release.
> >
> 
> freezing is only neccesary if you need time to gather all the patches, build
> and test them together etc. 

 Not exactly, freezing is done to stop/slowdown writing new code and focus
on bug fixing for a period of time.

 This is not only needed for a release, but projects should always try
to find the best balance between 'number of bugs' and 'feature addition rate'.

> If you don't feel you or the developers need to
> do that to get a reliable release out I think it only halts developers
> without any clear reason to do so. Calling 'attention' to a release is not a
> clear reason IMO.

 Having a functional and relatively stable release is not only
important, but it's the ultimate goal IMO.

 Obviously we should take care not to take extremes. No QEMU release
will be 100% bug free, that's why we have stables.

Re: Release plan for 0.12.0

[Juan Quintela]

Anthony Liguori <aliguori@us.ibm.com> wrote:
> Luiz Capitulino wrote:
>> On Tue, 29 Sep 2009 18:54:53 -0500
>> Anthony Liguori <aliguori@us.ibm.com> wrote:
>>
>>   
>>> I think aiming for early to mid-December would give us roughly a 3
>>> month cycle and would align well with some of the Linux
>>> distribution cycles.  I'd like to limit things to a single -rc that
>>> lasted only for about a week.  This is enough time to fix most of
>>> the obvious issues I think.
>>>     
>>
>>  How do you plan to do it? I mean, are you going to create a separate branch
>> or make master the -rc?
>>
>>  Creating a separate branch (which is what we do today, iiuc) makes it
>> get less attention, freezing master for a certain period is the best
>> way to stabilize.
>>
>>  Is this what you had in mind?
>>   
> What do people think?
>
> One reason I branch is because some people care a bit less about
> releases so it makes the process non-disruptive to them.  If the other
> maintainers agreed though, I would certainly like to have the master
> branch essentially frozen for the week before the release.

I am not a maintainer, but I still think that it is a good idea :)

Later, Juan.

Re: [Qemu-devel] Release plan for 0.12.0

[Gerd Hoffmann]

On 09/30/09 16:45, Anthony Liguori wrote:
> One reason I branch is because some people care a bit less about
> releases so it makes the process non-disruptive to them. If the other
> maintainers agreed though, I would certainly like to have the master
> branch essentially frozen for the week before the release.

We had much longer disruptions without a release freeze, so why worry 
about a single week?  One week freeze is short enougth that the 
disruption isn't a big issue.  It will help testing the to-be-released 
code.  Go for it.

cheers,
   Gerd

Re: Release plan for 0.12.0

[TAKEDA, toshiya]

Anthony Liguori さんは書きました:
>Hi,
>
>Now that 0.11.0 is behind us, it's time to start thinking about 0.12.0.
>
>I'd like to do a few things different this time around.  I don't think 
>the -rc process went very well as I don't think we got more testing out 
>of it.  I'd like to shorten the timeline for 0.12.0 a good bit.  The 
>0.10 stable tree got pretty difficult to maintain toward the end of the 
>cycle.  We also had a pretty huge amount of change between 0.10 and 0.11 
>so I think a shorter cycle is warranted.
>
>I think aiming for early to mid-December would give us roughly a 3 month 
>cycle and would align well with some of the Linux distribution cycles.  
>I'd like to limit things to a single -rc that lasted only for about a 
>week.  This is enough time to fix most of the obvious issues I think.
>
>I'd also like to try to enumerate some features for this release.  
>Here's a short list of things I expect to see for this release 
>(target-i386 centric).  Please add or comment on items that you'd either 
>like to see in the release or are planning on working on.
>
> o VMState conversion -- I expect most of the pc target to be completed
> o qdev conversion -- I hope that we'll get most of the pc target 
>completely converted to qdev
> o storage live migration
> o switch to SeaBIOS (need to finish porting features from Bochs)
> o switch to gPXE (need to resolve slirp tftp server issue)
> o KSM integration
> o in-kernel APIC support for KVM
> o guest SMP support for KVM
> o updates to the default pc machine type
>
>Please add to this list and I'll collect it all and post it somewhere.

o NEC PC-9821 family support on target-i386

In the last patch MS-DOS can boot on QEMU.
I think I can add support nic (LGY-98) and IDE in 0.12.0
and I hope I can boot FreeBSD/pc98 on it.

PS.
I will repost v3 patch in the next week, please wait reviewing v2 patch I post Oct.1.

Thanks,
TAKEDA, toshiya

>Thanks!
>
>-- 
>Regards,
>
>Anthony Liguori
>
>
>
>

Re: [Qemu-devel] Release plan for 0.12.0

[Jens Osterkamp]

On Wednesday 30 September 2009, Anthony Liguori wrote:

>  o VMState conversion -- I expect most of the pc target to be completed
>  o qdev conversion -- I hope that we'll get most of the pc target
> completely converted to qdev
>  o storage live migration
>  o switch to SeaBIOS (need to finish porting features from Bochs)
>  o switch to gPXE (need to resolve slirp tftp server issue)
>  o KSM integration
>  o in-kernel APIC support for KVM
>  o guest SMP support for KVM
>  o updates to the default pc machine type
>
> Please add to this list and I'll collect it all and post it somewhere.

What about Or Gerlitz' raw backend driver ? I did not see it go in yet, or did 
I miss something ?

Jens

Re: Release plan for 0.12.0

[Anthony Liguori]

Jens Osterkamp wrote:
> On Wednesday 30 September 2009, Anthony Liguori wrote:
>
>   
>>  o VMState conversion -- I expect most of the pc target to be completed
>>  o qdev conversion -- I hope that we'll get most of the pc target
>> completely converted to qdev
>>  o storage live migration
>>  o switch to SeaBIOS (need to finish porting features from Bochs)
>>  o switch to gPXE (need to resolve slirp tftp server issue)
>>  o KSM integration
>>  o in-kernel APIC support for KVM
>>  o guest SMP support for KVM
>>  o updates to the default pc machine type
>>
>> Please add to this list and I'll collect it all and post it somewhere.
>>     
>
> What about Or Gerlitz' raw backend driver ? I did not see it go in yet, or did 
> I miss something ?
>   

The patch seems to have not been updated after the initial posting and 
the first feedback cycle.

I'm generally inclined to oppose the functionality as I don't think it 
offers any advantages over the existing backends.

Regards,

Anthony Liguori

Re: [Qemu-devel] Release plan for 0.12.0

[Arnd Bergmann]

On Thursday 08 October 2009, Anthony Liguori wrote:
> Jens Osterkamp wrote:
> > On Wednesday 30 September 2009, Anthony Liguori wrote:
> >
> >> Please add to this list and I'll collect it all and post it somewhere.
> >>     
> >
> > What about Or Gerlitz' raw backend driver ? I did not see it go in yet, or did 
> > I miss something ?
> >   
> 
> The patch seems to have not been updated after the initial posting and 
> the first feedback cycle.
> 
> I'm generally inclined to oppose the functionality as I don't think it 
> offers any advantages over the existing backends.

There are two reasons why I think this backend is important:

- As an easy way to provide isolation between guests (private ethernet
  port aggregator, PEPA) and external enforcement of network priviledges
  (virtual ethernet port aggregator, VEPA) using the macvlan subsystem.

- As a counterpart to the vhost_net driver, providing an identical
  user interface with or without vhost_net acceleration in the kernel.

	Arnd <><

Re: [Qemu-devel] Release plan for 0.12.0

[Anthony Liguori]

Arnd Bergmann wrote:
> There are two reasons why I think this backend is important:
>
> - As an easy way to provide isolation between guests (private ethernet
>   port aggregator, PEPA) and external enforcement of network priviledges
>   (virtual ethernet port aggregator, VEPA) using the macvlan subsystem.
>   

Can't this all be done with tap and a bridge?

Regards,

Anthony Liguori

Re: [Qemu-devel] Release plan for 0.12.0

[Michael S. Tsirkin]

On Wed, Oct 14, 2009 at 08:53:55AM -0500, Anthony Liguori wrote:
> Arnd Bergmann wrote:
>> There are two reasons why I think this backend is important:
>>
>> - As an easy way to provide isolation between guests (private ethernet
>>   port aggregator, PEPA) and external enforcement of network priviledges
>>   (virtual ethernet port aggregator, VEPA) using the macvlan subsystem.
>>   
>
> Can't this all be done with tap and a bridge?

Not with existing kernels, I think.

> Regards,
>
> Anthony Liguori

Re: [Qemu-devel] Release plan for 0.12.0

[Michael S. Tsirkin]

On Wed, Oct 14, 2009 at 03:09:28PM +0200, Arnd Bergmann wrote:
> On Thursday 08 October 2009, Anthony Liguori wrote:
> > Jens Osterkamp wrote:
> > > On Wednesday 30 September 2009, Anthony Liguori wrote:
> > >
> > >> Please add to this list and I'll collect it all and post it somewhere.
> > >>     
> > >
> > > What about Or Gerlitz' raw backend driver ? I did not see it go in yet, or did 
> > > I miss something ?
> > >   
> > 
> > The patch seems to have not been updated after the initial posting and 
> > the first feedback cycle.
> > 
> > I'm generally inclined to oppose the functionality as I don't think it 
> > offers any advantages over the existing backends.
> 
> There are two reasons why I think this backend is important:
> 
> - As an easy way to provide isolation between guests (private ethernet
>   port aggregator, PEPA) and external enforcement of network priviledges
>   (virtual ethernet port aggregator, VEPA) using the macvlan subsystem.
> 
> - As a counterpart to the vhost_net driver, providing an identical
>   user interface with or without vhost_net acceleration in the kernel.
> 
> 	Arnd <><

I think raw sockets also support RX mac/vlan filtering in kernel, which
might be faster than doing it in virtio in userspace as it's done now.

-- 
MST

Re: Release plan for 0.12.0

[Michael S. Tsirkin]

On Thu, Oct 08, 2009 at 09:21:04AM -0500, Anthony Liguori wrote:
> Jens Osterkamp wrote:
>> On Wednesday 30 September 2009, Anthony Liguori wrote:
>>
>>   
>>>  o VMState conversion -- I expect most of the pc target to be completed
>>>  o qdev conversion -- I hope that we'll get most of the pc target
>>> completely converted to qdev
>>>  o storage live migration
>>>  o switch to SeaBIOS (need to finish porting features from Bochs)
>>>  o switch to gPXE (need to resolve slirp tftp server issue)
>>>  o KSM integration
>>>  o in-kernel APIC support for KVM
>>>  o guest SMP support for KVM
>>>  o updates to the default pc machine type
>>>
>>> Please add to this list and I'll collect it all and post it somewhere.
>>>     
>>
>> What about Or Gerlitz' raw backend driver ? I did not see it go in yet, 
>> or did I miss something ?
>>   
>
> The patch seems to have not been updated after the initial posting and  
> the first feedback cycle.

Looks like Or has abandoned it.  I have an updated version which works
with new APIs, etc.  Let me post it and we'll go from there.

> I'm generally inclined to oppose the functionality as I don't think it  
> offers any advantages over the existing backends.

I patch it in and use it all the time.  It's much easier to setup
on a random machine than a bridged config.

-- 
MST

Re: Release plan for 0.12.0

[Anthony Liguori]

Michael S. Tsirkin wrote:
> Looks like Or has abandoned it.  I have an updated version which works
> with new APIs, etc.  Let me post it and we'll go from there.
>
>   
>> I'm generally inclined to oppose the functionality as I don't think it  
>> offers any advantages over the existing backends.
>>     
>
> I patch it in and use it all the time.  It's much easier to setup
> on a random machine than a bridged config.
>   

Having two things that do the same thing is just going to lead to user 
confusion.  If the problem is tap is too hard to setup, we should try to 
simplify tap configuration.

Regards,

Anthony Liguori

Re: Release plan for 0.12.0

[Michael S. Tsirkin]

On Wed, Oct 14, 2009 at 09:17:15AM -0500, Anthony Liguori wrote:
> Michael S. Tsirkin wrote:
>> Looks like Or has abandoned it.  I have an updated version which works
>> with new APIs, etc.  Let me post it and we'll go from there.
>>
>>   
>>> I'm generally inclined to oppose the functionality as I don't think 
>>> it  offers any advantages over the existing backends.
>>>     
>>
>> I patch it in and use it all the time.  It's much easier to setup
>> on a random machine than a bridged config.
>>   
>
> Having two things that do the same thing is just going to lead to user  
> confusion.

They do not do the same thing. With raw socket you can use windows
update without a bridge in the host, with tap you can't.

> If the problem is tap is too hard to setup, we should try to  
> simplify tap configuration.

The problem is bridge is too hard to setup.
Simplifying that is a good idea, but outside the scope
of the qemu project.


> Regards,
>
> Anthony Liguori
>

Re: [Qemu-devel] Re: Release plan for 0.12.0

[Jamie Lokier]

Michael S. Tsirkin wrote:
> On Wed, Oct 14, 2009 at 09:17:15AM -0500, Anthony Liguori wrote:
> > Michael S. Tsirkin wrote:
> >> Looks like Or has abandoned it.  I have an updated version which works
> >> with new APIs, etc.  Let me post it and we'll go from there.
> >>
> >>   
> >>> I'm generally inclined to oppose the functionality as I don't think 
> >>> it  offers any advantages over the existing backends.
> >>>     
> >>
> >> I patch it in and use it all the time.  It's much easier to setup
> >> on a random machine than a bridged config.
> >>   
> >
> > Having two things that do the same thing is just going to lead to user  
> > confusion.
> 
> They do not do the same thing. With raw socket you can use windows
> update without a bridge in the host, with tap you can't.

On the other hand, with raw socket, guest Windows can't access files
on the host's Samba share can it?  So it's not that useful even for
Windows guests.

> > If the problem is tap is too hard to setup, we should try to  
> > simplify tap configuration.
> 
> The problem is bridge is too hard to setup.
> Simplifying that is a good idea, but outside the scope
> of the qemu project.

I venture it's important enough for qemu that it's worth working on
that.  Something that looks like the raw socket but behaves like an
automatically instantiated bridge attached to the bound interface
would be a useful interface.

I don't have much time, but I'll help anybody who wants to do that.

-- Jamie

Re: Re: Release plan for 0.12.0

[Michael S. Tsirkin]

On Wed, Oct 14, 2009 at 04:19:17PM +0100, Jamie Lokier wrote:
> Michael S. Tsirkin wrote:
> > On Wed, Oct 14, 2009 at 09:17:15AM -0500, Anthony Liguori wrote:
> > > Michael S. Tsirkin wrote:
> > >> Looks like Or has abandoned it.  I have an updated version which works
> > >> with new APIs, etc.  Let me post it and we'll go from there.
> > >>
> > >>   
> > >>> I'm generally inclined to oppose the functionality as I don't think 
> > >>> it  offers any advantages over the existing backends.
> > >>>     
> > >>
> > >> I patch it in and use it all the time.  It's much easier to setup
> > >> on a random machine than a bridged config.
> > >>   
> > >
> > > Having two things that do the same thing is just going to lead to user  
> > > confusion.
> > 
> > They do not do the same thing. With raw socket you can use windows
> > update without a bridge in the host, with tap you can't.
> 
> On the other hand, with raw socket, guest Windows can't access files
> on the host's Samba share can it?  So it's not that useful even for
> Windows guests.

I guess this depends on whether you use the same host for samba :)

> > > If the problem is tap is too hard to setup, we should try to  
> > > simplify tap configuration.
> > 
> > The problem is bridge is too hard to setup.
> > Simplifying that is a good idea, but outside the scope
> > of the qemu project.
> 
> I venture it's important enough for qemu that it's worth working on
> that.  Something that looks like the raw socket but behaves like an
> automatically instantiated bridge attached to the bound interface
> would be a useful interface.

I agree, that would be good to have.

> I don't have much time, but I'll help anybody who wants to do that.
> 
> -- Jamie

Re: [Qemu-devel] Re: Release plan for 0.12.0

[Sridhar Samudrala]

On Wed, 2009-10-14 at 17:50 +0200, Michael S. Tsirkin wrote:
> On Wed, Oct 14, 2009 at 04:19:17PM +0100, Jamie Lokier wrote:
> > Michael S. Tsirkin wrote:
> > > On Wed, Oct 14, 2009 at 09:17:15AM -0500, Anthony Liguori wrote:
> > > > Michael S. Tsirkin wrote:
> > > >> Looks like Or has abandoned it.  I have an updated version which works
> > > >> with new APIs, etc.  Let me post it and we'll go from there.
> > > >>
> > > >>   
> > > >>> I'm generally inclined to oppose the functionality as I don't think 
> > > >>> it  offers any advantages over the existing backends.
> > > >>>     
> > > >>
> > > >> I patch it in and use it all the time.  It's much easier to setup
> > > >> on a random machine than a bridged config.
> > > >>   
> > > >
> > > > Having two things that do the same thing is just going to lead to user  
> > > > confusion.
> > > 
> > > They do not do the same thing. With raw socket you can use windows
> > > update without a bridge in the host, with tap you can't.
> > 
> > On the other hand, with raw socket, guest Windows can't access files
> > on the host's Samba share can it?  So it's not that useful even for
> > Windows guests.
> 
> I guess this depends on whether you use the same host for samba :)
> 
> > > > If the problem is tap is too hard to setup, we should try to  
> > > > simplify tap configuration.
> > > 
> > > The problem is bridge is too hard to setup.
> > > Simplifying that is a good idea, but outside the scope
> > > of the qemu project.
> > 
> > I venture it's important enough for qemu that it's worth working on
> > that.  Something that looks like the raw socket but behaves like an
> > automatically instantiated bridge attached to the bound interface
> > would be a useful interface.
> 
> I agree, that would be good to have.

Can't we bind the raw socket to the tap interface instead of the
physical interface and allow the bridge config to work.

Thanks
Sridhar


> 
> > I don't have much time, but I'll help anybody who wants to do that.
> > 
> > -- Jamie
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Raw vs. tap (was: Re: Re: Release plan for 0.12.0)

[Anthony Liguori]

Sridhar Samudrala wrote:
> Can't we bind the raw socket to the tap interface instead of the
> physical interface and allow the bridge config to work.
>   

But why use the raw interface instead of tap directly.

Let me summarize the discussion so far:

Raw sockets
Pros:
 o User specifies a network interface to bind to
 o External traffic Just Works, guest-to-guest traffic Just Works

Cons:
 o Requires root (cannot chmod)
 o Guest<->host traffic does not work
 o No support for GSO/checksum offload

Some things that I'm not sure will work or not:
 o guest with a bridge (sending traffic with multiple mac addresses)
 o guest trying to enter promiscuous mode

Tap
Pros:
 o All types of networking works when configured
 o Supports non-root users via tunctl
 o Supports GSO/checksum offload

Cons:
 o Requires configuring a bridge which can be difficult for some users

Since I don't see any clear features in raw sockets that aren't present 
in tap, the argument really boils down to two things.  First, we should 
take any feature in qemu and let the user decide whether or not they 
want to use it.  I strongly feel this is a bad philosophy that will lead 
to increased user confusion and a poor user experience.

Second, even though raw looses performance and requires root, since it 
requires no external configuration it is easier to use and therefore 
should be an option for users.  I dislike this argument because it 
tricks a user into thinking that raw is a viable replacement for tap.  
It certainly isn't performance wise but most importantly, it isn't from 
a functional perspective.  I would be much more inclined to consider 
taking raw and improving the performance long term if guest<->host 
networking worked.  This appears to be a fundamental limitation though 
and I think it's something that will forever plague users if we include 
this feature.

So at this point, I think it's a mistake to include raw socket support.  
If the goal is to improve networking usability such that it just works 
as a root user, let's incorporate a default network script that creates 
a bridge or something like that.  There are better ways to achieve that 
goal.

Regards,

Anthony Liguori

Re: Raw vs. tap (was: Re: [Qemu-devel] Re: Release plan for 0.12.0)

[Mark McLoughlin]

On Wed, 2009-10-14 at 17:53 -0500, Anthony Liguori wrote:

> So at this point, I think it's a mistake to include raw socket support.  
> If the goal is to improve networking usability such that it just works 
> as a root user, let's incorporate a default network script that creates 
> a bridge or something like that.  There are better ways to achieve that 
> goal.

FWIW, I haven't really played with the raw backend yet, but my initial
thought was also "what exactly does this gain us apart from yet more
confusion for users?".

So, I tend to agree, but I'm not so hung up on the "user confusion"
aspect - the users that I worry about confusing (e.g. virt-manager
users) would never even know the backend exists, even if qemu did
support it.

The one hope I had for raw is that it might allow us to get closer to
the NIC, get more details on the NIC tx queue and have more intelligent
tx mitigation. This is probably better explored in the context of
vhost-net, though.

Wrt. to configuring bridges, libvirt comes with a good default setup - a
bridge without any physical NICs connected, but NAT set up for access to
the outside world.

For bridging to a physical NIC, our plan continues to be that
NetworkManager will eventually make this trivial for users, but that's
still in progress. In the meantime, the config isn't all that complex:

  http://wiki.libvirt.org/page/Networking#Fedora.2FRHEL_Bridging

Cheers,
Mark.

Re: Raw vs. tap (was: Re: Re: Release plan for 0.12.0)

[Michael S. Tsirkin]

On Wed, Oct 14, 2009 at 05:53:56PM -0500, Anthony Liguori wrote:
> I would be much more inclined to consider  
> taking raw and improving the performance long term if guest<->host  
> networking worked.  This appears to be a fundamental limitation though  
> and I think it's something that will forever plague users if we include  
> this feature.

In fact, I think it's fixable with a raw socket bound to a macvlan.
Would that be enough?

-- 
MST

Re: Raw vs. tap

[Anthony Liguori]

Michael S. Tsirkin wrote:
> On Wed, Oct 14, 2009 at 05:53:56PM -0500, Anthony Liguori wrote:
>   
>> I would be much more inclined to consider  
>> taking raw and improving the performance long term if guest<->host  
>> networking worked.  This appears to be a fundamental limitation though  
>> and I think it's something that will forever plague users if we include  
>> this feature.
>>     
>
> In fact, I think it's fixable with a raw socket bound to a macvlan.
> Would that be enough?
>   

What setup does that entail on the part of a user?  Wouldn't we be back 
to square one wrt users having to run archaic networking commands in 
order to set things up?

Regards,

Anthony Liguori

Re: Raw vs. tap

[Michael S. Tsirkin]

On Thu, Oct 15, 2009 at 08:32:03AM -0500, Anthony Liguori wrote:
> Michael S. Tsirkin wrote:
>> On Wed, Oct 14, 2009 at 05:53:56PM -0500, Anthony Liguori wrote:
>>   
>>> I would be much more inclined to consider  taking raw and improving 
>>> the performance long term if guest<->host  networking worked.  This 
>>> appears to be a fundamental limitation though  and I think it's 
>>> something that will forever plague users if we include  this feature.
>>>     
>>
>> In fact, I think it's fixable with a raw socket bound to a macvlan.
>> Would that be enough?
>>   
>
> What setup does that entail on the part of a user?  Wouldn't we be back  
> to square one wrt users having to run archaic networking commands in  
> order to set things up?

Unlike bridge, qemu could set up macvlan without disrupting
host networking. The only issue would be cleanup if qemu
is killed.

> Regards,
>
> Anthony Liguori
>

Re: Raw vs. tap

[Anthony Liguori]

Michael S. Tsirkin wrote:
> On Thu, Oct 15, 2009 at 08:32:03AM -0500, Anthony Liguori wrote:
>   
>> Michael S. Tsirkin wrote:
>>     
>>> On Wed, Oct 14, 2009 at 05:53:56PM -0500, Anthony Liguori wrote:
>>>   
>>>       
>>>> I would be much more inclined to consider  taking raw and improving 
>>>> the performance long term if guest<->host  networking worked.  This 
>>>> appears to be a fundamental limitation though  and I think it's 
>>>> something that will forever plague users if we include  this feature.
>>>>     
>>>>         
>>> In fact, I think it's fixable with a raw socket bound to a macvlan.
>>> Would that be enough?
>>>   
>>>       
>> What setup does that entail on the part of a user?  Wouldn't we be back  
>> to square one wrt users having to run archaic networking commands in  
>> order to set things up?
>>     
>
> Unlike bridge, qemu could set up macvlan without disrupting
> host networking. The only issue would be cleanup if qemu
> is killed.
>   

But this would require additional features in macvlan, correct?

This also only works if a guest uses the mac address assigned to it, 
correct?  If a guest was bridging the virtual nic, this would all come 
apart?

Regards,

Anthony Liguori

Re: Raw vs. tap

[Michael S. Tsirkin]

On Thu, Oct 15, 2009 at 10:18:18AM -0500, Anthony Liguori wrote:
> Michael S. Tsirkin wrote:
>> On Thu, Oct 15, 2009 at 08:32:03AM -0500, Anthony Liguori wrote:
>>   
>>> Michael S. Tsirkin wrote:
>>>     
>>>> On Wed, Oct 14, 2009 at 05:53:56PM -0500, Anthony Liguori wrote:
>>>>         
>>>>> I would be much more inclined to consider  taking raw and 
>>>>> improving the performance long term if guest<->host  networking 
>>>>> worked.  This appears to be a fundamental limitation though  and 
>>>>> I think it's something that will forever plague users if we 
>>>>> include  this feature.
>>>>>             
>>>> In fact, I think it's fixable with a raw socket bound to a macvlan.
>>>> Would that be enough?
>>>>         
>>> What setup does that entail on the part of a user?  Wouldn't we be 
>>> back  to square one wrt users having to run archaic networking 
>>> commands in  order to set things up?
>>>     
>>
>> Unlike bridge, qemu could set up macvlan without disrupting
>> host networking. The only issue would be cleanup if qemu
>> is killed.
>>   
>
> But this would require additional features in macvlan, correct?

Not sure: what is the "this" that you are talking about.
It can already be set up without disturbing host networking.

> This also only works if a guest uses the mac address assigned to it,  
> correct?  If a guest was bridging the virtual nic, this would all come  
> apart?

Hmm, you could enable promisc mode, but generally this is true:
if you require bridging, use a bridge.

> Regards,
>
> Anthony Liguori

Re: Raw vs. tap

[Anthony Liguori]

Michael S. Tsirkin wrote:
> Not sure: what is the "this" that you are talking about.
>   

I meant, fixing guest<->host traffic.  You're former argument hinged on 
being able to having networking that Just Worked without adding new 
things.  If this doesn't work today with macvlan, then the argument is 
invalid.

Regards,

Anthony Liguori

Re: Raw vs. tap

[Michael S. Tsirkin]

On Thu, Oct 15, 2009 at 01:37:20PM -0500, Anthony Liguori wrote:
> Michael S. Tsirkin wrote:
>> Not sure: what is the "this" that you are talking about.
>>   
>
> I meant, fixing guest<->host traffic.

This needs to be supported in networking core.

> You're former argument hinged on  
> being able to having networking that Just Worked without adding new  
> things.  If this doesn't work today with macvlan, then the argument is  
> invalid.
>
> Regards,
>
> Anthony Liguori

Re: Raw vs. tap

[Michael S. Tsirkin]

On Thu, Oct 15, 2009 at 05:48:19PM +0200, Michael S. Tsirkin wrote:
> On Thu, Oct 15, 2009 at 10:18:18AM -0500, Anthony Liguori wrote:
> > Michael S. Tsirkin wrote:
> >> On Thu, Oct 15, 2009 at 08:32:03AM -0500, Anthony Liguori wrote:
> >>   
> >>> Michael S. Tsirkin wrote:
> >>>     
> >>>> On Wed, Oct 14, 2009 at 05:53:56PM -0500, Anthony Liguori wrote:
> >>>>         
> >>>>> I would be much more inclined to consider  taking raw and 
> >>>>> improving the performance long term if guest<->host  networking 
> >>>>> worked.  This appears to be a fundamental limitation though  and 
> >>>>> I think it's something that will forever plague users if we 
> >>>>> include  this feature.
> >>>>>             
> >>>> In fact, I think it's fixable with a raw socket bound to a macvlan.
> >>>> Would that be enough?
> >>>>         
> >>> What setup does that entail on the part of a user?  Wouldn't we be 
> >>> back  to square one wrt users having to run archaic networking 
> >>> commands in  order to set things up?
> >>>     
> >>
> >> Unlike bridge, qemu could set up macvlan without disrupting
> >> host networking. The only issue would be cleanup if qemu
> >> is killed.
> >>   
> >
> > But this would require additional features in macvlan, correct?
> 
> Not sure: what is the "this" that you are talking about.
> It can already be set up without disturbing host networking.
> 
> > This also only works if a guest uses the mac address assigned to it,  
> > correct?  If a guest was bridging the virtual nic, this would all come  
> > apart?
> 
> Hmm, you could enable promisc mode, but generally this is true:
> if you require bridging, use a bridge.

Just to clarify this statement: if bridge in guest does not
configure mac filtering on guest uplink (linux
bridge by default does not do this now, OTOH macvlan does),
bridge in host will perform better in this setup
because it learns macs from traffic and does mac filtering in host.

> > Regards,
> >
> > Anthony Liguori

Re: [Qemu-devel] Re: Release plan for 0.12.0

[Michael S. Tsirkin]

On Wed, Oct 14, 2009 at 02:10:00PM -0700, Sridhar Samudrala wrote:
> On Wed, 2009-10-14 at 17:50 +0200, Michael S. Tsirkin wrote:
> > On Wed, Oct 14, 2009 at 04:19:17PM +0100, Jamie Lokier wrote:
> > > Michael S. Tsirkin wrote:
> > > > On Wed, Oct 14, 2009 at 09:17:15AM -0500, Anthony Liguori wrote:
> > > > > Michael S. Tsirkin wrote:
> > > > >> Looks like Or has abandoned it.  I have an updated version which works
> > > > >> with new APIs, etc.  Let me post it and we'll go from there.
> > > > >>
> > > > >>   
> > > > >>> I'm generally inclined to oppose the functionality as I don't think 
> > > > >>> it  offers any advantages over the existing backends.
> > > > >>>     
> > > > >>
> > > > >> I patch it in and use it all the time.  It's much easier to setup
> > > > >> on a random machine than a bridged config.
> > > > >>   
> > > > >
> > > > > Having two things that do the same thing is just going to lead to user  
> > > > > confusion.
> > > > 
> > > > They do not do the same thing. With raw socket you can use windows
> > > > update without a bridge in the host, with tap you can't.
> > > 
> > > On the other hand, with raw socket, guest Windows can't access files
> > > on the host's Samba share can it?  So it's not that useful even for
> > > Windows guests.
> > 
> > I guess this depends on whether you use the same host for samba :)
> > 
> > > > > If the problem is tap is too hard to setup, we should try to  
> > > > > simplify tap configuration.
> > > > 
> > > > The problem is bridge is too hard to setup.
> > > > Simplifying that is a good idea, but outside the scope
> > > > of the qemu project.
> > > 
> > > I venture it's important enough for qemu that it's worth working on
> > > that.  Something that looks like the raw socket but behaves like an
> > > automatically instantiated bridge attached to the bound interface
> > > would be a useful interface.
> > 
> > I agree, that would be good to have.
> 
> Can't we bind the raw socket to the tap interface instead of the
> physical interface and allow the bridge config to work.


We can, kind of (e.g. with veth) but what's the point then?

> Thanks
> Sridhar
> 
> 
> > 
> > > I don't have much time, but I'll help anybody who wants to do that.
> > > 
> > > -- Jamie
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Release plan for 0.12.0

[Takahiro Hirofuchi]

Hello,


2009/9/30 Anthony Liguori <aliguori@us.ibm.com>:
> Hi,
>
> Now that 0.11.0 is behind us, it's time to start thinking about 0.12.0.

> o storage live migration

Sorry for a bit off topic. But, my special NBD server can do this
independently of VMM implementations.
See http://bitbucket.org/hirofuchi/xnbd/wiki/Home if interested.


Takahiro