User space for lapic2

User space for lapic2

[Dong, Eddie]

Avi:
	To make lapic code into mainline earlier, I am thinking what
should the user space code look like.  If we wait till lapic branch has
all same functionality as mainline have today i.e. live migration, all
guests etc, we may have very long way to go given that only Greg (temply
off), me have contribution to the code besides you. So what I am
thinking is to reduce the bar to "no regression", which means a user
space command to choice kernel pic/apic. When choosing user pic/apic, we
make sure everything works including livemigration and all known good
guests; if choosing kernel pic/apic, we have better performance but
some feature lagging such as live migration.
	Does this make sense? If yes, How about command option "-irqchip
k" and "-irqchip u"? the default value is user.

thx, eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: User space for lapic2

[Avi Kivity]

Dong, Eddie wrote:
> Avi:
> 	To make lapic code into mainline earlier, I am thinking what
> should the user space code look like.  If we wait till lapic branch has
> all same functionality as mainline have today i.e. live migration, all
> guests etc, we may have very long way to go given that only Greg (temply
> off), me have contribution to the code besides you. So what I am
> thinking is to reduce the bar to "no regression", which means a user
> space command to choice kernel pic/apic. When choosing user pic/apic, we
> make sure everything works including livemigration and all known good
> guests; if choosing kernel pic/apic, we have better performance but
> some feature lagging such as live migration.
> 	Does this make sense? If yes, How about command option "-irqchip
> k" and "-irqchip u"? the default value is user.
>   

That doesn't work; one day we'll make kernel apic the default, and then 
that new userspace won't work with kernels that have in-kernel pic but 
not lapic.


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: User space for lapic2

[Dong, Eddie]

Avi Kivity wrote:
> Dong, Eddie wrote:
>> Avi:
>> 	To make lapic code into mainline earlier, I am thinking what
>> should the user space code look like.  If we wait till lapic branch
>> has all same functionality as mainline have today i.e. live
>> migration, all guests etc, we may have very long way to go given
>> that only Greg (temply off), me have contribution to the code
>> besides you. So what I am thinking is to reduce the bar to "no
>> regression", which means a user space command to choice kernel
>> pic/apic. When choosing user pic/apic, we make sure everything works
>> including livemigration and all known good guests; if choosing
>> kernel pic/apic, we have better performance but some feature lagging
>> 	such as live migration. Does this make sense? If yes, How about
>> command option "-irqchip 
>> k" and "-irqchip u"? the default value is user.
>> 
> 
> That doesn't work; one day we'll make kernel apic the default,
> and then
> that new userspace won't work with kernels that have in-kernel pic
> but not lapic. 
> 
After we get everything settled down, we can switch the default setting
to "kernel".
BTW, I am talking about generic lapic solution, not current one in
lapic2 branch. Definitely we need to pull in kernel apic+ioapic first
before the push back. But I think now we should make a high level
decision on how we will handle this merge and how will user space look
like.

Eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: User space for lapic2

[Dong, Eddie]

kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
> Avi Kivity wrote:
>> Dong, Eddie wrote:
>>> Avi:
>>> 	To make lapic code into mainline earlier, I am thinking what
>>> should the user space code look like.  If we wait till lapic branch
>>> has all same functionality as mainline have today i.e. live
>>> migration, all guests etc, we may have very long way to go given
>>> that only Greg (temply off), me have contribution to the code
>>> besides you. So what I am thinking is to reduce the bar to "no
>>> regression", which means a user space command to choice kernel
>>> pic/apic. When choosing user pic/apic, we make sure everything works
>>> including livemigration and all known good guests; if choosing
>>> kernel pic/apic, we have better performance but some feature lagging
>>> 	such as live migration. Does this make sense? If yes, How about
>>> command option "-irqchip k" and "-irqchip u"? the default value is
>>> user. 
>>> 
>> 
>> That doesn't work; one day we'll make kernel apic the default, and
>> then that new userspace won't work with kernels that have in-kernel
>> pic but not lapic. 
>> 
> After we get everything settled down, we can switch the default
> setting to "kernel". BTW, I am talking about generic lapic solution,
> not current one in lapic2 branch. Definitely we need to pull in
> kernel apic+ioapic first before the push back. But I think now we
> should make a high level decision on how we will handle this merge
> and how will user space look like. 
> 
Avi:
	Can u show clear light here? We need to go forward quickly:-)
Eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: User space for lapic2

[Avi Kivity]

Dong, Eddie wrote:
> kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
>   
>> Avi Kivity wrote:
>>     
>>> Dong, Eddie wrote:
>>>       
>>>> Avi:
>>>> 	To make lapic code into mainline earlier, I am thinking what
>>>> should the user space code look like.  If we wait till lapic branch
>>>> has all same functionality as mainline have today i.e. live
>>>> migration, all guests etc, we may have very long way to go given
>>>> that only Greg (temply off), me have contribution to the code
>>>> besides you. So what I am thinking is to reduce the bar to "no
>>>> regression", which means a user space command to choice kernel
>>>> pic/apic. When choosing user pic/apic, we make sure everything works
>>>> including livemigration and all known good guests; if choosing
>>>> kernel pic/apic, we have better performance but some feature lagging
>>>> 	such as live migration. Does this make sense? If yes, How about
>>>> command option "-irqchip k" and "-irqchip u"? the default value is
>>>> user. 
>>>>
>>>>         
>>> That doesn't work; one day we'll make kernel apic the default, and
>>> then that new userspace won't work with kernels that have in-kernel
>>> pic but not lapic. 
>>>
>>>       
>> After we get everything settled down, we can switch the default
>> setting to "kernel". BTW, I am talking about generic lapic solution,
>> not current one in lapic2 branch. Definitely we need to pull in
>> kernel apic+ioapic first before the push back. But I think now we
>> should make a high level decision on how we will handle this merge
>> and how will user space look like. 
>>
>>     
> Avi:
> 	Can u show clear light here? We need to go forward quickly:-)
> Eddie
>   

If we release a kernel with pic but not lapic, and userspace that 
defaults to user-pic+lapic, then that kernel will not work with a newer 
userspace that defaults to in-kernel pic+lapic.

We need to switch in one go.  I don't mind checking in patches to the 
lapic2 branch, and continually rebasing it against master, but a release 
will only happen when everything works.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: User space for lapic2

[Dong, Eddie]

> If we release a kernel with pic but not lapic, and userspace that
> defaults to user-pic+lapic, then that kernel will not work
> with a newer
> userspace that defaults to in-kernel pic+lapic.
> 
> We need to switch in one go.  I don't mind checking in patches to the
> lapic2 branch, and continually rebasing it against master, but
> a release
> will only happen when everything works.

I think the difference here is if we can leave the live migration to
future, we all agree lapic should be with pic together and that is
almost there.

Does "everything works" here mean live migration works?

Eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: User space for lapic2

[Avi Kivity]

Dong, Eddie wrote:
>> If we release a kernel with pic but not lapic, and userspace that
>> defaults to user-pic+lapic, then that kernel will not work
>> with a newer
>> userspace that defaults to in-kernel pic+lapic.
>>
>> We need to switch in one go.  I don't mind checking in patches to the
>> lapic2 branch, and continually rebasing it against master, but
>> a release
>> will only happen when everything works.
>>     
>
> I think the difference here is if we can leave the live migration to
> future, we all agree lapic should be with pic together and that is
> almost there.
>
> Does "everything works" here mean live migration works?
>
>   

Yes.

I don't think live migration is particularly difficult.  You need a APIs 
to read and write the PIC+LAPIC states, and you write the state into the 
qemu device model, which already supports migration.  You can even live 
migrate from a userspace PIC+LAPIC to a kernel PIC+LAPIC as the format 
is the same.  CPU registers work the same way in kvm/qemu.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: User space for lapic2

[Dong, Eddie]

> I don't think live migration is particularly difficult.  You
> need a APIs
> to read and write the PIC+LAPIC states, and you write the
> state into the

This is partily true party not. Like what did in Xen, a live migration
in kernel side need to consider not only device state, but also cpu
state. Especially when live migration happens when a VCPU get valid
IDT_Vectoring, we need to save all various cpu states,  (What current
code did to "push back" an vector can't service us with apic in kernel),
and we have NMI state to save too.
I didn't look into user level migration yet, it may be covered or not,
but kernel level migration makes it different.

Anyway I know your position now.


> qemu device model, which already supports migration.  You can
> even live
> migrate from a userspace PIC+LAPIC to a kernel PIC+LAPIC as the format
> is the same.  CPU registers work the same way in kvm/qemu.
> 
> --
> error compiling committee.c: too many arguments to function

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: User space for lapic2

[Gregory Haskins]

On Tue, 2007-07-10 at 22:08 +0800, Dong, Eddie wrote:
> > I don't think live migration is particularly difficult.  You
> > need a APIs
> > to read and write the PIC+LAPIC states, and you write the
> > state into the
> 
> This is partily true party not. 

I agree with Avi here.  The system is already set up to understand that
there are userspace and kernel components that need to save state for a
migration.  Its only a matter of adding any new elements to the set.

> Like what did in Xen, a live migration
> in kernel side need to consider not only device state, but also cpu
> state. 

Right, and this is already true in KVM as well.  

> Especially when live migration happens when a VCPU get valid
> IDT_Vectoring, we need to save all various cpu states,  (What current
> code did to "push back" an vector can't service us with apic in kernel),
> and we have NMI state to save too.

I had already solved these types of issues neatly in the lapic branch,
so perhaps you can model a solution from that (at least in spirit).  For
instance, the irq.deferred source has the IDT_Vectoring state,
irq.pending has the vcpu interrupt state (NMI, EXTINT, etc),
kvm->isa_int has the PIC state, and vcpu->apic has the lapic state, etc.
I never got to adding the live-migration support, but if I had I would
have added elements to the data set for these components (vcpu->irq,
vcpu->apic, kvm->isa_irq) in addition to whats already saved (most of
the rest of the vcpu state).




-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: User space for lapic2

[Dong, Eddie]

> I had already solved these types of issues neatly in the lapic branch,
> so perhaps you can model a solution from that (at least in
> spirit).  For
> instance, the irq.deferred source has the IDT_Vectoring state,
> irq.pending has the vcpu interrupt state (NMI, EXTINT, etc),
> kvm->isa_int has the PIC state, and vcpu->apic has the lapic
> state, etc.
No, what you mean is only for external irq. IDT_Vectoring can happen for
any exception injection.
Eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: User space for lapic2

[Gregory Haskins]

On Wed, 2007-07-11 at 09:34 +0800, Dong, Eddie wrote:

> No, what you mean is only for external irq. IDT_Vectoring can happen for
> any exception injection.

Well, yes, but that really is just an attribute the irq.deferred
"source" needs to maintain, right?  I had been thinking of adding
something like this to distinquish between NMIs and EXTINTs (since they
have different interruptibility rules, etc.  Today they are all treated
as homogeneous EXTINTs once deferred ).

However, there is no reason that concept cannot be extended to support
attributes beyond EXTINT/NMI. E.g. EXCEPTION.  After all, there can only
be one deferred vector at a time and they all follow the general model
of "vector between 0-255", right? 



-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/