live migration between kernel/user irqchip

live migration between kernel/user irqchip

[Dong, Eddie]

When thinking about live migration support for in kernel irqchip, one
question comes out which need to be solved first:
Do we need to support live migration among user level irqchip and kernel
level? If the answer is yes, kernel level irqchip must keep same state
with user level, i.e. if Qemu changes the pic/apic/ioapic state
definition, we need to do corresponding changes too, otherwise kernel
side can stay as it is.
thx,eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: live migration between kernel/user irqchip

[Avi Kivity]

Dong, Eddie wrote:
> When thinking about live migration support for in kernel irqchip, one
> question comes out which need to be solved first:
> Do we need to support live migration among user level irqchip and kernel
> level? 

Yes.

> If the answer is yes, kernel level irqchip must keep same state
> with user level, i.e. if Qemu changes the pic/apic/ioapic state
> definition, we need to do corresponding changes too, otherwise kernel
> side can stay as it is.
>   

The qemu state mimics the device state and should be independent of 
implementation details.  If the qemu state is added too, this most 
likely indicates a but that needs to be fixed in kvm as well.

The best way to do live migration is to copy the kernel state into the 
qemu device model, and let qemu do state serialization.  This ensures 
compatibility.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: live migration between kernel/user irqchip

[Dong, Eddie]

Avi Kivity wrote:
> Dong, Eddie wrote:
>> When thinking about live migration support for in kernel irqchip, one
>> question comes out which need to be solved first:
>> Do we need to support live migration among user level irqchip and
>> kernel level?
> 
> Yes.
> 
>> If the answer is yes, kernel level irqchip must keep same state
>> with user level, i.e. if Qemu changes the pic/apic/ioapic state
>> definition, we need to do corresponding changes too, otherwise
>> kernel side can stay as it is. 
>> 
> 
> The qemu state mimics the device state and should be independent of
> implementation details.  If the qemu state is added too, this most
> likely indicates a but that needs to be fixed in kvm as well.
> 
> The best way to do live migration is to copy the kernel state into the
> qemu device model, and let qemu do state serialization.  This ensures

"copy" means we define some device "state" both in kernel & user. 
I.e.  kernel use user level state definition.

> compatibility. 
> 
> --
> error compiling committee.c: too many arguments to function

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: live migration between kernel/user irqchip

[Avi Kivity]

Dong, Eddie wrote:
> Avi Kivity wrote:
>   
>> Dong, Eddie wrote:
>>     
>>> When thinking about live migration support for in kernel irqchip, one
>>> question comes out which need to be solved first:
>>> Do we need to support live migration among user level irqchip and
>>> kernel level?
>>>       
>> Yes.
>>
>>     
>>> If the answer is yes, kernel level irqchip must keep same state
>>> with user level, i.e. if Qemu changes the pic/apic/ioapic state
>>> definition, we need to do corresponding changes too, otherwise
>>> kernel side can stay as it is. 
>>>
>>>       
>> The qemu state mimics the device state and should be independent of
>> implementation details.  If the qemu state is added too, this most
>> likely indicates a but that needs to be fixed in kvm as well.
>>
>> The best way to do live migration is to copy the kernel state into the
>> qemu device model, and let qemu do state serialization.  This ensures
>>     
>
> "copy" means we define some device "state" both in kernel & user. 
> I.e.  kernel use user level state definition.
>   

Sorry, I don't understand.

Here's apic_save(), for example:

static void apic_save(QEMUFile *f, void *opaque)
{
    APICState *s = opaque;
    int i;

    qemu_put_be32s(f, &s->apicbase);
    qemu_put_8s(f, &s->id);
    qemu_put_8s(f, &s->arb_id);
    qemu_put_8s(f, &s->tpr);
    qemu_put_be32s(f, &s->spurious_vec);
    qemu_put_8s(f, &s->log_dest);
    qemu_put_8s(f, &s->dest_mode);
    for (i = 0; i < 8; i++) {
        qemu_put_be32s(f, &s->isr[i]);
        qemu_put_be32s(f, &s->tmr[i]);
        qemu_put_be32s(f, &s->irr[i]);
    }
    for (i = 0; i < APIC_LVT_NB; i++) {
        qemu_put_be32s(f, &s->lvt[i]);
    }
    qemu_put_be32s(f, &s->esr);
    qemu_put_be32s(f, &s->icr[0]);
    qemu_put_be32s(f, &s->icr[1]);
    qemu_put_be32s(f, &s->divide_conf);
    qemu_put_be32s(f, &s->count_shift);
    qemu_put_be32s(f, &s->initial_count);
    qemu_put_be64s(f, &s->initial_count_load_time);
    qemu_put_be64s(f, &s->next_time);

    qemu_put_timer(f, s->timer);
}

Do you see state here that does not exist in the kernel?  Or kernel 
state that is missing from here?

->timer and ->next_time may be non-trivial, but both are needed.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: live migration between kernel/user irqchip

[Dong, Eddie]

>> 
>> "copy" means we define some device "state" both in kernel & user.
>> I.e.  kernel use user level state definition.
>> 
> 
> Sorry, I don't understand.
> 
> Here's apic_save(), for example:
> 
> static void apic_save(QEMUFile *f, void *opaque)
> {
>    APICState *s = opaque;
>    int i;
> 
>    qemu_put_be32s(f, &s->apicbase);
>    qemu_put_8s(f, &s->id);
>    qemu_put_8s(f, &s->arb_id);
>    qemu_put_8s(f, &s->tpr);
>    qemu_put_be32s(f, &s->spurious_vec);
>    qemu_put_8s(f, &s->log_dest);
>    qemu_put_8s(f, &s->dest_mode);
>    for (i = 0; i < 8; i++) {
>        qemu_put_be32s(f, &s->isr[i]);
>        qemu_put_be32s(f, &s->tmr[i]);
>        qemu_put_be32s(f, &s->irr[i]);
>    }
>    for (i = 0; i < APIC_LVT_NB; i++) {
>        qemu_put_be32s(f, &s->lvt[i]);
>    }
>    qemu_put_be32s(f, &s->esr);
>    qemu_put_be32s(f, &s->icr[0]);
>    qemu_put_be32s(f, &s->icr[1]);
>    qemu_put_be32s(f, &s->divide_conf);
>    qemu_put_be32s(f, &s->count_shift);
>    qemu_put_be32s(f, &s->initial_count);
>    qemu_put_be64s(f, &s->initial_count_load_time);
>    qemu_put_be64s(f, &s->next_time);
> 
>    qemu_put_timer(f, s->timer);
> }
> 
> Do you see state here that does not exist in the kernel?  Or kernel
> state that is missing from here?
> 
> ->timer and ->next_time may be non-trivial, but both are needed.
> 

timer state is different, krenel uses hrtimer. But maybe not that
critical.

Maybe I didn't exactly understand what "copy" means for you. There are 2
ways 
in my mind to do save/restore:
1:  We provide a new apic_load/apic_save function. In this function we
may convert kernel state into user level state.


2: We use user level apic_load/apic_save, but at some point we read the
whole kernel state and then copy to user level APICState data structure
(use save as example).  IN this way normal user level save/load will
service us, but then kernel side struct kvm_apic should be same with
APICState.

BTW, we will take half week leave next week, will u work on apic too?
thx,eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: live migration between kernel/user irqchip

[Avi Kivity]

Dong, Eddie wrote:
>>> "copy" means we define some device "state" both in kernel & user.
>>> I.e.  kernel use user level state definition.
>>>
>>>       
>> Sorry, I don't understand.
>>
>> Here's apic_save(), for example:
>>
>> static void apic_save(QEMUFile *f, void *opaque)
>> {
>>    APICState *s = opaque;
>>    int i;
>>
>>    qemu_put_be32s(f, &s->apicbase);
>>    qemu_put_8s(f, &s->id);
>>    qemu_put_8s(f, &s->arb_id);
>>    qemu_put_8s(f, &s->tpr);
>>    qemu_put_be32s(f, &s->spurious_vec);
>>    qemu_put_8s(f, &s->log_dest);
>>    qemu_put_8s(f, &s->dest_mode);
>>    for (i = 0; i < 8; i++) {
>>        qemu_put_be32s(f, &s->isr[i]);
>>        qemu_put_be32s(f, &s->tmr[i]);
>>        qemu_put_be32s(f, &s->irr[i]);
>>    }
>>    for (i = 0; i < APIC_LVT_NB; i++) {
>>        qemu_put_be32s(f, &s->lvt[i]);
>>    }
>>    qemu_put_be32s(f, &s->esr);
>>    qemu_put_be32s(f, &s->icr[0]);
>>    qemu_put_be32s(f, &s->icr[1]);
>>    qemu_put_be32s(f, &s->divide_conf);
>>    qemu_put_be32s(f, &s->count_shift);
>>    qemu_put_be32s(f, &s->initial_count);
>>    qemu_put_be64s(f, &s->initial_count_load_time);
>>    qemu_put_be64s(f, &s->next_time);
>>
>>    qemu_put_timer(f, s->timer);
>> }
>>
>> Do you see state here that does not exist in the kernel?  Or kernel
>> state that is missing from here?
>>
>> ->timer and ->next_time may be non-trivial, but both are needed.
>>
>>     
>
> timer state is different, krenel uses hrtimer. But maybe not that
> critical.
>
> Maybe I didn't exactly understand what "copy" means for you. There are 2
> ways 
> in my mind to do save/restore:
> 1:  We provide a new apic_load/apic_save function. In this function we
> may convert kernel state into user level state.
>
>
> 2: We use user level apic_load/apic_save, but at some point we read the
> whole kernel state and then copy to user level APICState data structure
> (use save as example).  IN this way normal user level save/load will
> service us, but then kernel side struct kvm_apic should be same with
> APICState.
>   

This is what I'd like.  The kernel apic state doesn't have to be exactly
equal to APICState; we just have to write functions to convert into
APICState and back.  That ensures the protocol doesn't change.

(in this way we can even switch kernel apic on and off at runtime (not
that we want to)).


> BTW, we will take half week leave next week, will u work on apic too?
> thx,eddie
>   

Yes.  Let me know when you leave so we don't conflict.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: live migration between kernel/user irqchip

[Dong, Eddie]

> 
> This is what I'd like.  The kernel apic state doesn't have to
> be exactly
> equal to APICState; we just have to write functions to convert into
> APICState and back.  That ensures the protocol doesn't change.
> 
> (in this way we can even switch kernel apic on and off at runtime
> (not that we want to)). 
> 
> 
>> BTW, we will take half week leave next week, will u work on apic
>> too? thx,eddie 
>> 
> 
> Yes.  Let me know when you leave so we don't conflict.
> 
I will leave next Tuesday evening :-(

BTW, can anybody do more testing on stability of lapic2 branch after
you pulled in 3 pending patches? I will ask Intel guys to do too. I 
expect there will have some time to get it into certain level of
stability.
Eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/