[Qemu-devel] Tracking memory dirtying in QEMU

[Qemu-devel] Tracking memory dirtying in QEMU

[Anthony Liguori]

Howdy,

I've been working on migration for QEMU and have run into a snag.  I've 
got a non-live migration patch that works quite happily[1].  I modified 
the save/restore code to not seek at all, and then basically pipe a save 
over a pipe to a subprocess (usually, ssh).

Conceptually, adding support for live migration is really easy.  All I 
think I need to do is extend the current code, to have a pre-save hook 
that is activated before the VM is stopped.  This hook will be called 
until it says it's done and then the rest of the save/load handlers are 
invoked.  At first, I'm just going to do a pre-save handler for RAM 
which should significantly reduce the amount of down time.  I think the 
only other device we'll have to handle specially is the VGA memory but 
I'm happy to ignore that for now.

So, all I really need is to be able to track which pages are dirtied.  I 
also need the a method to reset the dirty map.

I started looking at adding another map like phys_ram_dirty.  That seems 
to work for some of the IO_MEM_RAM pages, but not all.  My initial 
thought is that all memory operations should go through one of the 
st[bwl]_phys functions but that doesn't seem to be the case.

Can anyone provide me with some advice on how to do this?  Am I right in 
assuming that all IO will go through some function?

[1] http://hg.codemonkey.ws/qemu-pq/?f=758c26c82f52;file=qemu-migration.diff

Thanks,

Anthony Liguori

Re: [Qemu-devel] Tracking memory dirtying in QEMU

[Fabrice Bellard]

Anthony Liguori wrote:
> Howdy,
> 
> I've been working on migration for QEMU and have run into a snag.  I've 
> got a non-live migration patch that works quite happily[1].  I modified 
> the save/restore code to not seek at all, and then basically pipe a save 
> over a pipe to a subprocess (usually, ssh).

qumranet has written some code to do live migration too. IMHO, 
client/server code should be integrated in QEMU in order to ease the use 
of live migration.

> Conceptually, adding support for live migration is really easy.  All I 
> think I need to do is extend the current code, to have a pre-save hook 
> that is activated before the VM is stopped.  This hook will be called 
> until it says it's done and then the rest of the save/load handlers are 
> invoked.  At first, I'm just going to do a pre-save handler for RAM 
> which should significantly reduce the amount of down time.  I think the 
> only other device we'll have to handle specially is the VGA memory but 
> I'm happy to ignore that for now.
> 
> So, all I really need is to be able to track which pages are dirtied.  I 
> also need the a method to reset the dirty map.
> 
> I started looking at adding another map like phys_ram_dirty.  That seems 
> to work for some of the IO_MEM_RAM pages, but not all.  My initial 
> thought is that all memory operations should go through one of the 
> st[bwl]_phys functions but that doesn't seem to be the case.
> 
> Can anyone provide me with some advice on how to do this?  Am I right in 
> assuming that all IO will go through some function?

RAM access is not handled via I/O for efficiency, but the phys_ram_dirty 
flags are always up to date. In order to use it, you must allocate one 
bit in the dirty flags not used by QEMU and kqemu. Then you can use:

cpu_physical_memory_reset_dirty() to mark a page as not dirty and 
cpu_physical_memory_get_dirty() to test for dirtiness.

Note that for performance reasons the dirty bits are not handled while 
QEMU modifies the A and D bits in the PTEs and it can be a problem for 
your application.

FYI, the dirty bits are currently used in QEMU to optimize VGA refreshs 
and to track self modifying code. They are also used internally by kqemu.

Regards,

Fabrice.

Re: [Qemu-devel] Tracking memory dirtying in QEMU

[Anthony Liguori]

Fabrice Bellard wrote:
> Anthony Liguori wrote:
>> Howdy,
>>
>> I've been working on migration for QEMU and have run into a snag.  
>> I've got a non-live migration patch that works quite happily[1].  I 
>> modified the save/restore code to not seek at all, and then basically 
>> pipe a save over a pipe to a subprocess (usually, ssh).
>
> qumranet has written some code to do live migration too.

Yes, I looked at their code before starting.  They don't currently do 
live migration (only offline migration which is really just a 
save/restore over the network).  I found their implementation to be a 
bit overly complicated so I decided to try my own.

> IMHO, client/server code should be integrated in QEMU in order to ease 
> the use of live migration.

Agreed.  What I'd like to eventually have is a migrate command that took 
a URL.  For instance:

(qemu) migrate ssh://woolly
(qemu) migrate tcp://woolly:8001

While still maintaining the ability to specific a full ssh command.  
This is useful if for many things (like changing to location of disks in 
case you mount an NFS partition in different locations).  While I see 
the value in a TCP transport, I personally want an SSH transport so I 
don't to bother setting up a server.

The current code is just something that works.
>> Can anyone provide me with some advice on how to do this?  Am I right 
>> in assuming that all IO will go through some function?
>
> RAM access is not handled via I/O for efficiency, but the 
> phys_ram_dirty flags are always up to date. In order to use it, you 
> must allocate one bit in the dirty flags not used by QEMU and kqemu. 
> Then you can use:
>
> cpu_physical_memory_reset_dirty() to mark a page as not dirty and 
> cpu_physical_memory_get_dirty() to test for dirtiness.

I didn't realize that you could reset an individual bit and it would be 
updated appropriately--but it definitely works.  This simplifies things 
a lot.  Thanks!

> Note that for performance reasons the dirty bits are not handled while 
> QEMU modifies the A and D bits in the PTEs and it can be a problem for 
> your application.

Right, I saw that.  For now, I've modified the stl_phys_notdirty() 
function to update the dirty bitmap iff there is an active migration 
going on.  I can wrap this check in an unexpected() so there should be 
very little performance impact.

I've noticed that while this works fine for QEMU and user KQEMU, it 
doesn't work with kernel KQEMU.  I suspect this is because kernel KQEMU 
is updating A/D bits without updating the dirty bitmap.  Hopefully you 
could fix that once we get migration working.

> FYI, the dirty bits are currently used in QEMU to optimize VGA 
> refreshs and to track self modifying code. They are also used 
> internally by kqemu.

Does KQEMU use bits other than 1,2 (VGA_DIRTY_FLAG, CODE_DIRTY_FLAG)?  
I've added a MIGRATION_DIRTY_FLAG as bit 3 and there doesn't seem to be 
a conflict.

Thanks again Fabrice,

Anthony Liguori

> Regards,
>
> Fabrice.
>
>
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel