live migration with DRBD devices

live migration with DRBD devices

[Michael Mey]

[-- Attachment #1.1: Type: text/plain, Size: 2877 bytes --]

Hi all,

I want to do live migration of domains using DRBD block devices for 
synchronisation between the two Xen hosts with Xen 2.0.7. 

I've written a script ( block-drbd ) which binds / unbinds DRBD devices for 
specific domains (with DRBD, only one host can write on a drbd device at the 
same time). This script sets the current host as master (bind) or secondary 
(unbind) in a DRBD "Netword-RAID"

Bringing up a domain works fine. But when I try to migrate it ( live with -l 
flag as well as non-live) I get an error from xfrd. 
The xfrd.log says:


==> xfrd.log <==
2508 [INF] XFRD> Accepted connection from 127.0.0.1:1327 on 2
2790 [INF] XFRD> Xfr service for 127.0.0.1:1327
[DEBUG] Conn_init> flags=1
[DEBUG] Conn_init> write stream...
[DEBUG] stream_init>mode=w flags=1 compress=0
[DEBUG] stream_init> unbuffer...
[DEBUG] stream_init< err=0
[DEBUG] Conn_init> read stream...
[DEBUG] stream_init>mode=r flags=1 compress=0
[DEBUG] stream_init> unbuffer...
[DEBUG] stream_init< err=0
[DEBUG] Conn_sxpr>
(xfr.hello 1 0)[DEBUG] Conn_sxpr< err=0
[DEBUG] Conn_sxpr>
(xfr.migrate 8 "(domain (id 8) (name Xen0) (memory 511) (maxmem 512) (state 
-b---) (cpu 0) (cpu_time 7.548731537) (up_time 1953.19617701) (start_time 
1130242370.96) (console (status listening) (id 12) (domain 8) (local_port 12) 
(remote_port 1) (console_port 9608)) (devices (vif (idx 0) (vif 0) (mac 
aa:00:00:00:42:00) (vifname vif8.0) (bridge xen-br0) (evtchn 14 4) (index 0)) 
(vbd (idx 0) (vdev 769) (device 37632) (mode w) (dev hda1) (uname drbd:drbd0) 
(node /dev/drbd0) (index 0)) (vbd (idx 1) (vdev 770) (device 37633) (mode w) 
(dev hda2) (uname drbd:drbd1) (node /dev/drbd1) (index 1))) (config (vm (name 
Xen0) (memory 512) (image (linux (kernel /boot/vmlinuz-2.6.11.10-xenU) (root 
'/dev/hda1 ro'))) (device (vbd (uname drbd:drbd0) (dev hda1) (mode w))) 
(device (vbd (uname drbd:drbd1) (dev hda2) (mode w))) (device (vif (mac 
aa:00:00:00:42:00) (bridge xen-br0))))))" testpc-023 8002 1 0)[DEBUG] 
Conn_sxpr< err=0
[DEBUG] Conn_connect> addr=192.168.111.23:8002
[DEBUG] Conn_init> flags=1
[DEBUG] Conn_init> write stream...
[DEBUG] stream_init>mode=w flags=1 compress=0
[DEBUG] stream_init> unbuffer...
[DEBUG] stream_init< err=0
[DEBUG] Conn_init> read stream...
[DEBUG] stream_init>mode=r flags=1 compress=0
[DEBUG] stream_init> unbuffer...
[DEBUG] stream_init< err=0
[DEBUG] Conn_sxpr>
(xfr.err 0)[DEBUG] Conn_sxpr< err=0

[1130244324.172538] xc_linux_save start 8

xc_linux_save start 8
[1130244324.174077] Saving memory pages: iter 1   0%
Saving memory pages: iter 1   0%Error when writing to state file (5)
Error when writing to state file (5)
2790 [WRN] XFRD> Transfer errors:
2790 [WRN] XFRD> state=XFR_STATE    err=1
2790 [INF] XFRD> Xfr service err=1


Does anybody have an idea? I'd appreciate.

Cheers,
Mike

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: live migration with DRBD devices

[Mark Williamson]

> I want to do live migration of domains using DRBD block devices for
> synchronisation between the two Xen hosts with Xen 2.0.7.
>
> I've written a script ( block-drbd ) which binds / unbinds DRBD devices for
> specific domains (with DRBD, only one host can write on a drbd device at
> the same time). This script sets the current host as master (bind) or
> secondary (unbind) in a DRBD "Netword-RAID"

Yep - btw this is fixed with DRBD 8 (still in development), which supports 
multiple writers.

How are you handling the sequencing here, though?  Move the device to the new 
host at *unbind* time from the origin?  I'm thinking that the DRBD script 
will get run at the destination whilst the origin domain is still running and 
using its disk...

> xc_linux_save start 8
> [1130244324.174077] Saving memory pages: iter 1   0%
> Saving memory pages: iter 1   0%Error when writing to state file (5)
> Error when writing to state file (5)
> 2790 [WRN] XFRD> Transfer errors:
> 2790 [WRN] XFRD> state=XFR_STATE    err=1
> 2790 [INF] XFRD> Xfr service err=1
>
>
> Does anybody have an idea? I'd appreciate.

Weird...  Does this work without your script?  Does migration work with a 
simpler domain?  If you boot a domain from an initrd (removing disk 
dependencies), will that migrate correctly?

If it's something weird in the migration-related code, somebody else might 
have some further ideas as to what caused this...

Cheers,
Mark

Re: live migration with DRBD devices

[Michael Paesold]

Mark Williamson wrote:
>>I want to do live migration of domains using DRBD block devices for
>>synchronisation between the two Xen hosts with Xen 2.0.7.
>>
>>I've written a script ( block-drbd ) which binds / unbinds DRBD devices for
>>specific domains (with DRBD, only one host can write on a drbd device at
>>the same time). This script sets the current host as master (bind) or
>>secondary (unbind) in a DRBD "Netword-RAID"
> 
> 
> Yep - btw this is fixed with DRBD 8 (still in development), which supports 
> multiple writers.
> 
> How are you handling the sequencing here, though?  Move the device to the new 
> host at *unbind* time from the origin?  I'm thinking that the DRBD script 
> will get run at the destination whilst the origin domain is still running and 
> using its disk...

Correct.

And it cannot work with drbd 0.7 currently without hacking Xen to change 
block device setup. What would be needed is that Xen should open the new 
block device read-only (there is a hack in drbd 0.7 that would allow 
this) and only when the source domain is shutdown reopen the block 
device r/w.

Xen want's to open the block device rw as soon as it does the setup of 
the destination domain. This may work currectly with drbd 0.8 since it 
supports multi-master.

I have done testing with the current development trunk of drbd some time 
ago. It did not work for me. Migration crashed the guest.

Best Regards,
Michael Paesold

Re: live migration with DRBD devices

[Michael Mey]

[-- Attachment #1.1: Type: text/plain, Size: 2513 bytes --]

On Tuesday 25 October 2005 16:49, Mark Williamson wrote:
> > I want to do live migration of domains using DRBD block devices for
> > synchronisation between the two Xen hosts with Xen 2.0.7.
> >
> > I've written a script ( block-drbd ) which binds / unbinds DRBD devices
> > for specific domains (with DRBD, only one host can write on a drbd device
> > at the same time). This script sets the current host as master (bind) or
> > secondary (unbind) in a DRBD "Netword-RAID"
>
> Yep - btw this is fixed with DRBD 8 (still in development), which supports
> multiple writers.
I'll give it a try, although I'm expecting it being not that successful 
because of its development state and Michael's experiences.

> How are you handling the sequencing here, though?  Move the device to the
> new host at *unbind* time from the origin?  I'm thinking that the DRBD
> script will get run at the destination whilst the origin domain is still
> running and using its disk...
So the solution would be to get Xen mounting the device in the following order 
for migration as Michael Paesold suggested:

1) create domain on target host and mount the drbd device r/o
2) do the rest of the migration, e.g. iterative copy of memory ...
3) unmount or mount the drbd device on the source host r/o
4) mount drbd device r/w on target host
5) start domain on target host

Is there any further developer's documentation for Xen 2.0.7?
Or maybe anybody can give me some hints at approx. which points I'd have to 
patch Xen to get that running? That would help me a lot saving some time for 
trying to implement that because I'm not so deep into Xen right now.

> > xc_linux_save start 8
> > [1130244324.174077] Saving memory pages: iter 1   0%
> > Saving memory pages: iter 1   0%Error when writing to state file (5)
> > Error when writing to state file (5)
> > 2790 [WRN] XFRD> Transfer errors:
> > 2790 [WRN] XFRD> state=XFR_STATE    err=1
> > 2790 [INF] XFRD> Xfr service err=1
> >
> >
> > Does anybody have an idea? I'd appreciate.
>
> Weird...  Does this work without your script?  Does migration work with a
> simpler domain?  If you boot a domain from an initrd (removing disk
> dependencies), will that migrate correctly?
>
> If it's something weird in the migration-related code, somebody else might
> have some further ideas as to what caused this...
Migration without my script works fine, so there shouldn't be anything weird 
in the migration-related code.

Regards,
Mike

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel