Re: [Qemu-devel] QEMU Live Snapshots / Commiting

[Robert P <robp236@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 10258 bytes --]

On Fri, Sep 30, 2011 at 4:54 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:

> On Fri, Sep 30, 2011 at 2:46 PM, Robert P <robp236@gmail.com> wrote:
> >
> >
> > On Fri, Sep 30, 2011 at 2:04 PM, Stefan Hajnoczi <stefanha@gmail.com>
> wrote:
> >>
> >> On Fri, Sep 30, 2011 at 11:32 AM, Robert P <robp236@gmail.com> wrote:
> >>
> >> Please use Reply-All to keep qemu-devel CCed so that others can
> >> contribute.
> >>
> >> >
> >> > On Fri, Sep 30, 2011 at 10:16 AM, Stefan Hajnoczi <stefanha@gmail.com
> >
> >> > wrote:
> >> >>
> >> >> On Thu, Sep 29, 2011 at 09:07:19PM +0200, Robert P wrote:
> >> >> > Hello,
> >> >> >
> >> >> > I still have a problem with the "Live Snapshot" feature of QEMU
> ....
> >> >> > and
> >> >> > before migrating to XEN, VMware or something similare, a quick post
> >> >> > here:
> >> >> >
> >> >> > OS: Ubuntu Natty 64bit
> >> >> >
> >> >> > First, i'm starting my KVM Machine with an image like this:
> >> >> > qemu-img create -f qcow2 -o backing_file=<NameOfBaseImage>
> >> >> > <Snapshotname>
> >> >> >
> >> >> > If i stop the KVM Machine later, and i commit <Snapshotname> into
> >> >> > <NameOfBaseImage>, all the new changes are in the
> <NameOfBaseImage>.
> >> >> > That would be ok.
> >> >> >
> >> >> > ---
> >> >> >
> >> >> > The Problem:
> >> >> >
> >> >> > Actually i'm trying to create "live snapshots" periodically  while
> >> >> > the
> >> >> > machine is running, like this (host2Qemu is just a special function
> >> >> > of
> >> >> > mine
> >> >> > (it works), to send a string to qemu-monitor).
> >> >> >
> >> >> >                 host2Qemu "cont"
> >> >> >                 host2Qemu "guest-agent-fsfreeze"
> >> >> >                 host2Qemu "stop"
> >> >> >
> >> >> >                 host2Qemu "info block"
> >> >> >                 host2Qemu "snapshot_blkdev ide0-hd0 <Snapshot1
> >> >> > (example)>
> >> >> > qcow2"
> >> >> >
> >> >> >                 host2Qemu "cont"
> >> >> >                 host2Qemu "guest-agent-fsthaw"
> >> >> >
> >> >> > My idea is, to commit them one by one afterwards, when the KVM
> >> >> > Machine
> >> >> > is
> >> >> > down into the BaseImage.
> >> >> >
> >> >> > So, the Snapshots are beeing written, and everytime i call that
> >> >> > function
> >> >> > new
> >> >> > data is beeing written to the new "alllocated" snapshot.
> >> >> > BUT, committing of that live-snapshots fails, and i've no idea why
> ?!
> >> >> >
> >> >> > I would commit it like that:
> >> >> >  qemu-img commit -f qcow2 <Snapshot, with KVM was started first>
> >> >> > qemu-img commit -f qcow2 <Snapshot1, newer>
> >> >> > qemu-img commit -f qcow2 <Snapshot1, more new>
> >> >> > ...
> >> >> > and so on.
> >>
> >> You should have something like this:
> >> <NameOfBaseImage> -> <Snapshot1> -> <Snapshot2> -> ... -> <SnapshotN>
> >>
> >> The qemu-img commit command merges down an image file into its backing
> >> file, so you would do:
> >>
> >> $ qemu-img commit <SnapshotN>
> >> ...
> >> $ qemu-img commit <Snapshot2>
> >> $ qemu-img commit <Snapshot1>
> >>
> >> Now all the <Snapshot*> files contain no actual data - the delta have
> >> been committed to their backing files.  You can discard them and use
> >> <NameOfBaseImage> directly.
> >>
> >> How does this compare to what you were trying?
> >
> > Hm, that means you recommend to commit the snapshots from newest to
> oldest
> > into the base-Image?
> > I am committing them (when the KVM is off) from oldest to newest, like
> this:
> >
> > $ qemu-img commit <Snapshot1>
> > $ qemu-img commit <Snapshot2>
> > ....
> > $ qemu-img commit <SnapshotN>
> >
> > Is that maybe causing that problem?
>
> Absolutely.  Each snapshot has a backing file linking back to the
> previous snapshot.  You need to push the latest data all the way back
> to the base backing file.
>

Ok, i've tried that also.
My testcase:

-) start the kvm
"aptitude -f install" reports no error, and the data gets written to "
snapshot-zeus-KVM_APH-2011-09-30-211303.ovl "

-) inside the kvm
"aptitude install csh" is also ok, then i trigger a "takeover" to a new
snapshot: "snapshot-zeus-KVM_APH-2011-09-30-211424.ovl"
(i do it like that in my bash-script:
        "sync")
                writeLog info "USER: somebody triggered a sync"

                host2Qemu "cont"
                host2Qemu "guest-agent-fsfreeze"
                host2Qemu "stop"

                writeLog dbg "Getting the Parameters of qemu-disk"
                host2Qemu "info block"
                writeLog dbg "Creating new  Snapshot $OVL_NAME for writing
..."

                host2Qemu "snapshot_blkdev ide0-hd0 $SNAPSHOT_DIR/$OVL_NAME
qcow2"
                qemu-img create -f qcow2 -o backing_file="$BASE_IMG"
"$SNAPSHOT_DIR/$OVL_NAME" 1>/dev/null

                writeLog dbg "Success. It's now safe to take the last
snapshot. "
                writeLog dbg "RSyncing Snapshots to ha-hpsrv-slave"

                host2Qemu "cont"
                host2Qemu "guest-agent-fsthaw"

                transferSnapshots;;
)
So that means in that way i've described earlier.

Now we should write into the next snapshot
"snapshot-zeus-KVM_APH-2011-09-30-211424.ovl".

-) inside the vm
"aptitude install <someotherpackage>" -> triggering sync again.

New Snapshot:  snapshot-zeus-KVM_APH-2011-09-30-211451.ovl

-) one more time inside the vm
"aptitude install <someotherinterestingpackage>" -> sync.

New Snapshot:
"snapshot-zeus-KVM_APH-2011-09-30-211527"

-) Then i make a commit like (this is an snipplet of my log-File)
Fre Sep 30 21:15:27 CEST 2011 | [zeus:KVM_APH] [INFO] USER: somebody
triggered a sync
Fre Sep 30 21:15:27 CEST 2011 | [zeus:KVM_APH] [DEBUG] New QemuMon Cmd:
cont
Fre Sep 30 21:15:28 CEST 2011 | [zeus:KVM_APH] [DEBUG] New QemuMon Cmd:
guest-agent-fsfreeze
Fre Sep 30 21:15:28 CEST 2011 | [zeus:KVM_APH] [DEBUG] New QemuMon Cmd:
stop
Fre Sep 30 21:15:29 CEST 2011 | [zeus:KVM_APH] [DEBUG] Getting the
Parameters of qemu-disk
Fre Sep 30 21:15:29 CEST 2011 | [zeus:KVM_APH] [DEBUG] New QemuMon Cmd: info
block
Fre Sep 30 21:15:29 CEST 2011 | [zeus:KVM_APH] [DEBUG] Creating new
 Snapshot snapshot-zeus-KVM_APH-2011-09-30-211527.ovl for writing ...
Fre Sep 30 21:15:29 CEST 2011 | [zeus:KVM_APH] [DEBUG] New QemuMon Cmd:
snapshot_blkdev ide0-hd0
/backup/KVM/Aphrodite//Snapshots/snapshot-zeus-KVM_APH-2011-09-30-211527.ovl
qcow2
Fre Sep 30 21:15:30 CEST 2011 | [zeus:KVM_APH] [DEBUG] Success. It's now
safe to take the last snapshot.
Fre Sep 30 21:15:30 CEST 2011 | [zeus:KVM_APH] [DEBUG] RSyncing Snapshots to
ha-hpsrv-slave
Fre Sep 30 21:15:30 CEST 2011 | [zeus:KVM_APH] [DEBUG] New QemuMon Cmd:
cont
Fre Sep 30 21:15:30 CEST 2011 | [zeus:KVM_APH] [DEBUG] New QemuMon Cmd:
guest-agent-fsthaw
Fre Sep 30 21:15:31 CEST 2011 | [zeus:KVM_APH] [DEBUG] RSync Done.
Fre Sep 30 21:15:36 CEST 2011 | [zeus:KVM_APH] [INFO] USER: somebody
triggered a stop
Fre Sep 30 21:15:36 CEST 2011 | [zeus:KVM_APH] [DEBUG] Sending a PowerDown
to KVM (CMD: kvmtool.sh KVM-APH-test send system_powerdown)...
Fre Sep 30 21:15:36 CEST 2011 | [zeus:KVM_APH] [DEBUG] New QemuMon Cmd:
system_powerdown
Fre Sep 30 21:16:06 CEST 2011 | [zeus:KVM_APH] [DEBUG] RSyncing with other
device ...
Fre Sep 30 21:16:06 CEST 2011 | [zeus:KVM_APH] [DEBUG] RSync Done.
Fre Sep 30 21:16:06 CEST 2011 | [zeus:KVM_APH] [DEBUG] Committing changes
into own image, this may take a while ...
Fre Sep 30 21:16:06 CEST 2011 | [zeus:KVM_APH] [INFO] USER: somebody
triggered a commit
Fre Sep 30 21:16:06 CEST 2011 | [zeus:KVM_APH] [DEBUG] Committing
snapshot-zeus-KVM_APH-2011-09-30-211527.ovl to
/backup/KVM/Aphrodite//KVM_Aphrodite_Ubuntu11.04_32bit_Master.img.qcow2
Fre Sep 30 21:16:07 CEST 2011 | [zeus:KVM_APH] [DEBUG] Commited.
Fre Sep 30 21:16:07 CEST 2011 | [zeus:KVM_APH] [DEBUG] Committing
snapshot-zeus-KVM_APH-2011-09-30-211451.ovl to
/backup/KVM/Aphrodite//KVM_Aphrodite_Ubuntu11.04_32bit_Master.img.qcow2
Fre Sep 30 21:16:27 CEST 2011 | [zeus:KVM_APH] [DEBUG] Commited.
Fre Sep 30 21:16:27 CEST 2011 | [zeus:KVM_APH] [DEBUG] Committing
snapshot-zeus-KVM_APH-2011-09-30-211424.ovl to
/backup/KVM/Aphrodite//KVM_Aphrodite_Ubuntu11.04_32bit_Master.img.qcow2
Fre Sep 30 21:16:40 CEST 2011 | [zeus:KVM_APH] [DEBUG] Commited.
Fre Sep 30 21:16:40 CEST 2011 | [zeus:KVM_APH] [DEBUG] Committing
snapshot-zeus-KVM_APH-2011-09-30-211303.ovl to
/backup/KVM/Aphrodite//KVM_Aphrodite_Ubuntu11.04_32bit_Master.img.qcow2
Fre Sep 30 21:16:42 CEST 2011 | [zeus:KVM_APH] [DEBUG] Commited.
Fre Sep 30 21:16:42 CEST 2011 | [zeus:KVM_APH] [INFO] Stopped Virtual
Machine

So that means, the order should be correct now, how the snapshots get
commited into the base Image.

-) i restart the machine, log in again and the state is almost like it was
when we started initially, but when i make a
"aptitude -f install" then the filesystem seems to be corrupt, and also none
of that packages from before are installed:

[root@aphrodite ~]# aptitude -f install
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.
localepurge: Disk space freed in /usr/share/locale: 0 KiB
du: cannot access `/usr/share/man/man1/ksh.1.gz': Input/output error
2011 Sep 30 21:17:44 aphrodite [   41.250887] EXT4-fs (sda1): Remounting
filesystem read-only
2011 Sep 30 21:17:44 aphrodite [   41.444655] EXT4-fs error (device sda1):
ext4_journal_start_sb:260: Detected aborted journal
du: cannot access `/usr/share/man/man1/tcsh.1.gz': Input/output error
du: cannot access `/usr/share/man/man1/ksh.1.gz': Input/output error
du: cannot access `/usr/share/man/man1/tcsh.1.gz': Input/output error
localepurge: Disk space freed in /usr/share/man: 0 KiB
localepurge: Disk space freed in /usr/share/omf: 0 KiB

Total disk space freed by localepurge: 0 KiB

E: Failed to write temporary StateFile /var/lib/apt/extended_states.tmp
W: Not using locking for read only lock file /var/lib/dpkg/lock
W: Not using locking for read only lock file /var/lib/dpkg/lock
E: Unable to write to /var/cache/apt/
E: The package lists or status file could not be parsed or opened.

I guess there got something corrupted in the filesystem after the commits
.... I've tried that a couple of times now, result is everytime the same  :(

Thanks.
Robert


> Stefan
>

[-- Attachment #2: Type: text/html, Size: 14511 bytes --]