[linux-lvm] LVM snapshots in a iSCSI and XenSource environment

[linux-lvm] LVM snapshots in a iSCSI and XenSource environment

[S. J. van Harmelen]

Hi list,

In advance my excusses for this radar long post (although it's easy
readable ;), but I want to make sure that I understand it correctly so I
don't end up making a very costly mistake.

I have a storage server (Debian Etch) with mutlipath-tools running and
on top of that I use IET iscsi-target software to export the multipathed
device to a XenSource server.

XenSource creates a PV on the entire exported disk, and then creates a
few LV's when I create some virtual machines.

Now I would like to take snapshots of these virtual machines as a
backup. So I want to take a snapshot every day, but hold them for only
one day. These are quite static machines, so I need want several
snapshots per machine. Just one in case something happens or a bad
adjustment is made.

I asume that when I drop the snapshot just before creating the new one,
all changes are merged back to the original volume. Correct? While this
merging is happening, does the disk then becomes unavailable to the
virtual machine, or doesn't the virtual machine doesn't notice the
merge?

Also I was wondering if it's a smart idea to create a PV and a LV (both
spanning the whole disk) on the storage server, and then exporting the
LV true iSCSI to the XenSource server. In that way I can take the
snapshots on to storage server directly.

Questions that I think of then are if it's not a problem that XenSource
then creates a new PV and some LV's in je LV I created adn exported on
the storage server. Is that a problem, or should this work fine?

And another question is how I can then restore a single LV Xen created,
from the snapshot of the LV that spans the whole disk on the storage
server? In that case I can not just revert to the old disk before taking
the snapshot, because then all the LV's created by Xen will be set back
to that point, and not just the LV that went bad.

Last question... Without creating a PV and a LV on the storage server
and just letting XenSource create what it needs to provision the virtual
machines, I can still see the PV and the LV's Xen created on the storage
server. Could I take the snapshots from there, although the PV and LV's
where not created here?

Hope someone can tell me what the best option is, and if all this is
possible?

Thanks!!

Sander

Re: [linux-lvm] LVM snapshots in a iSCSI and XenSource environment

[Tomasz Chmielewski]

S. J. van Harmelen schrieb:
> Hi list,
> 
> In advance my excusses for this radar long post (although it's easy
> readable ;), but I want to make sure that I understand it correctly so I
> don't end up making a very costly mistake.
> 
> I have a storage server (Debian Etch) with mutlipath-tools running and
> on top of that I use IET iscsi-target software to export the multipathed
> device to a XenSource server.
> 
> XenSource creates a PV on the entire exported disk, and then creates a
> few LV's when I create some virtual machines.

Which IMO is a pity, as logically, LVM exists and is usable on that 
given Xen server only. This means you can't really use multiple Xen 
servers, live migration etc.


> Now I would like to take snapshots of these virtual machines as a
> backup. So I want to take a snapshot every day, but hold them for only
> one day. These are quite static machines, so I need want several
> snapshots per machine. Just one in case something happens or a bad
> adjustment is made.

OK, should work.


> I asume that when I drop the snapshot just before creating the new one,
> all changes are merged back to the original volume. Correct? While this
> merging is happening, does the disk then becomes unavailable to the
> virtual machine, or doesn't the virtual machine doesn't notice the
> merge?

I believe there is no "merging back", as the changes are being made to 
the snapshot volume. If you drop the snapshot, it just doesn't exist 
anymore, and the original volume does exist as it did before.
So yes, you can safely drop the snapshots, the process is transparent.


> Also I was wondering if it's a smart idea to create a PV and a LV (both
> spanning the whole disk) on the storage server, and then exporting the
> LV true iSCSI to the XenSource server. In that way I can take the
> snapshots on to storage server directly.

Yes, it is indeed better and more flexible solution.


> Questions that I think of then are if it's not a problem that XenSource
> then creates a new PV and some LV's in je LV I created adn exported on
> the storage server. Is that a problem, or should this work fine?

Hmm? I don't think I understand what you mean.


> And another question is how I can then restore a single LV Xen created,
> from the snapshot of the LV that spans the whole disk on the storage
> server? In that case I can not just revert to the old disk before taking
> the snapshot, because then all the LV's created by Xen will be set back
> to that point, and not just the LV that went bad.

# Will only work if snapshot size is equal or greater than
# the original volume

dd if=/dev/LVM/volume-snapshot of=/dev/LVM/volume

# or, if the allowed snapshot size is smaller, we don't want our
# precious snapshot dropped

dd if=/dev/LVM/volume-snapshot of=/dev/LVM/new-volume
dd if=/dev/LVM/new-volume of=/dev/LVM/volume


> Last question... Without creating a PV and a LV on the storage server
> and just letting XenSource create what it needs to provision the virtual
> machines, I can still see the PV and the LV's Xen created on the storage
> server. Could I take the snapshots from there, although the PV and LV's
> where not created here?

Only when you don't use them on the Xen server at the same time (no 
iSCSI connection; LVM modules on the Xen server unloaded, LVM modules on 
the storage server loaded).
Which brings us to a point: it is better to manage LVMs on a storage 
server than on a Xen host.


-- 
Tomasz Chmielewski
http://lists.wpkg.org

Re: [linux-lvm] LVM snapshots in a iSCSI and XenSource environment

[S. J. van Harmelen]

Thanks for your reaction!

On Tue, 2007-11-20 at 12:37 +0100, Tomasz Chmielewski wrote:
> S. J. van Harmelen schrieb:
> > Hi list,
> > 
> > In advance my excusses for this radar long post (although it's easy
> > readable ;), but I want to make sure that I understand it correctly so I
> > don't end up making a very costly mistake.
> > 
> > I have a storage server (Debian Etch) with mutlipath-tools running and
> > on top of that I use IET iscsi-target software to export the multipathed
> > device to a XenSource server.
> > 
> > XenSource creates a PV on the entire exported disk, and then creates a
> > few LV's when I create some virtual machines.
> 
> Which IMO is a pity, as logically, LVM exists and is usable on that 
> given Xen server only. This means you can't really use multiple Xen 
> servers, live migration etc.

Are you sure about that? Accoring to Xen lvm over iSCSI is considered as
shared storage that can be used for live migration
(http://docs.xensource.com/XenServer/4.0.1/installation/ch03s03.html):

<snip>
3.3.3. XenServer Hosts with shared iSCSI storage

Adding shared storage to the XenServer network enables grouping of
XenServer Hosts into Resource Pools, enabling live relocation of VMs and
sharing of server resources.

Requirements

      * two or more x86 servers with local storage 
      * one or more Windows workstations, on same network as the
        XenServer Hosts 
      * a server providing a shared directory via iSCSI
<snip>
> 
> > Now I would like to take snapshots of these virtual machines as a
> > backup. So I want to take a snapshot every day, but hold them for only
> > one day. These are quite static machines, so I need want several
> > snapshots per machine. Just one in case something happens or a bad
> > adjustment is made.
> 
> OK, should work.
> 
> 
> > I asume that when I drop the snapshot just before creating the new one,
> > all changes are merged back to the original volume. Correct? While this
> > merging is happening, does the disk then becomes unavailable to the
> > virtual machine, or doesn't the virtual machine doesn't notice the
> > merge?
> 
> I believe there is no "merging back", as the changes are being made to 
> the snapshot volume. If you drop the snapshot, it just doesn't exist 
> anymore, and the original volume does exist as it did before.
> So yes, you can safely drop the snapshots, the process is transparent.
> 
> 
> > Also I was wondering if it's a smart idea to create a PV and a LV (both
> > spanning the whole disk) on the storage server, and then exporting the
> > LV true iSCSI to the XenSource server. In that way I can take the
> > snapshots on to storage server directly.
> 
> Yes, it is indeed better and more flexible solution.
> 
> 
> > Questions that I think of then are if it's not a problem that XenSource
> > then creates a new PV and some LV's in je LV I created adn exported on
> > the storage server. Is that a problem, or should this work fine?
> 
> Hmm? I don't think I understand what you mean.

I mean I will have 2 sets of PV's and LV's. One set on the storage
server (that has one PV and one LV that both span the whole disk), and
one set op the Xen server (the ones that Xen makes by itself when I add
a new vitrual machine).

Now the one LV from the storage server that Xen sees is being exported
true iSCSI, so it wouldn't know that it is infact a LV that it's talking
to. As far as the Xen server conserns this is just a raw disk. So it
will then just create the needed PV and LV's on it to provision virtual
mahines.

The question was if this PV/LV in another PV/LV (on another physical
machine) can do any harm?

> 
> 
> > And another question is how I can then restore a single LV Xen created,
> > from the snapshot of the LV that spans the whole disk on the storage
> > server? In that case I can not just revert to the old disk before taking
> > the snapshot, because then all the LV's created by Xen will be set back
> > to that point, and not just the LV that went bad.
> 
> # Will only work if snapshot size is equal or greater than
> # the original volume
> 
> dd if=/dev/LVM/volume-snapshot of=/dev/LVM/volume
> 
> # or, if the allowed snapshot size is smaller, we don't want our
> # precious snapshot dropped
> 
> dd if=/dev/LVM/volume-snapshot of=/dev/LVM/new-volume
> dd if=/dev/LVM/new-volume of=/dev/LVM/volume

Oke, but how about this when using LVM as I just described a few lines
above here with the PV/LV in another PV/LV setup. How can I then restore
a snapshot of a virtual machines' LV from the snapshot of the LV on the
storage server?

> 
> 
> > Last question... Without creating a PV and a LV on the storage server
> > and just letting XenSource create what it needs to provision the virtual
> > machines, I can still see the PV and the LV's Xen created on the storage
> > server. Could I take the snapshots from there, although the PV and LV's
> > where not created here?
> 
> Only when you don't use them on the Xen server at the same time (no 
> iSCSI connection; LVM modules on the Xen server unloaded, LVM modules on 
> the storage server loaded).
> Which brings us to a point: it is better to manage LVMs on a storage 
> server than on a Xen host.
> 
> 

Re: [linux-lvm] LVM snapshots in a iSCSI and XenSource environment

[Tomasz Chmielewski]

S. J. van Harmelen schrieb:
> Thanks for your reaction!
> 
> On Tue, 2007-11-20 at 12:37 +0100, Tomasz Chmielewski wrote:
>> S. J. van Harmelen schrieb:
>>> Hi list,
>>>
>>> In advance my excusses for this radar long post (although it's easy
>>> readable ;), but I want to make sure that I understand it correctly so I
>>> don't end up making a very costly mistake.
>>>
>>> I have a storage server (Debian Etch) with mutlipath-tools running and
>>> on top of that I use IET iscsi-target software to export the multipathed
>>> device to a XenSource server.
>>>
>>> XenSource creates a PV on the entire exported disk, and then creates a
>>> few LV's when I create some virtual machines.
>> Which IMO is a pity, as logically, LVM exists and is usable on that 
>> given Xen server only. This means you can't really use multiple Xen 
>> servers, live migration etc.
> 
> Are you sure about that? Accoring to Xen lvm over iSCSI is considered as
> shared storage that can be used for live migration
> (http://docs.xensource.com/XenServer/4.0.1/installation/ch03s03.html):
> 
> <snip>
> 3.3.3. XenServer Hosts with shared iSCSI storage

This implies turning a Xen server into a NAS server, as I understand?
Yes, this should work - I have a custom Xen setup, and I don't connect a 
NAS and a Xen server on one host.
But according to the description, yes, it should work.


(...)

>>> Questions that I think of then are if it's not a problem that XenSource
>>> then creates a new PV and some LV's in je LV I created adn exported on
>>> the storage server. Is that a problem, or should this work fine?
>> Hmm? I don't think I understand what you mean.
> 
> I mean I will have 2 sets of PV's and LV's. One set on the storage
> server (that has one PV and one LV that both span the whole disk), and
> one set op the Xen server (the ones that Xen makes by itself when I add
> a new vitrual machine).
> 
> Now the one LV from the storage server that Xen sees is being exported
> true iSCSI, so it wouldn't know that it is infact a LV that it's talking
> to. As far as the Xen server conserns this is just a raw disk. So it
> will then just create the needed PV and LV's on it to provision virtual
> mahines.
> 
> The question was if this PV/LV in another PV/LV (on another physical
> machine) can do any harm?

Should work, as long as you don't use it on two different machines 
(i.e., target/initiator) as LVM at the same time.


>>> And another question is how I can then restore a single LV Xen created,
>>> from the snapshot of the LV that spans the whole disk on the storage
>>> server? In that case I can not just revert to the old disk before taking
>>> the snapshot, because then all the LV's created by Xen will be set back
>>> to that point, and not just the LV that went bad.
>> # Will only work if snapshot size is equal or greater than
>> # the original volume
>>
>> dd if=/dev/LVM/volume-snapshot of=/dev/LVM/volume
>>
>> # or, if the allowed snapshot size is smaller, we don't want our
>> # precious snapshot dropped
>>
>> dd if=/dev/LVM/volume-snapshot of=/dev/LVM/new-volume
>> dd if=/dev/LVM/new-volume of=/dev/LVM/volume
> 
> Oke, but how about this when using LVM as I just described a few lines
> above here with the PV/LV in another PV/LV setup. How can I then restore
> a snapshot of a virtual machines' LV from the snapshot of the LV on the
> storage server?

As a prerequisite, you need to "see" that LVM.
But could you be more specific on what you want (write a bit more 
details in points, write a small diagram etc.)?


-- 
Tomasz Chmielewski
http://wpkg.org

Re: [linux-lvm] LVM snapshots in a iSCSI and XenSource environment

[S. J. van Harmelen]

On Tue, 2007-11-20 at 14:06 +0100, Tomasz Chmielewski wrote:
> S. J. van Harmelen schrieb:
> > Thanks for your reaction!
> > 
> > On Tue, 2007-11-20 at 12:37 +0100, Tomasz Chmielewski wrote:
> >> S. J. van Harmelen schrieb:
> >>> Hi list,
> >>>
> >>> In advance my excusses for this radar long post (although it's easy
> >>> readable ;), but I want to make sure that I understand it correctly so I
> >>> don't end up making a very costly mistake.
> >>>
> >>> I have a storage server (Debian Etch) with mutlipath-tools running and
> >>> on top of that I use IET iscsi-target software to export the multipathed
> >>> device to a XenSource server.
> >>>
> >>> XenSource creates a PV on the entire exported disk, and then creates a
> >>> few LV's when I create some virtual machines.
> >> Which IMO is a pity, as logically, LVM exists and is usable on that 
> >> given Xen server only. This means you can't really use multiple Xen 
> >> servers, live migration etc.
> > 
> > Are you sure about that? Accoring to Xen lvm over iSCSI is considered as
> > shared storage that can be used for live migration
> > (http://docs.xensource.com/XenServer/4.0.1/installation/ch03s03.html):
> > 
> > <snip>
> > 3.3.3. XenServer Hosts with shared iSCSI storage
> 
> This implies turning a Xen server into a NAS server, as I understand?
> Yes, this should work - I have a custom Xen setup, and I don't connect a 
> NAS and a Xen server on one host.
> But according to the description, yes, it should work.
> 
> 
> (...)
> 
> >>> Questions that I think of then are if it's not a problem that XenSource
> >>> then creates a new PV and some LV's in je LV I created adn exported on
> >>> the storage server. Is that a problem, or should this work fine?
> >> Hmm? I don't think I understand what you mean.
> > 
> > I mean I will have 2 sets of PV's and LV's. One set on the storage
> > server (that has one PV and one LV that both span the whole disk), and
> > one set op the Xen server (the ones that Xen makes by itself when I add
> > a new vitrual machine).
> > 
> > Now the one LV from the storage server that Xen sees is being exported
> > true iSCSI, so it wouldn't know that it is infact a LV that it's talking
> > to. As far as the Xen server conserns this is just a raw disk. So it
> > will then just create the needed PV and LV's on it to provision virtual
> > mahines.
> > 
> > The question was if this PV/LV in another PV/LV (on another physical
> > machine) can do any harm?
> 
> Should work, as long as you don't use it on two different machines 
> (i.e., target/initiator) as LVM at the same time.
> 
> 
> >>> And another question is how I can then restore a single LV Xen created,
> >>> from the snapshot of the LV that spans the whole disk on the storage
> >>> server? In that case I can not just revert to the old disk before taking
> >>> the snapshot, because then all the LV's created by Xen will be set back
> >>> to that point, and not just the LV that went bad.
> >> # Will only work if snapshot size is equal or greater than
> >> # the original volume
> >>
> >> dd if=/dev/LVM/volume-snapshot of=/dev/LVM/volume
> >>
> >> # or, if the allowed snapshot size is smaller, we don't want our
> >> # precious snapshot dropped
> >>
> >> dd if=/dev/LVM/volume-snapshot of=/dev/LVM/new-volume
> >> dd if=/dev/LVM/new-volume of=/dev/LVM/volume
> > 
> > Oke, but how about this when using LVM as I just described a few lines
> > above here with the PV/LV in another PV/LV setup. How can I then restore
> > a snapshot of a virtual machines' LV from the snapshot of the LV on the
> > storage server?
> 
> As a prerequisite, you need to "see" that LVM.
> But could you be more specific on what you want (write a bit more 
> details in points, write a small diagram etc.)?

Oke... Let's say I have a storage server with one LUN connected to it.
The LUN becomes /dev/sdi on the storage server. I will then create a PV
on /dev/sdi spanning the whole disk. In the PV I will create one LV also
spanning the whole disk. Say we name this LV lv_storage. So the path to
this LV gets to be /dev/lvm/lv_storage.

In iSCSI I will then create a target which points
to /dev/lvm/lv_storage. The Xen server will use open-iscsi to connect to
that target (/dev/lvm/lvstorage) and on the Xen server it
becomes /dev/sdb. At this point Xen has also created a PV that spans the
whole disk (/dev/sdb)

Then when I create a virtual machine on the Xen server, it creates a LV
on /dev/sdb with the size of the virtual disk. Say this LV gets named
lv_virtualmachine1 and the next gets to be named lv_virtualmachine2 and
so on.

So far everything clear?

Now my thoughts where to take snapshots on the storage server. So that
would mean I would take a snapshot of lv_storage. This should not be a
problem. But what if virtual machine 2 crashes and has to be restored?

In that case I cannot restore lv_storage with a dd command, because then
all the data will be restored (lv_virtualmachine1 and
lv_virtualmachine2, because they are both on lv_storage). So is it
possible to "restore" only lv_virtualmachine2 when I took a snapshot of
lv_storage?

I can see the PV and LV's Xen created on lv_storage on the storage
server when I run pvs by the way...

Hope you understand what I mean now?

> 
> 

Re: [linux-lvm] LVM snapshots in a iSCSI and XenSource environment

[malahal]

S. J. van Harmelen [svh@dds.nl] wrote:
> 
> Now my thoughts where to take snapshots on the storage server. So that
> would mean I would take a snapshot of lv_storage. This should not be a
> problem. But what if virtual machine 2 crashes and has to be restored?
> 
> In that case I cannot restore lv_storage with a dd command, because then
> all the data will be restored (lv_virtualmachine1 and
> lv_virtualmachine2, because they are both on lv_storage). So is it
> possible to "restore" only lv_virtualmachine2 when I took a snapshot of
> lv_storage?

I see, it is possible but a bit cumbersome. You should be able to run
LVM on the storage server on the restored/old data (LVM on top of LVM??) and
get the data for the lv_virtualmachine2 from the storage server to
XenSource (ftp/tar whatever method). Then, restore that data (tar
extract) at XenSource.

If things are NOT changing at the XenSource LVM, you could just "dd" on
the crashed LV, but things may go really bad otherwise.

Any better method(s)???

> I can see the PV and LV's Xen created on lv_storage on the storage
> server when I run pvs by the way...

Yes, you will see them but you want the storage server's LVM to NOT use
them under normal conditions.

BTW, for backup, you may want snapshot-origin target rather than snapshot
target directly..

Re: [linux-lvm] LVM snapshots in a iSCSI and XenSource environment

[S. J. van Harmelen]

On Tue, 2007-11-20 at 09:29 -0800, malahal@us.ibm.com wrote:
> S. J. van Harmelen [svh@dds.nl] wrote:
> > 
> > Now my thoughts where to take snapshots on the storage server. So that
> > would mean I would take a snapshot of lv_storage. This should not be a
> > problem. But what if virtual machine 2 crashes and has to be restored?
> > 
> > In that case I cannot restore lv_storage with a dd command, because then
> > all the data will be restored (lv_virtualmachine1 and
> > lv_virtualmachine2, because they are both on lv_storage). So is it
> > possible to "restore" only lv_virtualmachine2 when I took a snapshot of
> > lv_storage?
> 
> I see, it is possible but a bit cumbersome. You should be able to run
> LVM on the storage server on the restored/old data (LVM on top of LVM??) and
> get the data for the lv_virtualmachine2 from the storage server to
> XenSource (ftp/tar whatever method). Then, restore that data (tar
> extract) at XenSource.
> 
> If things are NOT changing at the XenSource LVM, you could just "dd" on
> the crashed LV, but things may go really bad otherwise.
> 
> Any better method(s)???

Well of course this could also be done from the Xen server. Then there
is no need for LVM on the storage server and the lv_storage part can be
left out of this picture.

I could then create a second iSCSI target on a second LUN to export to
the Xen server so that Xen can take snapshots to that second exported
disk. If a problem does occur, then a restore is simple because then I
could do a dd from the snapshot lv (on the Xen server) to the other lv
(also on the Xen server).

But how about network traffic on this one? Everything I do will go
across iSCSI so it will go over the network.

And then I will manually have to greate lv-snapshots for each virtual
machine instead of just one big lv-snapshot as I could do when
lv_storage exists.

Any thoughts?

> 
> > I can see the PV and LV's Xen created on lv_storage on the storage
> > server when I run pvs by the way...
> 
> Yes, you will see them but you want the storage server's LVM to NOT use
> them under normal conditions.
> 
> BTW, for backup, you may want snapshot-origin target rather than snapshot
> target directly..

Re: [linux-lvm] LVM snapshots in a iSCSI and XenSource environment

[S. J. van Harmelen]

On Tue, 2007-11-20 at 09:29 -0800, malahal@us.ibm.com wrote:
> S. J. van Harmelen [svh@dds.nl] wrote:
> > 
> > Now my thoughts where to take snapshots on the storage server. So that
> > would mean I would take a snapshot of lv_storage. This should not be a
> > problem. But what if virtual machine 2 crashes and has to be restored?
> > 
> > In that case I cannot restore lv_storage with a dd command, because then
> > all the data will be restored (lv_virtualmachine1 and
> > lv_virtualmachine2, because they are both on lv_storage). So is it
> > possible to "restore" only lv_virtualmachine2 when I took a snapshot of
> > lv_storage?
> 
> I see, it is possible but a bit cumbersome. You should be able to run
> LVM on the storage server on the restored/old data (LVM on top of LVM??) and
> get the data for the lv_virtualmachine2 from the storage server to
> XenSource (ftp/tar whatever method). Then, restore that data (tar
> extract) at XenSource.
> 
> If things are NOT changing at the XenSource LVM, you could just "dd" on
> the crashed LV, but things may go really bad otherwise.
> 
> Any better method(s)???

Would it also work when I create lv_storage, take a snapshot of it and
then when I need to restore a virtual machine I tempararly export the
snapshot true iSCSI to the Xen server.

Then on the Xen server I should see the PV and LV's Xen created both on
the real iSCSI exported disk, as the exported snapshot from the storage
server. So then I should be able to do a dd from one lv to the other,
right?

> 
> > I can see the PV and LV's Xen created on lv_storage on the storage
> > server when I run pvs by the way...
> 
> Yes, you will see them but you want the storage server's LVM to NOT use
> them under normal conditions.
> 
> BTW, for backup, you may want snapshot-origin target rather than snapshot
> target directly..