Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementation ready for beta testing

Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementation ready for beta testing

[Joe Thornber]

On Thu, Jan 31, 2002 at 01:52:29PM -0600, Steve Pratt wrote:
> Joe Thornber wrote:
> >On Wed, Jan 30, 2002 at 10:03:40PM +0000, Jim McDonald wrote:
> >> Also, does/where does this fit in with EVMS?
> 
> >EVMS differs from us in that they seem to be trying to move the whole
> >application into the kernel,
> 
> No, not really.  We only put in the kernel the things that make sense to be
> in the kernel, discovery logic, ioctl support, I/O path.  All configuration
> is handled in user space.

There's still a *lot* of code in there; > 26,000 lines in fact.
Whereas device-mapper weighs in at ~2,600 lines.  This is just because
you've decided to take a different route from us, you may be proven to
be correct.

> > whereas we've taken the opposite route
> >and stripped down the kernel side to just provide services.
> 
> Then why does snapshot.c in device mapper have a read_metadata function
> which populates the exception table from on disk metadata?  Seems like you
> agree with us that having metadata knowledge in the kernel is a GOOD thing.

In the case of snapshots the exception data has to be written by the
kernel for performance reasons, as you know.  This is still a far cry
from understanding the LVM1 metadata format.

> Since device_mapper does not support in kernel discovery, and EVMS relies
> on this, it would be very difficult to change EVMS to use device_mapper.

So do the discovery on the EVMS side, and then pass the tables across
to device-mapper to activate the LV's.

> Why compete, come on over and help us :-)

I would like the two projects to help each other, but not to the point
where one group of people has to say 'you are completely right, we
will stop developing our project'.  It's unlikely that either of us is
100% correct; but I do think device-mapper splits off a nice chunk of
services that is useful to *all* people who want to do volume
management.  As such I see that as one area where we may eventually
work together.

Similarly I expect to be providing an *optional* kernel module for LVM
users who wish to do in kernel discovery of a root LV, so if the EVMS
team has managed to get a nice generic way of iterating block devices
etc.  into the kernel, we would be able to take advantage of that.
Are you trying to break out functionality so it benefits other Linux
projects ? or is EVMS just one monolithic application embedded in the
kernel ?

- Joe

Re: [Evms-devel] Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementation ready for beta testing

[Joe Thornber]

On Fri, Feb 01, 2002 at 04:55:18AM -0500, Arjan van de Ven wrote:
> There is one thing that might spoil the device-mapper "just simple stuff
> only" thing: moving active volumes around. Doing that in userspace reliably
> is impossible and basically needs to be done in kernelspace (it's an
> operation comparable with raid1 resync, a not even that hard in kernel
> space). However, that sort of automatically requires kernelspace to know
> about volumes, and from there it's a small step....

I think you're talking about pvmove and friends, in which case I hope
the description below helps.

- Joe



Let's say we have an LV, made up of three segments of different PV's,
I've also added in the device major:minor as this will be useful
later:

+-----------------------------+
|  PV1     |   PV2   |   PV3  | 254:3
+----------+---------+--------+


Now our hero decides to PV move PV2 to PV4:

1. Suspend our LV (254:3), this starts queueing all io, and flushes
   all pending io.  Once the suspend has completed we are free to change
   the mapping table.

2. Set up *another* (254:4) device with the mapping table of our LV.

3. Load a new mapping table into (254:3) that has identity targets for
   parts that aren't moving, and a mirror target for parts that are.

4. Unsuspend (254:3)

So now we have:
                           destination of copy
               +--------------------->--------------+
               |                                    |
+-----------------------------+               + -----------+
| Identity | mirror  | Ident. | 254:3         |    PV4     |
+----------+---------+--------+               +------------+
     |         |         |
     \/        \/        \/
+-----------------------------+
|  PV1     |   PV2   |   PV3  | 254:4
+----------+---------+--------+

Any writes to segment2 of the LV get intercepted by the mirror target
who checks that that chunk has been copied to the new destination, if
it hasn't it queues the initial copy and defers the current io until
it has finished.  Then the current io is written to *both* PV2 and the
PV4.

5. When the copying has completed 254:3 is suspended/pending flushed.

6. 254:4 is taken down

7. metadata is updated on disk

8. 254:3 has new mapping table loaded:

+-----------------------------+
|  PV1     |   PV4   |   PV3  | 254:3
+----------+---------+--------+

Re: [ANNOUNCE] LVM reimplementation ready for beta testing

[Joe Thornber]

On Fri, Feb 01, 2002 at 05:12:51AM -0500, Arjan van de Ven wrote:
> On Thu, Jan 31, 2002 at 01:09:13PM +0000, Joe Thornber wrote:
> > 
> > Now our hero decides to PV move PV2 to PV4:
> > 
> > 1. Suspend our LV (254:3), this starts queueing all io, and flushes
> >    all pending io. 
> 
> But "flushes all pending io" is *far* from trivial. there's no current
> kernel functionality for this, so you'll have to do "weird shit" that will
> break easy and often.

Here's the weird shit.  If you can see how to break it, I'd like to
know.

Whenever the dm driver maps a buffer_head, I increment a 'pending'
counter for that device, and hook the bh->b_end_io, bh->b_private
function so that this counter is decremented when the io completes.
This doesn't work with ext3 on 2.4 kernels since ext3 believes the
b_private pointer is for general filesystem use rather than just
b_end_io, however Stephen Tweedie and I have been discussing ways to
get round this.  On 2.5 this works fine since the bio->bi_private
isn't abused in this way.

> Also "suspending" is rather dangerous because it can deadlock the machine
> (think about the VM needing to write back dirty data on this LV in order to
>  make memory available for your move)...

You are correct, this is the main flaw IMO with the LVM1 version of
pvmove (which was userland with locking on a per extent basis).

However for LVM2 the device will only be suspended while a table
is loaded, *not* while the move takes place.  I will however allocate
the struct deferred_io objects from a mempool in 2.5.

- Joe

Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementation ready for beta testing

[Steve Pratt]

Joe Thornber wrote:
>On Wed, Jan 30, 2002 at 10:03:40PM +0000, Jim McDonald wrote:
>> Also, does/where does this fit in with EVMS?

>EVMS differs from us in that they seem to be trying to move the whole
>application into the kernel,

No, not really.  We only put in the kernel the things that make sense to be
in the kernel, discovery logic, ioctl support, I/O path.  All configuration
is handled in user space.

> whereas we've taken the opposite route
>and stripped down the kernel side to just provide services.

Then why does snapshot.c in device mapper have a read_metadata function
which populates the exception table from on disk metadata?  Seems like you
agree with us that having metadata knowledge in the kernel is a GOOD thing.

>This is fine, I think there's room for both projects.  But it is worth
>noting that EVMS could be changed to use device-mapper for it's low
>level functionality.  That way they could take advantage of the cool
>work we're doing with snapshots and pvmove, and we could take
>advantage of having more eyes on the core driver.

Since device_mapper does not support in kernel discovery, and EVMS relies
on this, it would be very difficult to change EVMS to use device_mapper.
Besides, EVMS already has all the capabilities provided by device mapper,
including a complete LVM1 compatibility package.

>LVM2 may not seem that exciting initially, since the first release is
>just concentrating on reproducing LVM1 functionality.  But a lot of
>the reason for this rewrite is to enable us to add in the new features
>that we want (such as a transaction based disk format).  It's on this
>new feature list that we'll be mainly competing with EVMS.

Why compete, come on over and help us :-)

Steve


EVMS Development - http://www.sf.net/projects/evms
Linux Technology Center - IBM Corporation
(512) 838-9763  EMAIL: SLPratt@US.IBM.COM

Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementationre ady for beta testing

[Jeff Layton]

(snip)

>
> >LVM2 may not seem that exciting initially, since the first release is
> >just concentrating on reproducing LVM1 functionality.  But a lot of
> >the reason for this rewrite is to enable us to add in the new features
> >that we want (such as a transaction based disk format).  It's on this
> >new feature list that we'll be mainly competing with EVMS.
>
> Why compete, come on over and help us :-)

You guy man!

I've worked with Kevin a few months ago on using EVMS
to recover a filesystem ontop of LVM when a disk went
south and we had no backups (I know, but I was told by the
people in charge that backups were not needed - that's why
I left). I really like EVMS and can't wait for it to be in 2.5 and
then 2.6 (just keep up the patches for 2.4 please :)

Jeff Layton


>
>
> Steve
>
> EVMS Development - http://www.sf.net/projects/evms
> Linux Technology Center - IBM Corporation
> (512) 838-9763  EMAIL: SLPratt@US.IBM.COM
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

Re: [Evms-devel] Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementation ready for beta testing

[Andrew Clausen]

On Thu, Jan 31, 2002 at 01:52:29PM -0600, Steve Pratt wrote:
> Joe Thornber wrote:
> >On Wed, Jan 30, 2002 at 10:03:40PM +0000, Jim McDonald wrote:
> >> Also, does/where does this fit in with EVMS?
> 
> >EVMS differs from us in that they seem to be trying to move the whole
> >application into the kernel,
> 
> No, not really.  We only put in the kernel the things that make sense to be
> in the kernel, discovery logic, ioctl support, I/O path.  All configuration
> is handled in user space.

The notion of "volume group" and "physical volume", etc. does
not exist in the device mapper.

It is really much lower-level.

As it is, it could be used for partitioning as well, with a single
LOC changed.  In fact, I imagine it's a 100 LOC or so in libparted to
get partprobe to use it.  Then we could retire all partition code
from the kernel :)

> > whereas we've taken the opposite route
> >and stripped down the kernel side to just provide services.
> 
> Then why does snapshot.c in device mapper have a read_metadata function
> which populates the exception table from on disk metadata?  Seems like you
> agree with us that having metadata knowledge in the kernel is a GOOD thing.

It's good to *support* reading metadata, but better to not require it.

> Since device_mapper does not support in kernel discovery, and EVMS relies
> on this, it would be very difficult to change EVMS to use device_mapper.

Why?  Just because device_mapper itself doesn't support discovery
doesn't mean you can't do the discovery for it, and setup the
devices in-kernel.

I think a bigger issue will be the difference in interface for
EVMS stuff, like all the plugin ioctls... it would be great if
you could:
* decouple the evms runtime (kernel code) from the evms
engine (userland).  Something like libparted's PedArchitecture
would be nice.
* get everyone in the linux community to agree on some standard
ioctl's that provide an interface for everything that evms
(and everyone else) needs.

How hard is this to do?

Andrew

Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementation ready for beta testing

[Steve Pratt]

Joe Thornber wrote:
>On Wed, Jan 30, 2002 at 10:03:40PM +0000, Jim McDonald wrote:
>> Also, does/where does this fit in with EVMS?

>EVMS differs from us in that they seem to be trying to move the whole
>application into the kernel,

No, not really.  We only put in the kernel the things that make sense to be
in the kernel, discovery logic, ioctl support, I/O path.  All configuration
is handled in user space.

> whereas we've taken the opposite route
>and stripped down the kernel side to just provide services.

Then why does snapshot.c in device mapper have a read_metadata function
which populates the exception table from on disk metadata?  Seems like you
agree with us that having metadata knowledge in the kernel is a GOOD thing.

>This is fine, I think there's room for both projects.  But it is worth
>noting that EVMS could be changed to use device-mapper for it's low
>level functionality.  That way they could take advantage of the cool
>work we're doing with snapshots and pvmove, and we could take
>advantage of having more eyes on the core driver.

Since device_mapper does not support in kernel discovery, and EVMS relies
on this, it would be very difficult to change EVMS to use device_mapper.
Besides, EVMS already has all the capabilities provided by device mapper,
including a complete LVM1 compatibility package.

>LVM2 may not seem that exciting initially, since the first release is
>just concentrating on reproducing LVM1 functionality.  But a lot of
>the reason for this rewrite is to enable us to add in the new features
>that we want (such as a transaction based disk format).  It's on this
>new feature list that we'll be mainly competing with EVMS.

Why compete, come on over and help us :-)

Steve


EVMS Development - http://www.sf.net/projects/evms
Linux Technology Center - IBM Corporation
(512) 838-9763  EMAIL: SLPratt@US.IBM.COM

Re: [Evms-devel] Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementation ready for beta testing

[Andrew Clausen]

On Thu, Jan 31, 2002 at 01:52:29PM -0600, Steve Pratt wrote:
> Joe Thornber wrote:
> >On Wed, Jan 30, 2002 at 10:03:40PM +0000, Jim McDonald wrote:
> >> Also, does/where does this fit in with EVMS?
> 
> >EVMS differs from us in that they seem to be trying to move the whole
> >application into the kernel,
> 
> No, not really.  We only put in the kernel the things that make sense to be
> in the kernel, discovery logic, ioctl support, I/O path.  All configuration
> is handled in user space.

The notion of "volume group" and "physical volume", etc. does
not exist in the device mapper.

It is really much lower-level.

As it is, it could be used for partitioning as well, with a single
LOC changed.  In fact, I imagine it's a 100 LOC or so in libparted to
get partprobe to use it.  Then we could retire all partition code
from the kernel :)

> > whereas we've taken the opposite route
> >and stripped down the kernel side to just provide services.
> 
> Then why does snapshot.c in device mapper have a read_metadata function
> which populates the exception table from on disk metadata?  Seems like you
> agree with us that having metadata knowledge in the kernel is a GOOD thing.

It's good to *support* reading metadata, but better to not require it.

> Since device_mapper does not support in kernel discovery, and EVMS relies
> on this, it would be very difficult to change EVMS to use device_mapper.

Why?  Just because device_mapper itself doesn't support discovery
doesn't mean you can't do the discovery for it, and setup the
devices in-kernel.

I think a bigger issue will be the difference in interface for
EVMS stuff, like all the plugin ioctls... it would be great if
you could:
* decouple the evms runtime (kernel code) from the evms
engine (userland).  Something like libparted's PedArchitecture
would be nice.
* get everyone in the linux community to agree on some standard
ioctl's that provide an interface for everything that evms
(and everyone else) needs.

How hard is this to do?

Andrew

Re: [evms-devel] [linux-lvm] [ANNOUNCE] LVM reimplementation ready for beta testing

[Joe Thornber]

On Fri, Feb 01, 2002 at 03:59:06PM -0600, Kevin Corry wrote:
> I have been thinking about this today and looking over some of the 
> device-mapper interfaces. I will agree that, in concept, EVMS could be 
> modified to use device-mapper for I/O remapping. However, as things stand 
> today, I don't think the transition would be easy.
> 
> As I'm trying to envision it, the EVMS runtime would become a "volume 
> recognition" framework (see tanget below). Every current EVMS plugin would 
> then probe all available devices and communicate the necessary volume 
> remapping info to device-mapper through the ioctl interface. (An in-kernel 
> API set might be nice to avoid the overhead of the ioctl path).

The in kernel API already exists in drivers/md/dm.h, see dm-ioctl.c
for example code that uses this.


> A new device would then be created for every node that every plugin
> recognizes. This brings up my first objection. With this approach,
> there would be an exposed device for every node in a volume stack,
> whereas the current EVMS design only exposes nodes for the final
> volumes.

True, all the devices are exposed in /dev/device-mapper.  What we do
with LVM2 is stick with the /dev/<vg_name>/<lv_name> scheme, where the
lv is a symlink to the relevant node in the device-mapper dir.  That
way people only see the top level devices, unless they peek under the
hood by looking at /dev/device-mapper.

> Next, from what I've seen, device-mapper provides static remappings
> from one device to another. It seems to be a good approach for
> setting up things like disk partitions and LVM linear LVs. There is
> also striping support in device-mapper, but I'm assuming it uses one
> notion of striping. For instance, RAID-0 striping in MD is handled
> differently than striped LVs in LVM, and I think AIX striping is
> also different. I'm not sure if one stiping module could be generic
> enough to handle all of these cases. But, maybe it can. I'll have to
> think more about that one.

The thing that you've missed is that ranges of sectors are mapped by
dm onto 'targets', these targets are instances of a 'target_type'
virtual class.  You can define your own target types in a seperate
module and register them using the

int dm_register_target(struct target_type *t);

function (see <linux/device-mapper.h>).  We originally intended all
the standard target types to be defined in seperate modules, but that
seemed silly considering the tiny size of them (eg, < 200 lines for striping).

The targets type that we will be using in LVM2 are:

io-err - errors every io
linear
striped
snapshot, and snapshot origin
mirror (used to provide pvmove support as well as mirroring).

(both the snapshot targets and mirroring target will use kcopyd for
efficient copying using kiobufs.)

So if you invent a new form of striping, just implement a new target type.

> How about mirroring? Does the linear module in device-mapper allow
> for 1-to-n mappings?  This would be similar to the way AIX does
> mirroring, where each LP of an LV can map to up to three PPs on
> different PVs.

No, there's a seperate mirror target that Patrick Caulfields working
on at the moment.

> How would device-mapper handle a remapping that is dynamic at runtime?

Suspending a device flushes all pending io to the device, ie. all io
that has been through the driver but not yet completed.  Once you have
suspended you are free to loaded a new mapping table into the device
(this is essential for implementing snapshots and pvmove).  When the
new mapping is in place you just have to resume the device and away
you go.

- Joe

Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementationre ady for beta testing

[Heinz J . Mauelshagen]

On Thu, Jan 31, 2002 at 03:05:56PM -0500, Jeff Layton wrote:
> (snip)
> 
> >
> > >LVM2 may not seem that exciting initially, since the first release is
> > >just concentrating on reproducing LVM1 functionality.  But a lot of
> > >the reason for this rewrite is to enable us to add in the new features
> > >that we want (such as a transaction based disk format).  It's on this
> > >new feature list that we'll be mainly competing with EVMS.
> >
> > Why compete, come on over and help us :-)
> 
> You guy man!
> 
> I've worked with Kevin a few months ago on using EVMS
> to recover a filesystem ontop of LVM when a disk went
> south and we had no backups

It is quite intersting how many people don't take backups even though
almost everybody claims to know that backups are crucial and some of them
know that they are still, even though they might have RAID in place.

Well I guess it, but you didn't mention if you had success to recover.

Was it quorum (we are implementing it with LVM2 BTW) which helped and have
you been lucky that at least a part of your LVs where sitting on other than the
gone disk? Otherwise you probably had your VG back but no usable data.

> (I know, but I was told by the
> people in charge that backups were not needed - that's why
> I left).

Probably a good choice to leave if they don't understand such basics
and you couldn't convince them.


> I really like EVMS and can't wait for it to be in 2.5 and
> then 2.6 (just keep up the patches for 2.4 please :)

EVMS hallelujah on linux-lvm? ;-)

> 
> Jeff Layton
> 
> 
> >
> >
> > Steve
> >
> > EVMS Development - http://www.sf.net/projects/evms
> > Linux Technology Center - IBM Corporation
> > (512) 838-9763  EMAIL: SLPratt@US.IBM.COM
> >
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
> 
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementation ready for beta testing

[Joe Thornber]

On Thu, Jan 31, 2002 at 01:52:29PM -0600, Steve Pratt wrote:
> Joe Thornber wrote:
> >On Wed, Jan 30, 2002 at 10:03:40PM +0000, Jim McDonald wrote:
> >> Also, does/where does this fit in with EVMS?
> 
> >EVMS differs from us in that they seem to be trying to move the whole
> >application into the kernel,
> 
> No, not really.  We only put in the kernel the things that make sense to be
> in the kernel, discovery logic, ioctl support, I/O path.  All configuration
> is handled in user space.

There's still a *lot* of code in there; > 26,000 lines in fact.
Whereas device-mapper weighs in at ~2,600 lines.  This is just because
you've decided to take a different route from us, you may be proven to
be correct.

> > whereas we've taken the opposite route
> >and stripped down the kernel side to just provide services.
> 
> Then why does snapshot.c in device mapper have a read_metadata function
> which populates the exception table from on disk metadata?  Seems like you
> agree with us that having metadata knowledge in the kernel is a GOOD thing.

In the case of snapshots the exception data has to be written by the
kernel for performance reasons, as you know.  This is still a far cry
from understanding the LVM1 metadata format.

> Since device_mapper does not support in kernel discovery, and EVMS relies
> on this, it would be very difficult to change EVMS to use device_mapper.

So do the discovery on the EVMS side, and then pass the tables across
to device-mapper to activate the LV's.

> Why compete, come on over and help us :-)

I would like the two projects to help each other, but not to the point
where one group of people has to say 'you are completely right, we
will stop developing our project'.  It's unlikely that either of us is
100% correct; but I do think device-mapper splits off a nice chunk of
services that is useful to *all* people who want to do volume
management.  As such I see that as one area where we may eventually
work together.

Similarly I expect to be providing an *optional* kernel module for LVM
users who wish to do in kernel discovery of a root LV, so if the EVMS
team has managed to get a nice generic way of iterating block devices
etc.  into the kernel, we would be able to take advantage of that.
Are you trying to break out functionality so it benefits other Linux
projects ? or is EVMS just one monolithic application embedded in the
kernel ?

- Joe

Re: [Evms-devel] Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementation ready for beta testing

[Arjan van de Ven]

On Thu, Jan 31, 2002 at 12:52:11PM +0000, Joe Thornber wrote:
> On Thu, Jan 31, 2002 at 01:52:29PM -0600, Steve Pratt wrote:
> > Joe Thornber wrote:
> > >On Wed, Jan 30, 2002 at 10:03:40PM +0000, Jim McDonald wrote:
> > >> Also, does/where does this fit in with EVMS?
> > 
> > >EVMS differs from us in that they seem to be trying to move the whole
> > >application into the kernel,
> > 
> > No, not really.  We only put in the kernel the things that make sense to be
> > in the kernel, discovery logic, ioctl support, I/O path.  All configuration
> > is handled in user space.
> 
> There's still a *lot* of code in there; > 26,000 lines in fact.
> Whereas device-mapper weighs in at ~2,600 lines.  This is just because
> you've decided to take a different route from us, you may be proven to
> be correct.

There is one thing that might spoil the device-mapper "just simple stuff
only" thing: moving active volumes around. Doing that in userspace reliably
is impossible and basically needs to be done in kernelspace (it's an
operation comparable with raid1 resync, a not even that hard in kernel
space). However, that sort of automatically requires kernelspace to know
about volumes, and from there it's a small step....

Greetings,
   Arjan van de Ven

Re: [Evms-devel] Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementation ready for beta testing

[Joe Thornber]

On Fri, Feb 01, 2002 at 04:55:18AM -0500, Arjan van de Ven wrote:
> There is one thing that might spoil the device-mapper "just simple stuff
> only" thing: moving active volumes around. Doing that in userspace reliably
> is impossible and basically needs to be done in kernelspace (it's an
> operation comparable with raid1 resync, a not even that hard in kernel
> space). However, that sort of automatically requires kernelspace to know
> about volumes, and from there it's a small step....

I think you're talking about pvmove and friends, in which case I hope
the description below helps.

- Joe



Let's say we have an LV, made up of three segments of different PV's,
I've also added in the device major:minor as this will be useful
later:

+-----------------------------+
|  PV1     |   PV2   |   PV3  | 254:3
+----------+---------+--------+


Now our hero decides to PV move PV2 to PV4:

1. Suspend our LV (254:3), this starts queueing all io, and flushes
   all pending io.  Once the suspend has completed we are free to change
   the mapping table.

2. Set up *another* (254:4) device with the mapping table of our LV.

3. Load a new mapping table into (254:3) that has identity targets for
   parts that aren't moving, and a mirror target for parts that are.

4. Unsuspend (254:3)

So now we have:
                           destination of copy
               +--------------------->--------------+
               |                                    |
+-----------------------------+               + -----------+
| Identity | mirror  | Ident. | 254:3         |    PV4     |
+----------+---------+--------+               +------------+
     |         |         |
     \/        \/        \/
+-----------------------------+
|  PV1     |   PV2   |   PV3  | 254:4
+----------+---------+--------+

Any writes to segment2 of the LV get intercepted by the mirror target
who checks that that chunk has been copied to the new destination, if
it hasn't it queues the initial copy and defers the current io until
it has finished.  Then the current io is written to *both* PV2 and the
PV4.

5. When the copying has completed 254:3 is suspended/pending flushed.

6. 254:4 is taken down

7. metadata is updated on disk

8. 254:3 has new mapping table loaded:

+-----------------------------+
|  PV1     |   PV4   |   PV3  | 254:3
+----------+---------+--------+

Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementationre ady for beta testing

[STEFANG(GMX)]

Hallo list,

on Fre, 01 Feb 2002, Heinz J . Mauelshagen (mauelshagen@sistina.com) wrote:

> On Thu, Jan 31, 2002 at 03:05:56PM -0500, Jeff Layton wrote:
[...]
> > I've worked with Kevin a few months ago on using EVMS
> > to recover a filesystem ontop of LVM when a disk went
> > south and we had no backups
> 
> It is quite intersting how many people don't take backups even though
> almost everybody claims to know that backups are crucial and some of them
> know that they are still, even though they might have RAID in place.

You cannot replace Backups with a RAID system...a RAID system is
just fine to help you in cas of a hardware failure, but if a disk
driver/user/hacker goes wild you are in big trouble 'cause the data
is gone on all disks.  I had the very same discussion in many
companies; the MBA guys try to tell you there is no need for a
backup (Backup is essentially an unneeded "loss center" for them)
since they have installed a RAID system... ;-)

CU,
   Stefan
-- 
Stefan Giessler                                       e-mail:
sgiessler@gmx.net

artificial intelligence is no match for natural stupidity

[linux-lvm] Re: [ANNOUNCE] LVM reimplementation ready for beta testing

[Arjan van de Ven]

On Thu, Jan 31, 2002 at 01:09:13PM +0000, Joe Thornber wrote:
> 
> Now our hero decides to PV move PV2 to PV4:
> 
> 1. Suspend our LV (254:3), this starts queueing all io, and flushes
>    all pending io. 

But "flushes all pending io" is *far* from trivial. there's no current
kernel functionality for this, so you'll have to do "weird shit" that will
break easy and often.

Also "suspending" is rather dangerous because it can deadlock the machine
(think about the VM needing to write back dirty data on this LV in order to
 make memory available for your move)...

Greetings,
  Arjan van de Ven

Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementationre ady for beta testing

[Heinz J . Mauelshagen]

On Fri, Feb 01, 2002 at 11:02:18AM +0100, STEFANG(GMX) wrote:
> Hallo list,
> 
> on Fre, 01 Feb 2002, Heinz J . Mauelshagen (mauelshagen@sistina.com) wrote:
> 
> > On Thu, Jan 31, 2002 at 03:05:56PM -0500, Jeff Layton wrote:
> [...]
> > > I've worked with Kevin a few months ago on using EVMS
> > > to recover a filesystem ontop of LVM when a disk went
> > > south and we had no backups
> > 
> > It is quite intersting how many people don't take backups even though
> > almost everybody claims to know that backups are crucial and some of them
> > know that they are still, even though they might have RAID in place.
> 
> You cannot replace Backups with a RAID system...

Which is exactly my point above ;-)

RAID DOES NOT REPLACE BACKUPS!

> a RAID system is
> just fine to help you in cas of a hardware failure, but if a disk
> driver/user/hacker goes wild you are in big trouble 'cause the data
> is gone on all disks.  I had the very same discussion in many
> companies; the MBA guys try to tell you there is no need for a
> backup (Backup is essentially an unneeded "loss center" for them)
> since they have installed a RAID system... ;-)
> 
> CU,
>    Stefan
> -- 
> Stefan Giessler                                       e-mail:
> sgiessler@gmx.net
> 
> artificial intelligence is no match for natural stupidity
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementationre ady for beta testing

[STEFANG(GMX)]

Hallo list,

on Fre, 01 Feb 2002, Heinz J . Mauelshagen (mauelshagen@sistina.com) wrote:

> > > It is quite intersting how many people don't take backups even though
> > > almost everybody claims to know that backups are crucial and some of
them
> > > know that they are still, even though they might have RAID in place.
> > 
> > You cannot replace Backups with a RAID system...
> 
> Which is exactly my point above ;-)

So I got you wrong, sorry ;-)

> RAID DOES NOT REPLACE BACKUPS!

Full ACK.

CU,
   Stefan
-- 
Stefan Giessler                                       e-mail:
sgiessler@gmx.net

LET Jesus be YOUR anchor!

So when Satan rocks your boat, THROW Jesus overboard!

[linux-lvm] Re: [ANNOUNCE] LVM reimplementation ready for beta testing

[Joe Thornber]

On Fri, Feb 01, 2002 at 05:12:51AM -0500, Arjan van de Ven wrote:
> On Thu, Jan 31, 2002 at 01:09:13PM +0000, Joe Thornber wrote:
> > 
> > Now our hero decides to PV move PV2 to PV4:
> > 
> > 1. Suspend our LV (254:3), this starts queueing all io, and flushes
> >    all pending io. 
> 
> But "flushes all pending io" is *far* from trivial. there's no current
> kernel functionality for this, so you'll have to do "weird shit" that will
> break easy and often.

Here's the weird shit.  If you can see how to break it, I'd like to
know.

Whenever the dm driver maps a buffer_head, I increment a 'pending'
counter for that device, and hook the bh->b_end_io, bh->b_private
function so that this counter is decremented when the io completes.
This doesn't work with ext3 on 2.4 kernels since ext3 believes the
b_private pointer is for general filesystem use rather than just
b_end_io, however Stephen Tweedie and I have been discussing ways to
get round this.  On 2.5 this works fine since the bio->bi_private
isn't abused in this way.

> Also "suspending" is rather dangerous because it can deadlock the machine
> (think about the VM needing to write back dirty data on this LV in order to
>  make memory available for your move)...

You are correct, this is the main flaw IMO with the LVM1 version of
pvmove (which was userland with locking on a per extent basis).

However for LVM2 the device will only be suspended while a table
is loaded, *not* while the move takes place.  I will however allocate
the struct deferred_io objects from a mempool in 2.5.

- Joe

[linux-lvm] Re: [Evms-devel] Re: [ANNOUNCE] LVM reimplementation ready for beta testing

[Stephen C. Tweedie]

Hi,

On Fri, Feb 01, 2002 at 05:12:51AM -0500, Arjan van de Ven wrote:
> On Thu, Jan 31, 2002 at 01:09:13PM +0000, Joe Thornber wrote:
> > 
> > Now our hero decides to PV move PV2 to PV4:
> > 
> > 1. Suspend our LV (254:3), this starts queueing all io, and flushes
> >    all pending io. 
> 
> But "flushes all pending io" is *far* from trivial. there's no current
> kernel functionality for this, so you'll have to do "weird shit" that will
> break easy and often.

I've been all through this with Joe.  He *does* track pending IO in
device_mapper, and he's got a layered device_mirror driver which can
be overlayed on top of the segments of the device that you want to
copy.  His design looks solid and the necessary infrastructure for
getting the locking right is all there.

> Also "suspending" is rather dangerous because it can deadlock the machine
> (think about the VM needing to write back dirty data on this LV in order to
>  make memory available for your move)...

There's a copy thread which preallocates a fully-populated kiobuf for
the data.  There's no VM pressure, and since it uses unbuffered IO,
there are no cache coherency problems like current LVM has.

Arjan, I *told* you they have thought this stuff through.  :-)

Cheers,
 Stephen

[linux-lvm] Re: [ANNOUNCE] LVM reimplementation ready for beta testing

[Alan Cox]

> But "flushes all pending io" is *far* from trivial. there's no current
> kernel functionality for this, so you'll have to do "weird shit" that will
> break easy and often.
> 
> Also "suspending" is rather dangerous because it can deadlock the machine
> (think about the VM needing to write back dirty data on this LV in order to
>  make memory available for your move)...

I don't think you need to suspend I/O except momentarily. I don't use LVM and
while I can't resize volumes I migrate them like this

	mdhotadd /dev/md1 /dev/newvolume

[wait for it to resync]

	mdhotremove /dev/md1 /dev/oldvolume

the situation here seems analogous. You never need to suspend I/O to the
volume until you actually kill it, by which time you can just skip the write
to the dead volume.

(The above procedure with ext3 does btw make a great backup system 8))

Alan

[linux-lvm] Re: [Evms-devel] Re: [ANNOUNCE] LVM reimplementation ready for beta testing

[Stephen C. Tweedie]

Hi,

On Fri, Feb 01, 2002 at 02:44:24PM +0000, Alan Cox wrote:
> > But "flushes all pending io" is *far* from trivial. there's no current
> > kernel functionality for this, so you'll have to do "weird shit" that will
> > break easy and often.
> > 
> > Also "suspending" is rather dangerous because it can deadlock the machine
> > (think about the VM needing to write back dirty data on this LV in order to
> >  make memory available for your move)...
> 
> I don't think you need to suspend I/O except momentarily. I don't use LVM and
> while I can't resize volumes I migrate them like this

LVM1 has some problems here.  First, when it needs to flush IO as
part of its locking it does so with fsync_dev, which is not a valid
way of flushing certain types IO.  Second, its copy is done in user
space, so there is no cache coherence with the logical device contents
and there is enough VM pressure to give a good chance of deadlocking.

However, it _does_ do its locking at a finer granularity than the
whole disk (it locks an extent --- 4MB by default --- at a time), so
even with LVM1 it is possible to do the move on a live volume without
locking up all IO for the duration of the entire copy.

LVM2's device-mirror code is much closer to the raid1 mechanism in
design, so it doesn't even have to lock down an extent during the
copy.

> the situation here seems analogous. You never need to suspend I/O to the
> volume until you actually kill it, by which time you can just skip the write
> to the dead volume.

Right.  LVM1 doesn't actually suspend IO to the volume, just to an
extent.  What it does volume-wide is to flush IO, which is different.

The problem is that when we come to copy a chunk of the volume,
however large that chunk is, we need to make sure both that no new IOs
arrive on it, AND that we have waited for all outstanding IOs against
that chunk.  It's the latter part which is the problem.  It is 
expensive to keep track of all outstanding IOs on a per-stripe basis,
so when we place a lock on a stripe and come to wait for
already-submitted IOs to complete, it is much easier just to do that
flush volume-wide.  It's not a complete lock on the whole volume, just
a temporary mutex to ensure that there are no IOs left outstanding on
the stripe we're locking.

Cheers,
 Stephen

Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementationre ady for beta testing

[Jeff Layton]

"Heinz J . Mauelshagen" wrote:

> On Thu, Jan 31, 2002 at 03:05:56PM -0500, Jeff Layton wrote:
> > (snip)
> >
> > >
> > > >LVM2 may not seem that exciting initially, since the first release is
> > > >just concentrating on reproducing LVM1 functionality.  But a lot of
> > > >the reason for this rewrite is to enable us to add in the new features
> > > >that we want (such as a transaction based disk format).  It's on this
> > > >new feature list that we'll be mainly competing with EVMS.
> > >
> > > Why compete, come on over and help us :-)
> >
> > You guy man!
> >
> > I've worked with Kevin a few months ago on using EVMS
> > to recover a filesystem ontop of LVM when a disk went
> > south and we had no backups
>
> It is quite intersting how many people don't take backups even though
> almost everybody claims to know that backups are crucial and some of them
> know that they are still, even though they might have RAID in place.

I was ordered by my management NOT to make backups or
I would be fired (exact words). So, I didn't. I told everyone
concerned that we HAD to have backups and follow corporate
policy. Everyone said thank you and go away. So I did.


>
>
> Well I guess it, but you didn't mention if you had success to recover.
>
> Was it quorum (we are implementing it with LVM2 BTW) which helped and have
> you been lucky that at least a part of your LVs where sitting on other than the
> gone disk? Otherwise you probably had your VG back but no usable data.

Actually, we had partial success and got back most data. Then
on a whim we tried the bad drive again, and BINGO it came back
to life! I tried every trick I know to get that drive back and spoke
with the vendor that agreed I did everything they knew how to
do. Then another admin walked in, popped it back in the case and
and it came back. I had the biggest egg on my face! Oh well, at least
we got the data back.


>
>
> > (I know, but I was told by the
> > people in charge that backups were not needed - that's why
> > I left).
>
> Probably a good choice to leave if they don't understand such basics
> and you couldn't convince them.

<vent>

My manager at that time couldn't spell UNIX, doesn't have a college
degree of any type and has never taken a single college course. My lead
admin can only spell "Sun" even though he claims to have a Linux box at
home and is pretty much hated by all UNIX/Linux admins (as well as
most NT admins as well). He is the king of micro managers (and I have
seen plenty of these in the aerospace industry).

Thanks for the venting time and space. I apologize.

</vent>


>
>
> > I really like EVMS and can't wait for it to be in 2.5 and
> > then 2.6 (just keep up the patches for 2.4 please :)
>
> EVMS hallelujah on linux-lvm? ;-)

I like EVMS and it's concepts. However, I'm currently using LVM
(I learned LVM on HPUX, so it was trivial to move to LVM on
Linux!). So, I guess you could say that I started with LVM - like it,
use it, depend on it; moving on to EVMS after some experimentation
and testing. I hope the two  merge at some point to avoid
proprietary-isms that seem to be coming.

Plus, I'm interested in the Open Cluster Framework
developments with EVMS. I don't know if these kinds of
things are happening with LVM or not.

Jeff


>
>
> >
> > Jeff Layton
> >
> >
> > >
> > >
> > > Steve
> > >
> > > EVMS Development - http://www.sf.net/projects/evms
> > > Linux Technology Center - IBM Corporation
> > > (512) 838-9763  EMAIL: SLPratt@US.IBM.COM
> > >
> > > _______________________________________________
> > > linux-lvm mailing list
> > > linux-lvm@sistina.com
> > > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > > read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
> >
> >
> >
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
>
> --
>
> Regards,
> Heinz    -- The LVM Guy --
>
> *** Software bugs are stupid.
>     Nevertheless it needs not so stupid people to solve them ***
>
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>
> Heinz Mauelshagen                                 Sistina Software Inc.
> Senior Consultant/Developer                       Am Sonnenhang 11
>                                                   56242 Marienrachdorf
>                                                   Germany
> Mauelshagen@Sistina.com                           +49 2626 141200
>                                                        FAX 924446
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

Re: [Evms-devel] Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementation ready for beta testing

[Arjan van de Ven]

On Thu, Jan 31, 2002 at 12:52:11PM +0000, Joe Thornber wrote:
> On Thu, Jan 31, 2002 at 01:52:29PM -0600, Steve Pratt wrote:
> > Joe Thornber wrote:
> > >On Wed, Jan 30, 2002 at 10:03:40PM +0000, Jim McDonald wrote:
> > >> Also, does/where does this fit in with EVMS?
> > 
> > >EVMS differs from us in that they seem to be trying to move the whole
> > >application into the kernel,
> > 
> > No, not really.  We only put in the kernel the things that make sense to be
> > in the kernel, discovery logic, ioctl support, I/O path.  All configuration
> > is handled in user space.
> 
> There's still a *lot* of code in there; > 26,000 lines in fact.
> Whereas device-mapper weighs in at ~2,600 lines.  This is just because
> you've decided to take a different route from us, you may be proven to
> be correct.

There is one thing that might spoil the device-mapper "just simple stuff
only" thing: moving active volumes around. Doing that in userspace reliably
is impossible and basically needs to be done in kernelspace (it's an
operation comparable with raid1 resync, a not even that hard in kernel
space). However, that sort of automatically requires kernelspace to know
about volumes, and from there it's a small step....

Greetings,
   Arjan van de Ven

Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementationre ady for beta testing

[Steven Dake]

[-- Attachment #1: Type: text/plain, Size: 870 bytes --]



Heinz J . Mauelshagen wrote:

>On Fri, Feb 01, 2002 at 11:02:18AM +0100, STEFANG(GMX) wrote:
>
>>Hallo list,
>>
>>on Fre, 01 Feb 2002, Heinz J . Mauelshagen (mauelshagen@sistina.com) wrote:
>>
>>>On Thu, Jan 31, 2002 at 03:05:56PM -0500, Jeff Layton wrote:
>>>
>>[...]
>>
>>>>I've worked with Kevin a few months ago on using EVMS
>>>>to recover a filesystem ontop of LVM when a disk went
>>>>south and we had no backups
>>>>
>>>It is quite intersting how many people don't take backups even though
>>>almost everybody claims to know that backups are crucial and some of them
>>>know that they are still, even though they might have RAID in place.
>>>
>>You cannot replace Backups with a RAID system...
>>
>
>Which is exactly my point above ;-)
>
>RAID DOES NOT REPLACE BACKUPS!
>
If you have ever developed RAID solutions, you would know how right you 
are :)
-steve


[-- Attachment #2: Type: text/html, Size: 1648 bytes --]

Re: [ANNOUNCE] LVM reimplementation ready for beta testing

[Arjan van de Ven]

On Thu, Jan 31, 2002 at 01:09:13PM +0000, Joe Thornber wrote:
> 
> Now our hero decides to PV move PV2 to PV4:
> 
> 1. Suspend our LV (254:3), this starts queueing all io, and flushes
>    all pending io. 

But "flushes all pending io" is *far* from trivial. there's no current
kernel functionality for this, so you'll have to do "weird shit" that will
break easy and often.

Also "suspending" is rather dangerous because it can deadlock the machine
(think about the VM needing to write back dirty data on this LV in order to
 make memory available for your move)...

Greetings,
  Arjan van de Ven

Re: [Evms-devel] Re: [ANNOUNCE] LVM reimplementation ready for beta testing

[Stephen C. Tweedie]

Hi,

On Fri, Feb 01, 2002 at 05:12:51AM -0500, Arjan van de Ven wrote:
> On Thu, Jan 31, 2002 at 01:09:13PM +0000, Joe Thornber wrote:
> > 
> > Now our hero decides to PV move PV2 to PV4:
> > 
> > 1. Suspend our LV (254:3), this starts queueing all io, and flushes
> >    all pending io. 
> 
> But "flushes all pending io" is *far* from trivial. there's no current
> kernel functionality for this, so you'll have to do "weird shit" that will
> break easy and often.

I've been all through this with Joe.  He *does* track pending IO in
device_mapper, and he's got a layered device_mirror driver which can
be overlayed on top of the segments of the device that you want to
copy.  His design looks solid and the necessary infrastructure for
getting the locking right is all there.

> Also "suspending" is rather dangerous because it can deadlock the machine
> (think about the VM needing to write back dirty data on this LV in order to
>  make memory available for your move)...

There's a copy thread which preallocates a fully-populated kiobuf for
the data.  There's no VM pressure, and since it uses unbuffered IO,
there are no cache coherency problems like current LVM has.

Arjan, I *told* you they have thought this stuff through.  :-)

Cheers,
 Stephen

Re: [ANNOUNCE] LVM reimplementation ready for beta testing

[Alan Cox]

> But "flushes all pending io" is *far* from trivial. there's no current
> kernel functionality for this, so you'll have to do "weird shit" that will
> break easy and often.
> 
> Also "suspending" is rather dangerous because it can deadlock the machine
> (think about the VM needing to write back dirty data on this LV in order to
>  make memory available for your move)...

I don't think you need to suspend I/O except momentarily. I don't use LVM and
while I can't resize volumes I migrate them like this

	mdhotadd /dev/md1 /dev/newvolume

[wait for it to resync]

	mdhotremove /dev/md1 /dev/oldvolume

the situation here seems analogous. You never need to suspend I/O to the
volume until you actually kill it, by which time you can just skip the write
to the dead volume.

(The above procedure with ext3 does btw make a great backup system 8))

Alan

Re: [Evms-devel] Re: [ANNOUNCE] LVM reimplementation ready for beta testing

[Stephen C. Tweedie]

Hi,

On Fri, Feb 01, 2002 at 02:44:24PM +0000, Alan Cox wrote:
> > But "flushes all pending io" is *far* from trivial. there's no current
> > kernel functionality for this, so you'll have to do "weird shit" that will
> > break easy and often.
> > 
> > Also "suspending" is rather dangerous because it can deadlock the machine
> > (think about the VM needing to write back dirty data on this LV in order to
> >  make memory available for your move)...
> 
> I don't think you need to suspend I/O except momentarily. I don't use LVM and
> while I can't resize volumes I migrate them like this

LVM1 has some problems here.  First, when it needs to flush IO as
part of its locking it does so with fsync_dev, which is not a valid
way of flushing certain types IO.  Second, its copy is done in user
space, so there is no cache coherence with the logical device contents
and there is enough VM pressure to give a good chance of deadlocking.

However, it _does_ do its locking at a finer granularity than the
whole disk (it locks an extent --- 4MB by default --- at a time), so
even with LVM1 it is possible to do the move on a live volume without
locking up all IO for the duration of the entire copy.

LVM2's device-mirror code is much closer to the raid1 mechanism in
design, so it doesn't even have to lock down an extent during the
copy.

> the situation here seems analogous. You never need to suspend I/O to the
> volume until you actually kill it, by which time you can just skip the write
> to the dead volume.

Right.  LVM1 doesn't actually suspend IO to the volume, just to an
extent.  What it does volume-wide is to flush IO, which is different.

The problem is that when we come to copy a chunk of the volume,
however large that chunk is, we need to make sure both that no new IOs
arrive on it, AND that we have waited for all outstanding IOs against
that chunk.  It's the latter part which is the problem.  It is 
expensive to keep track of all outstanding IOs on a per-stripe basis,
so when we place a lock on a stripe and come to wait for
already-submitted IOs to complete, it is much easier just to do that
flush volume-wide.  It's not a complete lock on the whole volume, just
a temporary mutex to ensure that there are no IOs left outstanding on
the stripe we're locking.

Cheers,
 Stephen

Re: [evms-devel] [linux-lvm] [ANNOUNCE] LVM reimplementation ready for beta testing

[Kevin Corry]

On Thursday 31 January 2002 06:52, Joe Thornber wrote:
> On Thu, Jan 31, 2002 at 01:52:29PM -0600, Steve Pratt wrote:
> > No, not really.  We only put in the kernel the things that make sense to
> > be in the kernel, discovery logic, ioctl support, I/O path.  All
> > configuration is handled in user space.
>
> There's still a *lot* of code in there; > 26,000 lines in fact.
> Whereas device-mapper weighs in at ~2,600 lines.  This is just because
> you've decided to take a different route from us, you may be proven to
> be correct.

Just so everyone is clear on the amount of code we are talking about, here 
are my current counts of the different kernel drivers (based on code for 
2.4.17):

EVMS: 17685 (this includes drivers/evms and include/linux/evms)

device-mapper: 2895 (this includes device-mapper/kernel/common and 
device-mapper/kernel/ioctl - I've left out device-mapper/kernel/fs since only 
one interface can be active at a time)

Current MD and LVM1: 11105 (this includes drivers/md, include/linux/lvm.h, 
and include/linux/raid/)

Linux 2.4.17: 2519386 (a clean kernel, without EVMS or the latest LVM1 
updates, and not counting asm code)

See http://www.dwheeler.com/sloccount/ for the tool that I used to get these 
counts. 

So I will agree - device-mapper does provide a nice, general-purpose I/O 
remapping service in a small amount of code. Kernel bloat is obviously a big 
concern as more functionality is added to Linux, and achieving a desired set 
of functionality with less code is generally a good thing.

However, I don't think that the size of EVMS should just be written off as 
kernel bloat. (I don't think any of the LVM guys have specifically said this, 
but I have heard this comment from others, and I don't think they are looking 
at the whole picture). We are talking about seven-tenths of a one percent 
increase in the total size of the kernel. And if you consider that EVMS has 
implemented support for LVM1 and MD, then EVMS is really only adding 6580 
lines of code to the kernel. On top of that, EVMS has support for several 
disk partitioning formats. (This support does not yet fully duplicate the 
advanced partition support in the kernel, so I can't yet give any definite 
numeric comparisons.) There is also support for the AIX LVM and the OS/2 LVM, 
as well as general bad-block-relocation and snapshotting support. For all of 
this extra functionality, I don't believe the extra code is unwarranted.

> I would like the two projects to help each other, but not to the point
> where one group of people has to say 'you are completely right, we
> will stop developing our project'.  It's unlikely that either of us is
> 100% correct; but I do think device-mapper splits off a nice chunk of
> services that is useful to *all* people who want to do volume
> management.  As such I see that as one area where we may eventually
> work together.
>
> Similarly I expect to be providing an *optional* kernel module for LVM
> users who wish to do in kernel discovery of a root LV, so if the EVMS
> team has managed to get a nice generic way of iterating block devices
> etc.  into the kernel, we would be able to take advantage of that.
> Are you trying to break out functionality so it benefits other Linux
> projects ? or is EVMS just one monolithic application embedded in the
> kernel ?

I have been thinking about this today and looking over some of the 
device-mapper interfaces. I will agree that, in concept, EVMS could be 
modified to use device-mapper for I/O remapping. However, as things stand 
today, I don't think the transition would be easy.

As I'm trying to envision it, the EVMS runtime would become a "volume 
recognition" framework (see tanget below). Every current EVMS plugin would 
then probe all available devices and communicate the necessary volume 
remapping info to device-mapper through the ioctl interface. (An in-kernel 
API set might be nice to avoid the overhead of the ioctl path). A new device 
would then be created for every node that every plugin recognizes. This 
brings up my first objection. With this approach, there would be an exposed 
device for every node in a volume stack, whereas the current EVMS design only 
exposes nodes for the final volumes. Ignoring the dwindling minor-number 
issue which should go away in 2.5, you still take up space in the 
buffer-cache for every one of these devices, which introduces the possibility 
of cache-incoherencies.

Maybe this example will help: Say we have four disks. These four disks have 
one partition each, and are striped together with RAID-0 (using MD). This MD 
device is then made into an LVM PV and put in a volume group, and an LV is 
created from part of the space in that group. Then down the line, you decide 
to do some backups, and create another LV to use as a snapshot of the first 
LV. (For those who might be wondering, this is a very realistic scenario.)


                  Snap_of_LV1
                      |
          LV1        LV2
          _|__________|_
         | Volume Group |
          --------------
                | 
               md0
   _____________|_______________
  |         |         |         |
 sda1      sdb1      sdc1      sdd1
  |         |         |         |
 sda       sdb       sdc       sdd

In this scenario, we would wind up with exposed devices for every item in 
this graph (except the volume group). But in reality, we don't want someone 
coming along and mucking with md0 or with LV2 or with any of the disk 
partitions, because they are all in use by the two volumes at the top.

<tangent>

As we know, EVMS does volume discovery in the kernel. LVM1 does discovery in 
user-space, but Joe has hinted at an in-kernel LVM1 discovery module to work 
with device-mapper. Back when we started on EVMS, people were basically 
shouting "we need in-kernel discovery!", so that's the route we took. This is 
why it looks like EVMS has so much code. I'd say 50-75% of each plugin is 
devoted to discovery. Of course, today there seem to be people shouting, 
"let's move all discovery into user-space!". Well, I suppose that approach is 
feasible, but I personally don't agree with it. My belief is that it's the 
kernel's job to tell user-space what the hardware looks like, not the other 
way around. If we move partition/volume/etc discovery into user-space, at 
what point do we move device recognition into user-space? Looking down that 
path just seems more and more like a micro-kernel approach, and I'm sure we 
don't want to rehash that discussion.

</tangent>

Next, from what I've seen, device-mapper provides static remappings from one 
device to another. It seems to be a good approach for setting up things like 
disk partitions and LVM linear LVs. There is also striping support in 
device-mapper, but I'm assuming it uses one notion of striping. For instance, 
RAID-0 striping in MD is handled differently than striped LVs in LVM, and I 
think AIX striping is also different. I'm not sure if one stiping module 
could be generic enough to handle all of these cases. But, maybe it can. I'll 
have to think more about that one.

How about mirroring? Does the linear module in device-mapper allow for 1-to-n 
mappings?  This would be similar to the way AIX does mirroring, where each LP 
of an LV can map to up to three PPs on different PVs.

How would device-mapper handle a remapping that is dynamic at runtime? For 
instance, how would device-mapper handle bad-block-relocation? Actually, it 
seems you have dealt with this from one point of view in the snapshotting 
code in device-mapper. In order for persistent snapshots (or bad-block) to 
work, device-mapper needs a module which knows the metadata format, because 
that metadata has to be written at runtime. So another device-mapper "module" 
would need to be written to handle bad-block. This implicitly limits the 
capabilities of device-mapper, or else ties it directly to the recognition 
code. For EVMS and device-mapper to work together, we would have to agree on 
metadata formats for these types of modules. Other similar example that come 
to mind are RAID-5 and block-level encryption.

Now, don't get me totally wrong. I'm not saying using device-mapper in EVMS 
is impossible. I'm just pointing out some of the issues I currently see with 
making such a transition. Perhaps some of these issues can be addressed, from 
either side.

Ultimately, I agree with Joe and Alasdair - I think there is room for both 
projects. There are plenty of other examples of so-called competing projects 
that co-exist just fine - KDE/Gnome, ReiserFS/JFS/XFS/ext3 - hell, there's 
even two virtual memory managers to choose from! So if it just turns out that 
Linux has a choice of two volume managers, then I don't have any problem with 
it. I will say that it is somewhat unfortunate that we couldn't have worked 
together more, but it seems to me that timing is what kept it from happening. 
EVMS was under development when LVM was getting ready for their 1.0 release, 
and now EVMS is trying to get a final release out as the new device-mapper is 
coming along. Unfortunately we each have our own deadlines to think about. 
Maybe down the line there will be more time to get together and figure out 
ways to get these different technologies to work together.

-Kevin
 

Re: [evms-devel] [linux-lvm] [ANNOUNCE] LVM reimplementation ready for beta testing

[Alan Cox]

> As I'm trying to envision it, the EVMS runtime would become a "volume 
> recognition" framework (see tanget below). Every current EVMS plugin would 

Volume recognition is definitely a user space area. There are a huge array
of things you want to do in some environments that you cannot do from
kernel space

Simple example: We have mount by label, imagine trying to extend that in
kernel space to automatically do LDAP queries to find a remote handle to
the volume and NFS mount it. It's easy in user space.

Alan

Re: [evms-devel] [linux-lvm] [ANNOUNCE] LVM reimplementation ready for beta testing

[Kevin Corry]

On Thursday 31 January 2002 06:52, Joe Thornber wrote:
> On Thu, Jan 31, 2002 at 01:52:29PM -0600, Steve Pratt wrote:
> > No, not really.  We only put in the kernel the things that make sense to
> > be in the kernel, discovery logic, ioctl support, I/O path.  All
> > configuration is handled in user space.
>
> There's still a *lot* of code in there; > 26,000 lines in fact.
> Whereas device-mapper weighs in at ~2,600 lines.  This is just because
> you've decided to take a different route from us, you may be proven to
> be correct.

Just so everyone is clear on the amount of code we are talking about, here 
are my current counts of the different kernel drivers (based on code for 
2.4.17):

EVMS: 17685 (this includes drivers/evms and include/linux/evms)

device-mapper: 2895 (this includes device-mapper/kernel/common and 
device-mapper/kernel/ioctl - I've left out device-mapper/kernel/fs since only 
one interface can be active at a time)

Current MD and LVM1: 11105 (this includes drivers/md, include/linux/lvm.h, 
and include/linux/raid/)

Linux 2.4.17: 2519386 (a clean kernel, without EVMS or the latest LVM1 
updates, and not counting asm code)

See http://www.dwheeler.com/sloccount/ for the tool that I used to get these 
counts. 

So I will agree - device-mapper does provide a nice, general-purpose I/O 
remapping service in a small amount of code. Kernel bloat is obviously a big 
concern as more functionality is added to Linux, and achieving a desired set 
of functionality with less code is generally a good thing.

However, I don't think that the size of EVMS should just be written off as 
kernel bloat. (I don't think any of the LVM guys have specifically said this, 
but I have heard this comment from others, and I don't think they are looking 
at the whole picture). We are talking about seven-tenths of a one percent 
increase in the total size of the kernel. And if you consider that EVMS has 
implemented support for LVM1 and MD, then EVMS is really only adding 6580 
lines of code to the kernel. On top of that, EVMS has support for several 
disk partitioning formats. (This support does not yet fully duplicate the 
advanced partition support in the kernel, so I can't yet give any definite 
numeric comparisons.) There is also support for the AIX LVM and the OS/2 LVM, 
as well as general bad-block-relocation and snapshotting support. For all of 
this extra functionality, I don't believe the extra code is unwarranted.

> I would like the two projects to help each other, but not to the point
> where one group of people has to say 'you are completely right, we
> will stop developing our project'.  It's unlikely that either of us is
> 100% correct; but I do think device-mapper splits off a nice chunk of
> services that is useful to *all* people who want to do volume
> management.  As such I see that as one area where we may eventually
> work together.
>
> Similarly I expect to be providing an *optional* kernel module for LVM
> users who wish to do in kernel discovery of a root LV, so if the EVMS
> team has managed to get a nice generic way of iterating block devices
> etc.  into the kernel, we would be able to take advantage of that.
> Are you trying to break out functionality so it benefits other Linux
> projects ? or is EVMS just one monolithic application embedded in the
> kernel ?

I have been thinking about this today and looking over some of the 
device-mapper interfaces. I will agree that, in concept, EVMS could be 
modified to use device-mapper for I/O remapping. However, as things stand 
today, I don't think the transition would be easy.

As I'm trying to envision it, the EVMS runtime would become a "volume 
recognition" framework (see tanget below). Every current EVMS plugin would 
then probe all available devices and communicate the necessary volume 
remapping info to device-mapper through the ioctl interface. (An in-kernel 
API set might be nice to avoid the overhead of the ioctl path). A new device 
would then be created for every node that every plugin recognizes. This 
brings up my first objection. With this approach, there would be an exposed 
device for every node in a volume stack, whereas the current EVMS design only 
exposes nodes for the final volumes. Ignoring the dwindling minor-number 
issue which should go away in 2.5, you still take up space in the 
buffer-cache for every one of these devices, which introduces the possibility 
of cache-incoherencies.

Maybe this example will help: Say we have four disks. These four disks have 
one partition each, and are striped together with RAID-0 (using MD). This MD 
device is then made into an LVM PV and put in a volume group, and an LV is 
created from part of the space in that group. Then down the line, you decide 
to do some backups, and create another LV to use as a snapshot of the first 
LV. (For those who might be wondering, this is a very realistic scenario.)


                  Snap_of_LV1
                      |
          LV1        LV2
          _|__________|_
         | Volume Group |
          --------------
                | 
               md0
   _____________|_______________
  |         |         |         |
 sda1      sdb1      sdc1      sdd1
  |         |         |         |
 sda       sdb       sdc       sdd

In this scenario, we would wind up with exposed devices for every item in 
this graph (except the volume group). But in reality, we don't want someone 
coming along and mucking with md0 or with LV2 or with any of the disk 
partitions, because they are all in use by the two volumes at the top.

<tangent>

As we know, EVMS does volume discovery in the kernel. LVM1 does discovery in 
user-space, but Joe has hinted at an in-kernel LVM1 discovery module to work 
with device-mapper. Back when we started on EVMS, people were basically 
shouting "we need in-kernel discovery!", so that's the route we took. This is 
why it looks like EVMS has so much code. I'd say 50-75% of each plugin is 
devoted to discovery. Of course, today there seem to be people shouting, 
"let's move all discovery into user-space!". Well, I suppose that approach is 
feasible, but I personally don't agree with it. My belief is that it's the 
kernel's job to tell user-space what the hardware looks like, not the other 
way around. If we move partition/volume/etc discovery into user-space, at 
what point do we move device recognition into user-space? Looking down that 
path just seems more and more like a micro-kernel approach, and I'm sure we 
don't want to rehash that discussion.

</tangent>

Next, from what I've seen, device-mapper provides static remappings from one 
device to another. It seems to be a good approach for setting up things like 
disk partitions and LVM linear LVs. There is also striping support in 
device-mapper, but I'm assuming it uses one notion of striping. For instance, 
RAID-0 striping in MD is handled differently than striped LVs in LVM, and I 
think AIX striping is also different. I'm not sure if one stiping module 
could be generic enough to handle all of these cases. But, maybe it can. I'll 
have to think more about that one.

How about mirroring? Does the linear module in device-mapper allow for 1-to-n 
mappings?  This would be similar to the way AIX does mirroring, where each LP 
of an LV can map to up to three PPs on different PVs.

How would device-mapper handle a remapping that is dynamic at runtime? For 
instance, how would device-mapper handle bad-block-relocation? Actually, it 
seems you have dealt with this from one point of view in the snapshotting 
code in device-mapper. In order for persistent snapshots (or bad-block) to 
work, device-mapper needs a module which knows the metadata format, because 
that metadata has to be written at runtime. So another device-mapper "module" 
would need to be written to handle bad-block. This implicitly limits the 
capabilities of device-mapper, or else ties it directly to the recognition 
code. For EVMS and device-mapper to work together, we would have to agree on 
metadata formats for these types of modules. Other similar example that come 
to mind are RAID-5 and block-level encryption.

Now, don't get me totally wrong. I'm not saying using device-mapper in EVMS 
is impossible. I'm just pointing out some of the issues I currently see with 
making such a transition. Perhaps some of these issues can be addressed, from 
either side.

Ultimately, I agree with Joe and Alasdair - I think there is room for both 
projects. There are plenty of other examples of so-called competing projects 
that co-exist just fine - KDE/Gnome, ReiserFS/JFS/XFS/ext3 - hell, there's 
even two virtual memory managers to choose from! So if it just turns out that 
Linux has a choice of two volume managers, then I don't have any problem with 
it. I will say that it is somewhat unfortunate that we couldn't have worked 
together more, but it seems to me that timing is what kept it from happening. 
EVMS was under development when LVM was getting ready for their 1.0 release, 
and now EVMS is trying to get a final release out as the new device-mapper is 
coming along. Unfortunately we each have our own deadlines to think about. 
Maybe down the line there will be more time to get together and figure out 
ways to get these different technologies to work together.

-Kevin
 

Re: [evms-devel] [linux-lvm] [ANNOUNCE] LVM reimplementation ready for beta testing

[Alan Cox]

> As I'm trying to envision it, the EVMS runtime would become a "volume 
> recognition" framework (see tanget below). Every current EVMS plugin would 

Volume recognition is definitely a user space area. There are a huge array
of things you want to do in some environments that you cannot do from
kernel space

Simple example: We have mount by label, imagine trying to extend that in
kernel space to automatically do LDAP queries to find a remote handle to
the volume and NFS mount it. It's easy in user space.

Alan

Re: [evms-devel] [linux-lvm] [ANNOUNCE] LVM reimplementation ready for beta testing

[Andrew Clausen]

On Fri, Feb 01, 2002 at 03:59:06PM -0600, Kevin Corry wrote:
> Current MD and LVM1: 11105 (this includes drivers/md, include/linux/lvm.h, 
> and include/linux/raid/)

Why are you counting MD?  It should be trivial to implement raid
with device-mapper.

> So I will agree - device-mapper does provide a nice, general-purpose I/O 
> remapping service in a small amount of code.

It provides everything you need (from a kernel) to do Volume
Management in general.  (What's missing?)

> And if you consider that EVMS has 
> implemented support for LVM1 and MD, then EVMS is really only adding 6580 
> lines of code to the kernel.

Yep, it's not the size, but the complexity.

device-mapper is *simple*.  It couldn't be simpler!
simple is *good*, as long as it's not "too simple", in the sense
that it doesn't get the job done.  So, tell us what the disadvantages
of device-mapper's "simplemindedness" is!

> In this scenario, we would wind up with exposed devices for every item in 
> this graph (except the volume group). But in reality, we don't want someone 
> coming along and mucking with md0 or with LV2 or with any of the disk 
> partitions, because they are all in use by the two volumes at the top.

It's the user's fault if they choose to write on such a device.

> As we know, EVMS does volume discovery in the kernel. LVM1 does discovery in 
> user-space, but Joe has hinted at an in-kernel LVM1 discovery module to work 
> with device-mapper. Back when we started on EVMS, people were basically 
> shouting "we need in-kernel discovery!",

Not me!  Anyway, device-mapper is compatible with in-kernel discovery.
But, why not just stick it in initramfs?  (Are there any Issues?)

> My belief is that it's the 
> kernel's job to tell user-space what the hardware looks like, not the other 
> way around.  If we move partition/volume/etc discovery into user-space, at 
> what point do we move device recognition into user-space?

There's a big difference: partition discovery isn't actually a hardware
thing, it's a software thing.

Some types of device recognition (iSCSI?) are also software things.

WRT: beliefs, I think the kernel should provide all the services that
can't be implemented well in userspace.

> Looking down that 
> path just seems more and more like a micro-kernel approach, and I'm sure we 
> don't want to rehash that discussion.

What's your point?

> Next, from what I've seen, device-mapper provides static remappings from one 
> device to another. It seems to be a good approach for setting up things like 
> disk partitions and LVM linear LVs. There is also striping support in 
> device-mapper, but I'm assuming it uses one notion of striping. For instance, 
> RAID-0 striping in MD is handled differently than striped LVs in LVM, and I 
> think AIX striping is also different. I'm not sure if one stiping module 
> could be generic enough to handle all of these cases.

Why do you need a single striping module?

> How about mirroring? Does the linear module in device-mapper allow for 1-to-n 
> mappings?

Wouldn't you use the strip module, with the chunk size set to
a good approximation of infinity?  Haven't tried it...

Anyway, this is just a small detail... if there was no way of doing it,
it would be easy to add a module...

> So another device-mapper "module" 
> would need to be written to handle bad-block.  This implicitly limits the 
> capabilities of device-mapper, or else ties it directly to the recognition 
> code. For EVMS and device-mapper to work together, we would have to agree on 
> metadata formats for these types of modules. Other similar example that come 
> to mind are RAID-5 and block-level encryption.

Agreed... but these shouldn't be big problems?  (raid-5 is already
a standard...)

Andrew

Re: [evms-devel] [linux-lvm] [ANNOUNCE] LVM reimplementation ready for beta testing

[Alan Cox]

> > this graph (except the volume group). But in reality, we don't want someone 
> > coming along and mucking with md0 or with LV2 or with any of the disk 
> > partitions, because they are all in use by the two volumes at the top.
> 
> It's the user's fault if they choose to write on such a device.

You want access to the raw devices as well as the virtual volumes. You
try doing SMART diagnostics online on an md volume if you can't get
at /dev/hd* and /dev/sd*.

You don't want to mount both layers at once I suspect, but even that
may be questionable for a read only mirror.

Alan

Re: [evms-devel] [linux-lvm] [ANNOUNCE] LVM reimplementation ready for beta testing

[Andrew Clausen]

On Fri, Feb 01, 2002 at 03:59:06PM -0600, Kevin Corry wrote:
> Current MD and LVM1: 11105 (this includes drivers/md, include/linux/lvm.h, 
> and include/linux/raid/)

Why are you counting MD?  It should be trivial to implement raid
with device-mapper.

> So I will agree - device-mapper does provide a nice, general-purpose I/O 
> remapping service in a small amount of code.

It provides everything you need (from a kernel) to do Volume
Management in general.  (What's missing?)

> And if you consider that EVMS has 
> implemented support for LVM1 and MD, then EVMS is really only adding 6580 
> lines of code to the kernel.

Yep, it's not the size, but the complexity.

device-mapper is *simple*.  It couldn't be simpler!
simple is *good*, as long as it's not "too simple", in the sense
that it doesn't get the job done.  So, tell us what the disadvantages
of device-mapper's "simplemindedness" is!

> In this scenario, we would wind up with exposed devices for every item in 
> this graph (except the volume group). But in reality, we don't want someone 
> coming along and mucking with md0 or with LV2 or with any of the disk 
> partitions, because they are all in use by the two volumes at the top.

It's the user's fault if they choose to write on such a device.

> As we know, EVMS does volume discovery in the kernel. LVM1 does discovery in 
> user-space, but Joe has hinted at an in-kernel LVM1 discovery module to work 
> with device-mapper. Back when we started on EVMS, people were basically 
> shouting "we need in-kernel discovery!",

Not me!  Anyway, device-mapper is compatible with in-kernel discovery.
But, why not just stick it in initramfs?  (Are there any Issues?)

> My belief is that it's the 
> kernel's job to tell user-space what the hardware looks like, not the other 
> way around.  If we move partition/volume/etc discovery into user-space, at 
> what point do we move device recognition into user-space?

There's a big difference: partition discovery isn't actually a hardware
thing, it's a software thing.

Some types of device recognition (iSCSI?) are also software things.

WRT: beliefs, I think the kernel should provide all the services that
can't be implemented well in userspace.

> Looking down that 
> path just seems more and more like a micro-kernel approach, and I'm sure we 
> don't want to rehash that discussion.

What's your point?

> Next, from what I've seen, device-mapper provides static remappings from one 
> device to another. It seems to be a good approach for setting up things like 
> disk partitions and LVM linear LVs. There is also striping support in 
> device-mapper, but I'm assuming it uses one notion of striping. For instance, 
> RAID-0 striping in MD is handled differently than striped LVs in LVM, and I 
> think AIX striping is also different. I'm not sure if one stiping module 
> could be generic enough to handle all of these cases.

Why do you need a single striping module?

> How about mirroring? Does the linear module in device-mapper allow for 1-to-n 
> mappings?

Wouldn't you use the strip module, with the chunk size set to
a good approximation of infinity?  Haven't tried it...

Anyway, this is just a small detail... if there was no way of doing it,
it would be easy to add a module...

> So another device-mapper "module" 
> would need to be written to handle bad-block.  This implicitly limits the 
> capabilities of device-mapper, or else ties it directly to the recognition 
> code. For EVMS and device-mapper to work together, we would have to agree on 
> metadata formats for these types of modules. Other similar example that come 
> to mind are RAID-5 and block-level encryption.

Agreed... but these shouldn't be big problems?  (raid-5 is already
a standard...)

Andrew

Re: [evms-devel] [linux-lvm] [ANNOUNCE] LVM reimplementation ready for beta testing

[Alan Cox]

> > this graph (except the volume group). But in reality, we don't want someone 
> > coming along and mucking with md0 or with LV2 or with any of the disk 
> > partitions, because they are all in use by the two volumes at the top.
> 
> It's the user's fault if they choose to write on such a device.

You want access to the raw devices as well as the virtual volumes. You
try doing SMART diagnostics online on an md volume if you can't get
at /dev/hd* and /dev/sd*.

You don't want to mount both layers at once I suspect, but even that
may be questionable for a read only mirror.

Alan

Re: [evms-devel] [linux-lvm] [ANNOUNCE] LVM reimplementation ready for beta testing

[Joe Thornber]

On Fri, Feb 01, 2002 at 03:59:06PM -0600, Kevin Corry wrote:
> I have been thinking about this today and looking over some of the 
> device-mapper interfaces. I will agree that, in concept, EVMS could be 
> modified to use device-mapper for I/O remapping. However, as things stand 
> today, I don't think the transition would be easy.
> 
> As I'm trying to envision it, the EVMS runtime would become a "volume 
> recognition" framework (see tanget below). Every current EVMS plugin would 
> then probe all available devices and communicate the necessary volume 
> remapping info to device-mapper through the ioctl interface. (An in-kernel 
> API set might be nice to avoid the overhead of the ioctl path).

The in kernel API already exists in drivers/md/dm.h, see dm-ioctl.c
for example code that uses this.


> A new device would then be created for every node that every plugin
> recognizes. This brings up my first objection. With this approach,
> there would be an exposed device for every node in a volume stack,
> whereas the current EVMS design only exposes nodes for the final
> volumes.

True, all the devices are exposed in /dev/device-mapper.  What we do
with LVM2 is stick with the /dev/<vg_name>/<lv_name> scheme, where the
lv is a symlink to the relevant node in the device-mapper dir.  That
way people only see the top level devices, unless they peek under the
hood by looking at /dev/device-mapper.

> Next, from what I've seen, device-mapper provides static remappings
> from one device to another. It seems to be a good approach for
> setting up things like disk partitions and LVM linear LVs. There is
> also striping support in device-mapper, but I'm assuming it uses one
> notion of striping. For instance, RAID-0 striping in MD is handled
> differently than striped LVs in LVM, and I think AIX striping is
> also different. I'm not sure if one stiping module could be generic
> enough to handle all of these cases. But, maybe it can. I'll have to
> think more about that one.

The thing that you've missed is that ranges of sectors are mapped by
dm onto 'targets', these targets are instances of a 'target_type'
virtual class.  You can define your own target types in a seperate
module and register them using the

int dm_register_target(struct target_type *t);

function (see <linux/device-mapper.h>).  We originally intended all
the standard target types to be defined in seperate modules, but that
seemed silly considering the tiny size of them (eg, < 200 lines for striping).

The targets type that we will be using in LVM2 are:

io-err - errors every io
linear
striped
snapshot, and snapshot origin
mirror (used to provide pvmove support as well as mirroring).

(both the snapshot targets and mirroring target will use kcopyd for
efficient copying using kiobufs.)

So if you invent a new form of striping, just implement a new target type.

> How about mirroring? Does the linear module in device-mapper allow
> for 1-to-n mappings?  This would be similar to the way AIX does
> mirroring, where each LP of an LV can map to up to three PPs on
> different PVs.

No, there's a seperate mirror target that Patrick Caulfields working
on at the moment.

> How would device-mapper handle a remapping that is dynamic at runtime?

Suspending a device flushes all pending io to the device, ie. all io
that has been through the driver but not yet completed.  Once you have
suspended you are free to loaded a new mapping table into the device
(this is essential for implementing snapshots and pvmove).  When the
new mapping is in place you just have to resume the device and away
you go.

- Joe

Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementationre ady for beta testing

[James Hawtin]

On Fri, 1 Feb 2002, Heinz J . Mauelshagen wrote:

> 
> It is quite intersting how many people don't take backups even though
> almost everybody claims to know that backups are crucial and some of them
> know that they are still, even though they might have RAID in place.
> 

I think the reason most people don't do backups is because, hard disk
drives are really big and really cheap. Tape drives are really expensive,
media is also expensive and frankly tapes are very small for the cost.
Effective backup can double the cost of a system and requires time to
manage it.

For the "home" market its just to much. The only system just about
affordable is DAT tape, DLT blanks are about double the cost for the
storage ammount. I bought a DDS4 drive for "home" use, 1 week after I got
it I had my laptop stolen, so it justified its cost pretty fast :-) A 60
gig hard disk, costing 100 pounds requires 2-3 DSS4 tapes to backup At 15
pounds each, one round of backups cost 1/3 of the cost of the disk
multiply that by the number of disks, add in the hassle factor of changing
tapes. Auto changers are nice, but alot more! For "PC" based systems the
backup costs are oftain higher than the price of the machine, and thats
hard for people to justify.

If anyone suggests they use "hard drives" to back up I will scream, cost
that system just don't scale, and yup you have an old copy but do people
ever update it? Yes... when they get another hard drive... however before
that happens they have put the old on back into service because they
needed a little more space ;-)

All in all, backup is expensive/slow, my sister/mom/dad ain't going to pay
that kind of money for home... For Work its slow, fine if your a sysadmin
and that your job... If you are a sysadmin you probably don't get to do
bugets, your manager does, and they don't understand the cost and aways
want to cut corners to keep costs down, XXX is a "developement" system so
we don't need to back it up cos its not "production" etc etc. Store all
your data on the network file server, (which is aways full with p0rn) and
connected via a doggie network, taping 3 minutes to access the smallest
file, so people store data on the local machine, and forget to copy it
back to the network...

This why people don't have backups in my book.

Sorry I didn't mean this to become a rant, just got on a roll...

James

Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementationre ady for beta testing

[Patrick Caulfield]

On Tue, Feb 05, 2002 at 02:03:20PM +0000, James Hawtin wrote:
> 
> 
> On Fri, 1 Feb 2002, Heinz J . Mauelshagen wrote:
> 
> > 
> > It is quite intersting how many people don't take backups even though
> > almost everybody claims to know that backups are crucial and some of them
> > know that they are still, even though they might have RAID in place.
> > 
> 

> <snip>

They are not reasons, they are excuses. If you value your data you take backups.

patrick

Re: [linux-lvm] Re: [lvm-devel] [ANNOUNCE] LVM reimplementationre ady for beta testing

[Jesus Manuel NAVARRO LOPEZ]

Hi, James:

James Hawtin wrote:

> 
> On Fri, 1 Feb 2002, Heinz J . Mauelshagen wrote:
> 
> 
>>It is quite intersting how many people don't take backups even though
>>almost everybody claims to know that backups are crucial and some of them
>>know that they are still, even though they might have RAID in place.
>>
>>
> 
> I think the reason most people don't do backups is because, hard disk
> drives are really big and really cheap. Tape drives are really expensive,
> media is also expensive and frankly tapes are very small for the cost.
> Effective backup can double the cost of a system and requires time to
> manage it.
> 


Yes.  That's true... as you say, for grandma's PC.  Backups (and all the 
other disaster/recovery plans) are defensive, so they *must* be 
cost-effective solutions.  *If* you really don't care loosing a week's 
data or even all the data, and with low probability you just can go with 
hard-drive backups or even without backups at all... if you *really* 
know that there's not *if* you're going to loose data, but *when*, and 
accept it.


> For the "home" market its just to much. The only system just about


It is too much *work*, not money: five to twelve zip "disks" (100/250MB 
each) is usually more than enough for incremental backups.  Well planned 
you don't need full backups... you don't need full backups for programs 
you already have in another media (like your programs' original CD's) 
you don't need backups for cold-storable data (six month old MP3 you 
just listen from time to time) you just burn a CD from time to time... 
at the end only personal data is *needed* to be backuped and this is 
*usually* able to fit on one/two zips.  The problem is the time and the 
savvyness to plan it (in advance).  The same goes with any other 
insurance system: for most people you just insurance if you're legally 
enforced, or it's the "common" way (like the health care insurance in USA).

...But the other guy was talking about a *professional* environment.


> affordable is DAT tape, DLT blanks are about double the cost for the
> storage ammount. I bought a DDS4 drive for "home" use, 1 week after I got
> it I had my laptop stolen, so it justified its cost pretty fast :-) A 60
> gig hard disk, costing 100 pounds requires 2-3 DSS4 tapes to backup At 15
> pounds each, one round of backups cost 1/3 of the cost of the disk
> multiply that by the number of disks, add in the hassle factor of changing
> tapes. Auto changers are nice, but alot more! For "PC" based systems the
> backup costs are oftain higher than the price of the machine, and thats
> hard for people to justify.
> 


Hard drives are *not* cheaper.  Indeed they're more expensive by far!!!
You seem to consider that a single copy on the same system can be 
considered a backup, but it is not:
*If you don't have incrementals for a variable amount of time you have 
just crap
*If your backup media is around the system to be protected you just have 
crap

So, for HDDs to be a *real* backup solution, you need as many HDDs as 
you would have tapes (say 10 to 20) so you can have, let's say, daily 
incrementals, placed in-house, weekly "bigs" somehow "near", and monthly 
off-site.  Now, tapes are cheaper than disks, so if you need to have the 
same amount (more or less) disks will be more expensive... and, 
remember, that, while statically-placed disks are more confiable than 
tapes, that's not true as soon as you start moving them here and there. 
  Obviously you can go with PCMCIA disks, but they're *a lot* more 
expensive.


> If anyone suggests they use "hard drives" to back up I will scream, cost
> that system just don't scale, and yup you have an old copy but do people
> ever update it? Yes... when they get another hard drive... however before
> that happens they have put the old on back into service because they
> needed a little more space ;-)
> 


This is true too.


> All in all, backup is expensive/slow, my sister/mom/dad ain't going to pay
> that kind of money for home... For Work its slow, fine if your a sysadmin


They're expensive *by definition* (just in the same way any other 
insurance is: it doesn't *produce* anything).  Anyway, probably your 
mom's life doesn't deppend on data on her home PC, so it's fine going 
with no backup at all, or burning a CD from time to time.


> and that your job... If you are a sysadmin you probably don't get to do
> bugets, your manager does, and they don't understand the cost and aways
> want to cut corners to keep costs down, XXX is a "developement" system so
> we don't need to back it up cos its not "production" etc etc. Store all
> your data on the network file server, (which is aways full with p0rn) and
> connected via a doggie network, taping 3 minutes to access the smallest
> file, so people store data on the local machine, and forget to copy it
> back to the network...
> 


Truly enough: idiocy doesn't pay.


> This why people don't have backups in my book.
> 


And I'm with you.

-- 
SALUD,
Jes�s
***
jesus_navarro@promofinarsa.es
***

[linux-lvm] (OT) Backups (was Re: LVM reimplementationre ady for beta testing...)

[Chad C. Walstrom]

[-- Attachment #1: Type: text/plain, Size: 1623 bytes --]

Off-topic: multivolume tar archive + CDROM == pain!

For those home backup solutions, don't ever listen to anyone who tells
you that doing multivolume tar archives on CDROM is a good idea.  It
isn't!  I know this from experience! ;-)  It's very painful.  CDROM's
are much better used as random access devices as opposed to linear.  tar
!= random access :. tar != good backup solution w/CDROM media.  Maybe
tar'ing up smaller chunks of files that != multivolume would be ok ;-).

Instead, make a good plan of attack for backing up files based on their
volatility and importance.  With CDROM's as a media, think regularily
scheduled full backups of selected data groups.

There are a few things I like to back up frequently:

        * $HOME (excluding things like netscape cache or download directory)
        * /var/cvs/$repositories
        * dpkg --get-selections > /var/backup/dpkg-selections.list
        * /var/backup/*
        * /etc/*
        * /var/lib/$pkgmgrfiles
        * /var/log/$newlogfiles

Things I back up infrequently:

        * *.{gif,png,jpg,mov,mp3,mpeg,ogg} # etc
        * /home/ftp/pub/*
        * /var/log/$oldlogfiles

Actually, I try to burn the ogg and mp3 files to disc immediately.
Sometimes I get smart and include the source code or binary code for the
a media player.

Anyway, back to your regularily scheduled off-topic thread. ;-)

-- 
Chad Walstrom <chewie@wookimus.net>                 | a.k.a. ^chewie
http://www.wookimus.net/                            | s.k.a. gunnarr
Get my public key, ICQ#, etc. $(mailx -s 'get info' chewie@wookimus.net)

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

[linux-lvm] Backup costs (was: LVM reimplementationre)

[Benjamin Scott]

On Tue, 5 Feb 2002, James Hawtin wrote:
> I think the reason most people don't do backups is because, hard disk
> drives are really big and really cheap. Tape drives are really expensive,
> media is also expensive and frankly tapes are very small for the cost.

  Here are some prices from a recent Datacomm Warehouse catalog:

  Item            GB   Cost  $/GB
  --------------  ---  ----  ----
  IDE HDD         100   270  2.70
  AIT2             50    90  1.80
  SuperDLT        110   150  1.36
  LTO Ultrium     100   130  1.30
  DDS-4            20    24  1.20

  As you can see, hard disk is actually the most expensive media, not the
least.  This whole "hard disks are cheaper" thing is a myth propagated by
people who have never actually looked at the numbers.

> Effective backup can double the cost of a system and requires time to
> manage it.

  Sure it can.  Losing your data will generally cost even more.

> For the "home" market its just to much.

  The home user has maybe, what, 100 MB of data to protect, tops?  You can
fit that on a $2 CD-RW, for crying out loud.

> This why people don't have backups in my book.

  "Most people" don't have backups because they don't know any better, and
likely don't care.  They bought their Pee Cee at Wal-Mart, and the most
valuable thing on there is their collection of illegal music they downloaded
before Napster went bust.  For those who have legitimately valuable data
(e.g., financial data), they likely don't realize just how much it is going
to cost them when (note -- *WHEN*, not *IF*) they lose it.

  Sure, backups cost money.  You might spend $1500 on a data protection
scheme for a small office server.  Of course, consider how much it would
cost if they did not do that.  I know business that have *gone out of
business* when they lost their only copy of mission-critical data.

  $1500 vs bankruptcy.  Gee, tough call.

  Anyone who has ever bothered to work out the numbers knows that backups
are a bargain.  People who think otherwise are either ignorant or just plain
don't want to know the truth.

-- 
Ben Scott <bscott@ntisys.com>
| The opinions expressed in this message are those of the author and do not |
| necessarily represent the views or policy of any other person, entity or  |
| organization.  All information is provided without warranty of any kind.  |

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Daniel Whicker]

[-- Attachment #1: Type: text/plain, Size: 1537 bytes --]

Ben,
	I think you missed a very important aspect.  The actual investment of
the tape drives.  A SDLT or LTO drive is going to run from about $4000
on up.  According to datacomm warehouse, a HP 230 external LTO drives
runs $3700.  An SDLT drive runs $4300.  You can't compare just the media
costs with that of the drive.  Let's consider a 1TB of data.

SDLT:
$4300 + (10 tapes * $50) = $4800

LTO:
$3700 + (10 tapes * $130) = $5000

AIT2:
$2800 + (20 tapes * $90) = $4600

IDE:
10  Drives * 270 = $2700

DDS4:
$900 + (50 tapes * $24) = $2100

From these numbers (using DataComm Warehouse pricing), you can see that
to backup 1TB, the IDE drives are actually the second cheapest.  And
from a raw performance point of view, they far exceed all the other
alternatives from a random access standpoint, and they are comparable to
even LTO from a transfer rate point of view.

Sorry... had to address this point.

On Wed, 2002-02-06 at 13:14, Benjamin Scott wrote:
>   Here are some prices from a recent Datacomm Warehouse catalog:
> 
>   Item            GB   Cost  $/GB
>   --------------  ---  ----  ----
>   IDE HDD         100   270  2.70
>   AIT2             50    90  1.80
>   SuperDLT        110   150  1.36
>   LTO Ultrium     100   130  1.30
>   DDS-4            20    24  1.20
> 
>   As you can see, hard disk is actually the most expensive media, not the
> least.  This whole "hard disks are cheaper" thing is a myth propagated by
> people who have never actually looked at the numbers.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 232 bytes --]

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[James Mello]

With your argument, were would you stick 10 IDE drives then? Add the cost
of controllers, etc and you start to add more and more cost. The deal is,
backups to tape are done to tape when the media cost savings justify the
outlay of cash for the hardware. Like it was said before, most small
business could get away with burning a CD. But what you *can't* do is
ignore the fact that if you want to be safe, you *must* do backups. Disk
is cool, but make sure that you fully account for the expense in your
environment.

	-- Cheers
	-- James

> Ben,
> 	I think you missed a very important aspect.  The actual investment of
> the tape drives.  A SDLT or LTO drive is going to run from about $4000
> on up.  According to datacomm warehouse, a HP 230 external LTO drives
> runs $3700.  An SDLT drive runs $4300.  You can't compare just the media
> costs with that of the drive.  Let's consider a 1TB of data.
> 
> SDLT:
> $4300 + (10 tapes * $50) = $4800
> 
> LTO:
> $3700 + (10 tapes * $130) = $5000
> 
> AIT2:
> $2800 + (20 tapes * $90) = $4600
> 
> IDE:
> 10  Drives * 270 = $2700
> 
> DDS4:
> $900 + (50 tapes * $24) = $2100
> 
> >From these numbers (using DataComm Warehouse pricing), you can see that
> to backup 1TB, the IDE drives are actually the second cheapest.  And
> from a raw performance point of view, they far exceed all the other
> alternatives from a random access standpoint, and they are comparable to
> even LTO from a transfer rate point of view.
> 
> Sorry... had to address this point.
> 
> On Wed, 2002-02-06 at 13:14, Benjamin Scott wrote:
> >   Here are some prices from a recent Datacomm Warehouse catalog:
> > 
> >   Item            GB   Cost  $/GB
> >   --------------  ---  ----  ----
> >   IDE HDD         100   270  2.70
> >   AIT2             50    90  1.80
> >   SuperDLT        110   150  1.36
> >   LTO Ultrium     100   130  1.30
> >   DDS-4            20    24  1.20
> > 
> >   As you can see, hard disk is actually the most expensive media, not the
> > least.  This whole "hard disks are cheaper" thing is a myth propagated by
> > people who have never actually looked at the numbers.
> 
> 

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Andreas Dilger]

On Feb 06, 2002  14:14 -0500, Benjamin Scott wrote:
>   Here are some prices from a recent Datacomm Warehouse catalog:
> 
>   Item            GB   Cost  $/GB
>   --------------  ---  ----  ----
>   IDE HDD         100   270  2.70
>   AIT2             50    90  1.80
>   SuperDLT        110   150  1.36
>   LTO Ultrium     100   130  1.30
>   DDS-4            20    24  1.20
> 
>   As you can see, hard disk is actually the most expensive media, not the
> least.  This whole "hard disks are cheaper" thing is a myth propagated by
> people who have never actually looked at the numbers.

You don't include the cost of the tape drive, which is often high, while
the cost of the IDE drive has no such additional cost.

>   Sure, backups cost money.  You might spend $1500 on a data protection
> scheme for a small office server.  Of course, consider how much it would
> cost if they did not do that.  I know business that have *gone out of
> business* when they lost their only copy of mission-critical data.
> 
>   $1500 vs bankruptcy.  Gee, tough call.
> 
>   Anyone who has ever bothered to work out the numbers knows that backups
> are a bargain.  People who think otherwise are either ignorant or just plain
> don't want to know the truth.

Oh, I totally agree.  I used to work for IBM and met a lot of people who
lost their hard drive data.  I personally do a lot of backups these days.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Theo Van Dinter]

On Wed, Feb 06, 2002 at 02:14:05PM -0500, Benjamin Scott wrote:
>   Here are some prices from a recent Datacomm Warehouse catalog:
> 
>   Item            GB   Cost  $/GB
>   --------------  ---  ----  ----
>   IDE HDD         100   270  2.70
>   AIT2             50    90  1.80
>   SuperDLT        110   150  1.36
>   LTO Ultrium     100   130  1.30
>   DDS-4            20    24  1.20
> 
>   As you can see, hard disk is actually the most expensive media, not the
> least.  This whole "hard disks are cheaper" thing is a myth propagated by
> people who have never actually looked at the numbers.

Pot, kettle, black.

Yes, a AIT2 _TAPE_ is $90, but you need to factor in the drive, otherwise
the media is useless.  A $270 IDE drive is ready to go.  A quick look at
warehouse.com (to keep the numbers on-par):

AIT2 drive/Sony		2800 + 90 (tape) / 50GB   = 57.80
SuperDLT drive/LaCie	5000 + 150 (tape) / 110GB = 46.82
LTO Ultrium drive/HP	3650 + 130 (tape) / 100GB = 37.80
DDS-4 drive/Seagate	970 + 24 (tape) / 20GB    = 49.70


So yes, not surprisingly, tape is more expensive than disk everytime.
Even if you halfed the prices assuming full (2x) compression on the
tape. Now over time, the per GB price of tape will come down since you
only need the large outlay of cash once, but you can only use one tape
at a time at any given moment.


-- 
Randomly Generated Tagline:
"perl is awesome, perl is great
  perl gets my programs running without all the wait,
  perl lets me do my job smooth as silk,
  it even stays crispy in milk.." - Greg London

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Petro]

On Wed, Feb 06, 2002 at 02:14:05PM -0500, Benjamin Scott wrote:
> On Tue, 5 Feb 2002, James Hawtin wrote:
> > I think the reason most people don't do backups is because, hard disk
> > drives are really big and really cheap. Tape drives are really expensive,
> > media is also expensive and frankly tapes are very small for the cost.
>   Here are some prices from a recent Datacomm Warehouse catalog:
>   Item            GB   Cost  $/GB
>   --------------  ---  ----  ----
>   IDE HDD         100   270  2.70
>   AIT2             50    90  1.80
>   SuperDLT        110   150  1.36
>   LTO Ultrium     100   130  1.30
>   DDS-4            20    24  1.20
>   As you can see, hard disk is actually the most expensive media, not the
> least.  This whole "hard disks are cheaper" thing is a myth propagated by
> people who have never actually looked at the numbers.

    Ok, now factor in time to back up a volume. 

    1 terabyte of small images from netapp F760 to a tape unit takes us
    almost 45 hours.

    Of course, that data has changed by 5 to 10 percent in that 45
    hours.

    It takes less than 20 to dump it to a cheap IDE JBOD/Linux raid
    solution. 

> > Effective backup can double the cost of a system and requires time to
> > manage it.
>   Sure it can.  Losing your data will generally cost even more.

    Yup. Losing data often enough, or in sufficient quantites can kill
    your company.

> > For the "home" market its just to much.
>   The home user has maybe, what, 100 MB of data to protect, tops?  You can
> fit that on a $2 CD-RW, for crying out loud.
> > This why people don't have backups in my book.
 
>   "Most people" don't have backups because they don't know any better, and

    I don't do backups at home because I generally have the important
    data (err...my resume) replicated slapdash over several machines.
    Everything lese I can re-download from the net, or re-install from
    CD. 

    At work, well that's a different story. 
-- 
Share and Enjoy. 

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Kirby C. Bohling]

Yes, but people who think they can backup to harddrives think they can 
backup to one drive.  They think they need gobs of tapes. Sometimes they 
don't think things thru.  They do need gobs of tapes, but in order to 
get the same redundancy they need gobs of drives.  They don't think like 
that.

I have a several gig at a minimum of stuff I would want backed up 
personally.  I work on a lot of code, and I have a lot of modifications 
and/or patches to software I would want to keep around.  Just be cause 
it is a home user machine, doesn't mean they don't use disk space.

Most small businesses generally don't do a good job at testing backups 
and that is the real trick.  I have worked at several places that failed 
to check if backups even work on a regular basis, they just rotate tapes 
and that is it.  Common things overheard when dealing with backups: 
Labels?  What are labels?  Which tape is for last Tuesday, I guess I 
will just have to try them all.  Cleaning tape, nobody told me I needed 
a cleaning tape!

And sometimes it is easier to go out of business then come up with 
$1,500.  You think like an insurance expert.  You understand bad things 
happen, and if you pay a little up front that it will cover the big 
badass expense later on.  Bean counters never see that, they see an 
unecessary expense because bad things don't happen on a weekly basis. 
They have other expense they know will happen so they want to spend 
money on that.

		Kirby





Benjamin Scott wrote:
> On Tue, 5 Feb 2002, James Hawtin wrote:
> 
>>I think the reason most people don't do backups is because, hard disk
>>drives are really big and really cheap. Tape drives are really expensive,
>>media is also expensive and frankly tapes are very small for the cost.
>>
> 
>   Here are some prices from a recent Datacomm Warehouse catalog:
> 
>   Item            GB   Cost  $/GB
>   --------------  ---  ----  ----
>   IDE HDD         100   270  2.70
>   AIT2             50    90  1.80
>   SuperDLT        110   150  1.36
>   LTO Ultrium     100   130  1.30
>   DDS-4            20    24  1.20
> 
>   As you can see, hard disk is actually the most expensive media, not the
> least.  This whole "hard disks are cheaper" thing is a myth propagated by
> people who have never actually looked at the numbers.
> 
> 
>>Effective backup can double the cost of a system and requires time to
>>manage it.
>>
> 
>   Sure it can.  Losing your data will generally cost even more.
> 
> 
>>For the "home" market its just to much.
>>
> 
>   The home user has maybe, what, 100 MB of data to protect, tops?  You can
> fit that on a $2 CD-RW, for crying out loud.
> 
> 
>>This why people don't have backups in my book.
>>
> 
>   "Most people" don't have backups because they don't know any better, and
> likely don't care.  They bought their Pee Cee at Wal-Mart, and the most
> valuable thing on there is their collection of illegal music they downloaded
> before Napster went bust.  For those who have legitimately valuable data
> (e.g., financial data), they likely don't realize just how much it is going
> to cost them when (note -- *WHEN*, not *IF*) they lose it.
> 
>   Sure, backups cost money.  You might spend $1500 on a data protection
> scheme for a small office server.  Of course, consider how much it would
> cost if they did not do that.  I know business that have *gone out of
> business* when they lost their only copy of mission-critical data.
> 
>   $1500 vs bankruptcy.  Gee, tough call.
> 
>   Anyone who has ever bothered to work out the numbers knows that backups
> are a bargain.  People who think otherwise are either ignorant or just plain
> don't want to know the truth.
> 
> 

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Jeff Layton]

Benjamin Scott wrote:

> On Tue, 5 Feb 2002, James Hawtin wrote:
> > I think the reason most people don't do backups is because, hard disk
> > drives are really big and really cheap. Tape drives are really expensive,
> > media is also expensive and frankly tapes are very small for the cost.
>
>   Here are some prices from a recent Datacomm Warehouse catalog:
>
>   Item            GB   Cost  $/GB
>   --------------  ---  ----  ----
>   IDE HDD         100   270  2.70
>   AIT2             50    90  1.80
>   SuperDLT        110   150  1.36
>   LTO Ultrium     100   130  1.30
>   DDS-4            20    24  1.20
>
>   As you can see, hard disk is actually the most expensive media, not the
> least.  This whole "hard disks are cheaper" thing is a myth propagated by
> people who have never actually looked at the numbers.

   Be careful here. There are other possible issues (costs of
the tape drives, etc.). These appear to be just media numbers.
What about backup software costs? For the home user, this
is expensive (however, I agree with your comment below
about CD-RWs for home users).

   Large amounts of data brings in even more issues. How do
you stream Petabytes of data to tape in a resonable of time?
Unless you spend HUGE amounts of money to buy multiple
silos to handle slices of the data and put in a fantastic network
to handle the traffic, you can't backup in a reasonable amount
of time. In this case, disk is your only choice. The data size
break point between disk and tape depends on lots of things,
but that point is moving downward all the time (perhaps in
the Terabyte range).
   Of course, I don't know of any home users with this much
data, but I know lots of companies with this amount of data.

Jeff


>
>
> > Effective backup can double the cost of a system and requires time to
> > manage it.
>
>   Sure it can.  Losing your data will generally cost even more.
>
> > For the "home" market its just to much.
>
>   The home user has maybe, what, 100 MB of data to protect, tops?  You can
> fit that on a $2 CD-RW, for crying out loud.
>
> > This why people don't have backups in my book.
>
>   "Most people" don't have backups because they don't know any better, and
> likely don't care.  They bought their Pee Cee at Wal-Mart, and the most
> valuable thing on there is their collection of illegal music they downloaded
> before Napster went bust.  For those who have legitimately valuable data
> (e.g., financial data), they likely don't realize just how much it is going
> to cost them when (note -- *WHEN*, not *IF*) they lose it.
>
>   Sure, backups cost money.  You might spend $1500 on a data protection
> scheme for a small office server.  Of course, consider how much it would
> cost if they did not do that.  I know business that have *gone out of
> business* when they lost their only copy of mission-critical data.
>
>   $1500 vs bankruptcy.  Gee, tough call.
>
>   Anyone who has ever bothered to work out the numbers knows that backups
> are a bargain.  People who think otherwise are either ignorant or just plain
> don't want to know the truth.
>
> --
> Ben Scott <bscott@ntisys.com>
> | The opinions expressed in this message are those of the author and do not |
> | necessarily represent the views or policy of any other person, entity or  |
> | organization.  All information is provided without warranty of any kind.  |
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[James Mello]

>    Large amounts of data brings in even more issues. How do
> you stream Petabytes of data to tape in a resonable of time?
> Unless you spend HUGE amounts of money to buy multiple
> silos to handle slices of the data and put in a fantastic network
> to handle the traffic, you can't backup in a reasonable amount
> of time. In this case, disk is your only choice. The data size
> break point between disk and tape depends on lots of things,
> but that point is moving downward all the time (perhaps in
> the Terabyte range).

Petabytes are another matter entirely. If you must absolutely have that
much data backed up, you start looking at more exotic solutions. Backing
up solely to disk is *way* to expensive, hence the wonderful, but of
poorly implemented idea of HSM.... Disk and tape :)

	-- Cheers
	-- James

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Anders Widman]

Well, have you ever heard of diskcabbinettes? I use them alot wich means I don't have to have 10 disks in the same coputer at the same time. Then what, hot swap? Well this is very much possible, and secure 
(look at Promise.com).

IDE disk drives are very cheap and fast enough (Maxtor 160GB has a write performance of about 35MB/s in average) for most purposes and does not require special hardware or software.

//Anders

2002-02-06 20:45:42, James Mello <kingjamm@colltech.com> wrote:

>With your argument, were would you stick 10 IDE drives then? Add the cost
>of controllers, etc and you start to add more and more cost. The deal is,
>backups to tape are done to tape when the media cost savings justify the
>outlay of cash for the hardware. Like it was said before, most small
>business could get away with burning a CD. But what you *can't* do is
>ignore the fact that if you want to be safe, you *must* do backups. Disk
>is cool, but make sure that you fully account for the expense in your
>environment.
>
>	-- Cheers
>	-- James
>
>> Ben,
>> 	I think you missed a very important aspect.  The actual investment of
>> the tape drives.  A SDLT or LTO drive is going to run from about $4000
>> on up.  According to datacomm warehouse, a HP 230 external LTO drives
>> runs $3700.  An SDLT drive runs $4300.  You can't compare just the media
>> costs with that of the drive.  Let's consider a 1TB of data.
>> 
>> SDLT:
>> $4300 + (10 tapes * $50) = $4800
>> 
>> LTO:
>> $3700 + (10 tapes * $130) = $5000
>> 
>> AIT2:
>> $2800 + (20 tapes * $90) = $4600
>> 
>> IDE:
>> 10  Drives * 270 = $2700
>> 
>> DDS4:
>> $900 + (50 tapes * $24) = $2100
>> 
>> >From these numbers (using DataComm Warehouse pricing), you can see that
>> to backup 1TB, the IDE drives are actually the second cheapest.  And
>> from a raw performance point of view, they far exceed all the other
>> alternatives from a random access standpoint, and they are comparable to
>> even LTO from a transfer rate point of view.
>> 
>> Sorry... had to address this point.
>> 
>> On Wed, 2002-02-06 at 13:14, Benjamin Scott wrote:
>> >   Here are some prices from a recent Datacomm Warehouse catalog:
>> > 
>> >   Item            GB   Cost  $/GB
>> >   --------------  ---  ----  ----
>> >   IDE HDD         100   270  2.70
>> >   AIT2             50    90  1.80
>> >   SuperDLT        110   150  1.36
>> >   LTO Ultrium     100   130  1.30
>> >   DDS-4            20    24  1.20
>> > 
>> >   As you can see, hard disk is actually the most expensive media, not the
>> > least.  This whole "hard disks are cheaper" thing is a myth propagated by
>> > people who have never actually looked at the numbers.
>> 
>> 
>
>
>_______________________________________________
>linux-lvm mailing list
>linux-lvm@sistina.com
>http://lists.sistina.com/mailman/listinfo/linux-lvm
>read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
>
>

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Austin Gonyou]

[-- Attachment #1: Type: text/plain, Size: 2624 bytes --]

Thing about AIT2 though, you can store your data for 50 years without
degradation. Longer if you take care of it properly(not that you'd need
to anyway, you'll have to re-spin your tapes once in a while)

AIT2 is 50/100, not just 50. Most of the time I can get about 110GB per
tape. AIT3 is 200+ however. :) Not to mention, you can get 80MB/s/drive
if you have 4 dedicated scsi controllers. 

And 80 of them only take up a smallamount of space. 

On Wed, 2002-02-06 at 13:51, Theo Van Dinter wrote:
> On Wed, Feb 06, 2002 at 02:14:05PM -0500, Benjamin Scott wrote:
> >   Here are some prices from a recent Datacomm Warehouse catalog:
> > 
> >   Item            GB   Cost  $/GB
> >   --------------  ---  ----  ----
> >   IDE HDD         100   270  2.70
> >   AIT2             50    90  1.80
> >   SuperDLT        110   150  1.36
> >   LTO Ultrium     100   130  1.30
> >   DDS-4            20    24  1.20
> > 
> >   As you can see, hard disk is actually the most expensive media, not
> the
> > least.  This whole "hard disks are cheaper" thing is a myth propagated
> by
> > people who have never actually looked at the numbers.
> 
> Pot, kettle, black.
> 
> Yes, a AIT2 _TAPE_ is $90, but you need to factor in the drive,
> otherwise
> the media is useless.  A $270 IDE drive is ready to go.  A quick look at
> warehouse.com (to keep the numbers on-par):
> 
> AIT2 drive/Sony		2800 + 90 (tape) / 50GB   = 57.80
> SuperDLT drive/LaCie	5000 + 150 (tape) / 110GB = 46.82
> LTO Ultrium drive/HP	3650 + 130 (tape) / 100GB = 37.80
> DDS-4 drive/Seagate	970 + 24 (tape) / 20GB    = 49.70
> 
> 
> So yes, not surprisingly, tape is more expensive than disk everytime.
> Even if you halfed the prices assuming full (2x) compression on the
> tape. Now over time, the per GB price of tape will come down since you
> only need the large outlay of cash once, but you can only use one tape
> at a time at any given moment.
> 
> 
> -- 
> Randomly Generated Tagline:
> "perl is awesome, perl is great
>   perl gets my programs running without all the wait,
>   perl lets me do my job smooth as silk,
>   it even stays crispy in milk.." - Greg London
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
-- 
Austin Gonyou
Systems Architect, CCNA
Coremetrics, Inc.
Phone: 512-698-7250
email: austin@coremetrics.com

"It is the part of a good shepherd to shear his flock, not to skin it."
Latin Proverb

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 232 bytes --]

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Jeff Layton]

James Mello wrote:

> >    Large amounts of data brings in even more issues. How do
> > you stream Petabytes of data to tape in a resonable of time?
> > Unless you spend HUGE amounts of money to buy multiple
> > silos to handle slices of the data and put in a fantastic network
> > to handle the traffic, you can't backup in a reasonable amount
> > of time. In this case, disk is your only choice. The data size
> > break point between disk and tape depends on lots of things,
> > but that point is moving downward all the time (perhaps in
> > the Terabyte range).
>
> Petabytes are another matter entirely. If you must absolutely have that
> much data backed up, you start looking at more exotic solutions. Backing
> up solely to disk is *way* to expensive, hence the wonderful, but of
> poorly implemented idea of HSM.... Disk and tape :)

Yes, HSM has never panned out the way it was supposed to.
However, try backing up Petabytes with tape in a reasonable
amount of time. While I've never configure such a system,
the numbers from systems I have seen indicate that disk is
much cheaper than tape (even if tape is a feasible solution
because of speed).

Jeff


>
>         -- Cheers
>         -- James
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[James Mello]

> > Petabytes are another matter entirely. If you must absolutely have that
> > much data backed up, you start looking at more exotic solutions. Backing
> > up solely to disk is *way* to expensive, hence the wonderful, but of
> > poorly implemented idea of HSM.... Disk and tape :)
> 
> Yes, HSM has never panned out the way it was supposed to.
> However, try backing up Petabytes with tape in a reasonable
> amount of time. While I've never configure such a system,
> the numbers from systems I have seen indicate that disk is
> much cheaper than tape (even if tape is a feasible solution
> because of speed).

HSM systems that I've seen work generally used some form of nearline disk
storage and an *HUGE* array of tape drives to write too (100+ tape
drives). The data went to near line storage (which they only held for
like 24 hours) and was then written to the tape. The deal is, the more
data you *must* save, the more money you *must* spend.

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Jeff Layton]

James Mello wrote:

> > > Petabytes are another matter entirely. If you must absolutely have that
> > > much data backed up, you start looking at more exotic solutions. Backing
> > > up solely to disk is *way* to expensive, hence the wonderful, but of
> > > poorly implemented idea of HSM.... Disk and tape :)
> >
> > Yes, HSM has never panned out the way it was supposed to.
> > However, try backing up Petabytes with tape in a reasonable
> > amount of time. While I've never configure such a system,
> > the numbers from systems I have seen indicate that disk is
> > much cheaper than tape (even if tape is a feasible solution
> > because of speed).
>
> HSM systems that I've seen work generally used some form of nearline disk
> storage and an *HUGE* array of tape drives to write too (100+ tape
> drives). The data went to near line storage (which they only held for
> like 24 hours) and was then written to the tape. The deal is, the more
> data you *must* save, the more money you *must* spend.

Agreed.

I've never had good luck keeping HSM systems up
for very long. The ones I have experience with liked to lock
up at least once a week and the software never liked to be
restarted - had to reboot the whole box.

Jeff


>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Jesus Manuel NAVARRO LOPEZ]

Hi, Anders:

Anders Widman wrote:

> Well, have you ever heard of diskcabbinettes? I use them alot wich means I don't have to have 10 disks in the same coputer at the same time. Then what, hot swap? Well this is very much possible, and secure 
> (look at Promise.com).
> 
> IDE disk drives are very cheap and fast enough (Maxtor 160GB has a write performance of about 35MB/s in average) for most purposes and does not require special hardware or software.
> 


Well... let's consider all aspects.  I'm a sysadmin the kind of BOFH, so 
late in the evening I usually find myself a bit overloaded on beer. 
Specially on friday, if I have to stay at work past 5PM I have the 
irresistible temptation to go to the closet and piss*1 on the diskcabbinet.

Guess what?  when I do so *all* our backup media (your IDE disks) goes 
nuts at a time.

For a backup policy you *must* take appart the media from the on-line 
data to be protected.  Having all your backup media in a single place is 
*BAAAAAAAD* idea (TM).

Now, start moving from the cabinet to the in-place closet your disks one 
day and another, and to the off-site closet at least once a month, and 
tell me you won't need to replace your disks on a shorter basis than 
your DTLs or DDSs.

*1: Standard dramatization in place.  I *don't* do that (specilly true 
if my boss is reading this article X^D  )
-- 
SALUD,
Jes�s
***
jesus_navarro@promofinarsa.es
***

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Jesus Manuel NAVARRO LOPEZ]

Hi, Jeff:

Jeff Layton wrote:

>>
> 
>    Be careful here. There are other possible issues (costs of
> the tape drives, etc.). These appear to be just media numbers.
> What about backup software costs? For the home user, this
> is expensive (however, I agree with your comment below
> about CD-RWs for home users).
> 


That's plainly untrue: AMANDA for one is free and very good for small 
systems.
-- 
SALUD,
Jes�s
***
jesus_navarro@promofinarsa.es
***

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Jesus Manuel NAVARRO LOPEZ]

Jeff Layton wrote:

> 
> I've never had good luck keeping HSM systems up
> for very long. The ones I have experience with liked to lock
> up at least once a week and the software never liked to be
> restarted - had to reboot the whole box.
> 


Huh? kindof Microsoft guy?
X^P
-- 
SALUD,
Jes�s
***
jesus_navarro@promofinarsa.es
***

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Petro]

On Thu, Feb 07, 2002 at 10:00:45AM +0100, Jesus Manuel NAVARRO LOPEZ wrote:
> Hi, Anders:
> Anders Widman wrote:
> >Well, have you ever heard of diskcabbinettes? I use them alot wich means 
> >I don't have to have 10 disks in the same coputer at the same time. Then 
> >what, hot swap? Well this is very much possible, and secure (look at 
> >Promise.com).
> >IDE disk drives are very cheap and fast enough (Maxtor 160GB has a write 
> >performance of about 35MB/s in average) for most purposes and does not 
> >require special hardware or software.
> Well... let's consider all aspects.  I'm a sysadmin the kind of BOFH, so 
> late in the evening I usually find myself a bit overloaded on beer. 
> Specially on friday, if I have to stay at work past 5PM I have the 
> irresistible temptation to go to the closet and piss*1 on the diskcabbinet.

    Good way to guarentee you won't have kids. 
    
> For a backup policy you *must* take appart the media from the on-line 
> data to be protected.  Having all your backup media in a single place is 
> *BAAAAAAAD* idea (TM).

    Depends on your needs.
    Depends on your needs.
    Depends on your needs.
    Depends on your needs.
    Depends on your needs.
    Depends on your needs.
    Depends on your needs.
    Depends on your needs.
    Depends on your needs.
    Depends on your needs.
    Depends on your needs.
    Depends on your needs.
    Depends on your needs.
    Depends on your needs.

    I have one database that changes fast enough that if it's 36 hours
    old, we're basically just recovering it for the table structures. 

    I've got another one that changes fast enough that it's not worth
    backing up. If it's more than 2 hours old, starting from scratch is
    fine. 

    It the first case, off site backups don't make sense, so we have 2
    backup hosts (seperated by about 10 feet currently, less in a day or
    two) that get backups on an alternating (daily) basis. 

    Oh, this is in addition to these databases being replicated pairs. 

    Again, your backup strategy depends on your needs, your budget, and
    your risk tolerance. 

    It doesn't make sense to spend $10k for a backup solution for $20k of
    data. It does make sense to spend $10k to backup $100k.  

> Now, start moving from the cabinet to the in-place closet your disks one 
> day and another, and to the off-site closet at least once a month, and 
> tell me you won't need to replace your disks on a shorter basis than 
> your DTLs or DDSs.


> 
> *1: Standard dramatization in place.  I *don't* do that (specilly true 
> if my boss is reading this article X^D  )

    No, but a handful of metal filings are (1) more effective, (2) less
    traceble, and (3) less likely to get your pecker arced off. 

-- 
Share and Enjoy. 

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Petro]

On Thu, Feb 07, 2002 at 10:03:12AM +0100, Jesus Manuel NAVARRO LOPEZ wrote:
> That's plainly untrue: AMANDA for one is free and very good for small 
> systems.

    Define "small". 


-- 
Share and Enjoy. 

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Dieter Stueken]

Anders Widman wrote:
> 
> Well, have you ever heard of diskcabbinettes? I use them alot wich means I don't have to have 10 disks in the same coputer at the same time. Then what, hot swap? Well this is very much possible, and secure
> (look at Promise.com).
> 
> IDE disk drives are very cheap and fast enough (Maxtor 160GB has a write performance of about 35MB/s in average) for most purposes and does not require special hardware or software.

I also developed a backup strategy based on a LVM Disk Array and it
works great.
I currently have to backup about 100GB of data. I build a mirroring
system copying
all data each night. Our traditional incremental backup showed, that
about 2Gb are
changed each day. So, for the next day, a new directory is created and
all backuped
data from yesterday are hard linked (cp -al) to the new destination.
Second, a cp -u
updates all files which have been changed. With 140Gb available I can
hold about 2 weeks
on disk. After that only one snapshot per week, later on one per month
is kept. I have
to see, how far I can go. In addition i make full backups from these
snapshots as an
disaster backup.

This procedure has a lot of advantages. Recent backups of single files
can be done
instantaneous. This took me hours of spooling DDS tapes back an forth
before. The backup
system can even be accessed by the users themselves. The backup mirror
is located on
an other building, an both systems see each other read only.

I think about saving the incremental data on tape too, to have a 3rd
stage, if the
backup disk fails. Any tape backup can now be done off-line from the
mirror. I don't
have to spend my weekends changing tapes to realize a fill backup from
the hot system.
I also may continue to use my old DDS4 backup drive, because it does not
really matter
if the tape backup is slow.

Recovering all data from DDS4 tape may take longer than using some fancy
LTO, this will
be necessary only if both buildings burned down, and I thing we have a
bigger problem then...

Dieter.
-- 
Dieter St�ken, con terra GmbH, M�nster
    stueken@conterra.de         stueken@qgp.uni-muenster.de
    http://www.conterra.de/     http://qgp.uni-muenster.de/~stueken
    (0)251-980-2027             (0)251-83-334974

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Jesus Manuel NAVARRO LOPEZ]

Hi, Petro:

Petro wrote:

> On Thu, Feb 07, 2002 at 10:03:12AM +0100, Jesus Manuel NAVARRO LOPEZ wrote:
> 
>>That's plainly untrue: AMANDA for one is free and very good for small 
>>systems.
>>
> 
>     Define "small". 

I'm succesfully using it for 4 servers with two DDS drivers, and I would say it could be easy expandible to the numbers of tens.

It have some "caveats" anyway.  Rigth now it can't manage multivolume 
quite efficiently (a single dump must go to single storage device, for 
instance, and the multichanger specifications is a bit... "raw". 
Management for automatic tape cabinets is out of the system too.  It 
uses an rsh-like client/server protocol to remote access systems so it's 
not soo secure -while it can be ssh-tunneled -not out-of-the-box.  It 
doesn't support multiserial backups on a single config -but you can have 
multiple definitions so off-site monthly backups don't interfere with 
your monday-to-friday daily series, for instance).  What it does pretty 
well is automatically balance the backup levels for all your partitions 
to optimize tape space and indexing your tapes (both semi-automatically 
indexing tapes and maintain the index about what was where... it can 
even help you the physical tape-labels, so you just print what amanda 
tells you to, and have some -primitive, ability to create graphic logs 
to help you planning capacity in advance).  It will e-mail you too if 
you forget to put the tape in the drive too.
Appart from this, it uses underlying backup tools (dump by default, tar 
or smbtar if you choose it) so you can restore even without amanda 
software present (while it's more difficult: you just don't have a tape 
labelled like "first-monday-of-the-month-so-level0-backups-here", but 
only something like "Comm-dpt-tape-no12": if you loose your indexes you 
will have to look at the labels for the dumps to guess what's in-there. 
  Anyway, those indexes are simple text files so they're easy to copy to 
another box in advance, and they're sent day-to-day by e-mail to the 
backup operator, so you do have options to avoid that situation).
-- 
SALUD,
Jes�s
***
jesus_navarro@promofinarsa.es
***

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Jesus Manuel NAVARRO LOPEZ]

Hi, Petro:

Petro wrote:

> On Thu, Feb 07, 2002 at 10:00:45AM +0100, Jesus Manuel NAVARRO LOPEZ wrote:


[...]

>>Well... let's consider all aspects.  I'm a sysadmin the kind of BOFH, so 
>>late in the evening I usually find myself a bit overloaded on beer. 
>>Specially on friday, if I have to stay at work past 5PM I have the 
>>irresistible temptation to go to the closet and piss*1 on the diskcabbinet.
>>
> 
>     Good way to guarentee you won't have kids. 
>


Fair enough.  I *don't* have childrens... but I tend to consider my PFY 
like a bastard of my bastardness, does it counts?

     
> 
>>For a backup policy you *must* take appart the media from the on-line 
>>data to be protected.  Having all your backup media in a single place is 
>>*BAAAAAAAD* idea (TM).


> 
>     Depends on your needs.


Yes: depends on my needs: If I need to recover any data, having *all*
your backup media in a single place is *BAAAAAAAD* idea (TM).  If this 
place is the same (or near to) the place where the production data is 
stored is simply unadjectivable.
If I don't need to recover data, I can go without any backup at all.


>     Depends on your needs.
>     Depends on your needs.
>     Depends on your needs.
>     Depends on your needs.
>     Depends on your needs.
>     Depends on your needs.
>     Depends on your needs.
>     Depends on your needs.
>     Depends on your needs.
>     Depends on your needs.
>     Depends on your needs.
>     Depends on your needs.
>     Depends on your needs.
> 
>     I have one database that changes fast enough that if it's 36 hours
>     old, we're basically just recovering it for the table structures. 
> 


In case of disaster, if your backup media is in the proximities of your 
production database (define "proximities" as needed) you won't be able 
to recover table structures.  One thing is what I say, and a *completely 
different* issue is to decide *what* to backup, and how, not where.


>     I've got another one that changes fast enough that it's not worth
>     backing up. If it's more than 2 hours old, starting from scratch is
>     fine. 
> 


So, the amount of change by unit time is your key to decide what you 
*need* to backup and what you don't?
Sound *extremly* odd to me.  I would say it should be *the value* of the 
material (this include the cost to recreate it anew too, obviously), not 
its change rate.


>     It the first case, off site backups don't make sense, so we have 2
>     backup hosts (seperated by about 10 feet currently, less in a day or
>     two) that get backups on an alternating (daily) basis. 
> 


They won't make sense deppending on its *cost*, not its change rate.


>     Oh, this is in addition to these databases being replicated pairs. 
> 


This doesn't add nothing to value your backup strategy.  Again the value 
is the key.  You have replicated in pairs, do you?  What then in case of 
fire on you operations center?  I'd bet you'll loose both of your servers.


>     Again, your backup strategy depends on your needs, your budget, and
>     your risk tolerance. 
>
>     It doesn't make sense to spend $10k for a backup solution for $20k of
>     data. It does make sense to spend $10k to backup $100k.  
> 


Of course.  Your backup strategy surely deppends... *on the data value* 
and only on this.  From the very beginning I stated that for "home data" 
just burning some CDs from time to time, or no backup at all can be good 
enough (but again, even burning some CDs has no sense in case you 
want/need your data even if your "PC-room" burns out, if you have your 
CDs on the same table the PC is).


> 
>>*1: Standard dramatization in place.  I *don't* do that (specilly true 
>>if my boss is reading this article X^D  )
>>
> 
>     No, but a handful of metal filings are (1) more effective, (2) less
>     traceble, and (3) less likely to get your pecker arced off. 
 

Yep.  But I don't find it quite as funny

-- 
SALUD,
Jes�s
***
jesus_navarro@promofinarsa.es
***

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Jeff Layton]

Jesus Manuel NAVARRO LOPEZ wrote:

> Jeff Layton wrote:
>
> >
> > I've never had good luck keeping HSM systems up
> > for very long. The ones I have experience with liked to lock
> > up at least once a week and the software never liked to be
> > restarted - had to reboot the whole box.
> >
>
> Huh? kindof Microsoft guy?

Nope. HPUX.


>
> X^P
> --
> SALUD,
> Jesús
> ***
> jesus_navarro@promofinarsa.es
> ***
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[James Hawtin]

Meep, What I thread I started. I did orginally have an LVM related point,
but I kind of failed to include it and sent a rant insted ;-) Anyway my
point was that backups are slow and expensive and not everyone has them.
Many of the commands in lvm say "make sure you do a backup first" This is
not necessarily possible. I personally believe lvm should not in its self
FSCK period. Personally I have suffered with 1.0.1 release on a 2.4.17
system that would crash on vgremove operations, I also could not vgreduce 
my volume group etc (This was on an SMP system). I got damn good at
working arround it using reboots, dd and multiple disk... But dang I
should not have to. I know that problem is fixed in the CVS version that I
am currently using and now there (I believe there is a newer version). I
am sure I am not alone in not liking to run CVS version and prefer snap
shot releases which have identifiable version numbers. It took months for
1.0.2 to be released, then shortly after I hear there is LVM 2. As I said
I believe LVM should not fsck up, why was no time invested in doing a
1.0.1.1 release.

I beleive LVM operations should be undoable, unless, someone has rebuilt a
filesystem over the top of old data, it should not matter what you do to
the lvm in the vertual layout the orginal stats should be EASILY
restoreable. I am not talking about playting hunt the .conf file, An undo
should be included.

The time undo would come is most useful is for resize operations, at the
moment you have to resize the underlying filesystem, then strink the the
lvm, however if you strink the lvm too much your in problems. I was told
if you expand it again and lvm picks the same blocks to use in the same
order everything would be ok... Fine great.. why not just allow an undo so
you can aways get the right blocks.

The reasons resize scares me, is all in needs is someone to define MB or
GB differently and your up S..T creek, is the block could 1k or 512
bytes... Was that M option MiB or MB most people say MB whent hey mean
MiB. Was there and "overhead" of a few blocks magically required when i
asked for a 1GiB filesystem... etc.

James

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Petro]

On Thu, Feb 07, 2002 at 11:34:08AM +0100, Jesus Manuel NAVARRO LOPEZ wrote:
> Hi, Petro:
> Petro wrote:
> >On Thu, Feb 07, 2002 at 10:00:45AM +0100, Jesus Manuel NAVARRO LOPEZ wrote:
> [...]
> >>Well... let's consider all aspects.  I'm a sysadmin the kind of BOFH, so 
> >>late in the evening I usually find myself a bit overloaded on beer. 
> >>Specially on friday, if I have to stay at work past 5PM I have the 
> >>irresistible temptation to go to the closet and piss*1 on the 
> >>diskcabbinet.
> >    Good way to guarentee you won't have kids. 
> Fair enough.  I *don't* have childrens... but I tend to consider my PFY 
> like a bastard of my bastardness, does it counts?
    
    That was intended to be humorous. Urinating on live electrical
    components tends to be a shocking experience.

> >>For a backup policy you *must* take appart the media from the on-line 
> >>data to be protected.  Having all your backup media in a single place is 
> >>*BAAAAAAAD* idea (TM).
> >    Depends on your needs.
> Yes: depends on my needs: If I need to recover any data, having *all*
> your backup media in a single place is *BAAAAAAAD* idea (TM).  If this 

    No, when you need to recover stuff, it's a *great* idea to have it
    in one place.

    When you suffer an environmental calamity, it's a bad idea. 

> place is the same (or near to) the place where the production data is 
> stored is simply unadjectivable.

    Ever tried to push tens of gigabytes over a WAN? 

> >    Depends on your needs.
> >    I have one database that changes fast enough that if it's 36 hours
> >    old, we're basically just recovering it for the table structures. 
> In case of disaster, if your backup media is in the proximities of your 
> production database (define "proximities" as needed) you won't be able 
> to recover table structures.  One thing is what I say, and a *completely 
> different* issue is to decide *what* to backup, and how, not where.

    That wasn't my point. 

    My point was that for some stuff, 36 hour old data is useless, and
    even a normal tape rotation schedule can put data out of reach for
    10 or 12 hours minimum. 

> >    I've got another one that changes fast enough that it's not worth
> >    backing up. If it's more than 2 hours old, starting from scratch is
> So, the amount of change by unit time is your key to decide what you 
> *need* to backup and what you don't?
> Sound *extremly* odd to me.  I would say it should be *the value* of the 
> material (this include the cost to recreate it anew too, obviously), not 
> its change rate.

    Not the necessity of backing up, but the cost. If the data is
    changing that fast, it could easily be that by the time it's on
    tape, it's out of date and effectively useless. 

> >    It the first case, off site backups don't make sense, so we have 2
> >    backup hosts (seperated by about 10 feet currently, less in a day or
> >    two) that get backups on an alternating (daily) basis. 
> They won't make sense deppending on its *cost*, not its change rate.

    No, it would be a lot cheaper to dump the dbs to tape, and carry the
    tapes offsite, but (1) recovery time is almost tripled, and (2) 
 
> >    Oh, this is in addition to these databases being replicated pairs. 
> 
> This doesn't add nothing to value your backup strategy.  Again the value 
> is the key.  You have replicated in pairs, do you?  What then in case of 
> fire on you operations center?  I'd bet you'll loose both of your servers.

    In case of fire in the colo, we lose the entire site. Then there is
    nothing to recover the data onto. 

    And yes, we know the problems with this. It's a calculated risk. We
    can't afford geographically seperated facilities right now. 

> >    Again, your backup strategy depends on your needs, your budget, and
> >    your risk tolerance. 
> >    It doesn't make sense to spend $10k for a backup solution for $20k of
> >    data. It does make sense to spend $10k to backup $100k.  
> 
> Of course.  Your backup strategy surely deppends... *on the data value* 
> and only on this.  From the very beginning I stated that for "home data" 

    No, it doesn't. It depend on several things:

    (1) Value of data. 
    (2) Cost of downtime. 
    (3) Rate of change. (If your data set is completely worthless after
    24 hours, but worth several million for the first hour, offsite
    backups don't necessarily make sense etc.) 

    And probably some more I haven't thought of. 

-- 
Share and Enjoy. 

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Jesus Manuel NAVARRO LOPEZ]

Petro wrote:

> On Thu, Feb 07, 2002 at 11:34:08AM +0100, Jesus Manuel NAVARRO LOPEZ wrote:
> 
>>Hi, Petro:
>>Petro wrote:
>>
>>>On Thu, Feb 07, 2002 at 10:00:45AM +0100, Jesus Manuel NAVARRO LOPEZ wrote:
>>>
>>[...]
>>
>>>>Well... let's consider all aspects.  I'm a sysadmin the kind of BOFH, so 
>>>>late in the evening I usually find myself a bit overloaded on beer. 
>>>>Specially on friday, if I have to stay at work past 5PM I have the 
>>>>irresistible temptation to go to the closet and piss*1 on the 
>>>>diskcabbinet.
>>>>
>>>   Good way to guarentee you won't have kids. 
>>>
>>Fair enough.  I *don't* have childrens... but I tend to consider my PFY 
>>like a bastard of my bastardness, does it counts?
>>
>     
>     That was intended to be humorous. Urinating on live electrical
>     components tends to be a shocking experience.
> 


Yep.  Me too.  I neither have childs nor PFY, so it really doesn't matter.


> 
>>>>For a backup policy you *must* take appart the media from the on-line 
>>>>data to be protected.  Having all your backup media in a single place is 
>>>>*BAAAAAAAD* idea (TM).
>>>>
>>>   Depends on your needs.
>>>
>>Yes: depends on my needs: If I need to recover any data, having *all*
>>your backup media in a single place is *BAAAAAAAD* idea (TM).  If this 
>>
> 
>     No, when you need to recover stuff, it's a *great* idea to have it
>     in one place.
> 


Well, but this is not what I said.  I purpousely bolded *all*, for 
that's the key.  It's a great idea have *some* backups at hand and 
grouped (if only for the chance of your boss asking you for recovering 
that porn... err... that paper he accidentaly deleted).  But not *all* 
your media.  Obviously is not too operative to have all my backup media 
in Dallas (I'm in Spain so it's not cost-effective).


>     When you suffer an environmental calamity, it's a bad idea. 
> 
> 
>>place is the same (or near to) the place where the production data is 
>>stored is simply unadjectivable.
>>
> 
>     Ever tried to push tens of gigabytes over a WAN? 
> 


Ever heared about "Never understimate the bandwith of a wagon full of 
tapes"?


> 
>>>   Depends on your needs.
>>>   I have one database that changes fast enough that if it's 36 hours
>>>   old, we're basically just recovering it for the table structures. 
>>>
>>In case of disaster, if your backup media is in the proximities of your 
>>production database (define "proximities" as needed) you won't be able 
>>to recover table structures.  One thing is what I say, and a *completely 
>>different* issue is to decide *what* to backup, and how, not where.
>>
> 
>     That wasn't my point. 
> 
>     My point was that for some stuff, 36 hour old data is useless, and


Then its *value* past 36 hours is... nihil.  You told (implictly) that 
your database schema was of some value even if they were more than 36 
hours old.


>     even a normal tape rotation schedule can put data out of reach for
>     10 or 12 hours minimum. 
> 


Yes, that's *potentially* true.


> 
>>>   I've got another one that changes fast enough that it's not worth
>>>   backing up. If it's more than 2 hours old, starting from scratch is
>>>
>>So, the amount of change by unit time is your key to decide what you 
>>*need* to backup and what you don't?
>>Sound *extremly* odd to me.  I would say it should be *the value* of the 
>>material (this include the cost to recreate it anew too, obviously), not 
>>its change rate.
>>
> 
>     Not the necessity of backing up, but the cost. If the data is
>     changing that fast, it could easily be that by the time it's on
>     tape, it's out of date and effectively useless. 
> 


Again: it's not its change rate but its *value*.  The more it values, 
the more you can expense to "insure" it (part of your insurance policy 
talks about *within a time frame*).


> 
>>>   It the first case, off site backups don't make sense, so we have 2
>>>   backup hosts (seperated by about 10 feet currently, less in a day or
>>>   two) that get backups on an alternating (daily) basis. 
>>>
>>They won't make sense deppending on its *cost*, not its change rate.
>>
> 
>     No, it would be a lot cheaper to dump the dbs to tape, and carry the
>     tapes offsite, but (1) recovery time is almost tripled, and (2) 
>  


...and its *value* once recovered will be lower than having no data at 
all.  Again, *if* you manage to find a method so the value of the 
recovered data is higher than the costs of having that method in place, 
your job (if that's your job, of course) is point it out and implement it.


> 
>     And yes, we know the problems with this. It's a calculated risk. We
>     can't afford geographically seperated facilities right now. 
> 


*Value* again.  And about it, I recently knew about a multinational 
company (so it were not only a one-site company) which main office was 
at the twin towers.  It would be able to restore from the people death 
(though *many* of upper management died) but it didn't from the 
data/facilities loss.


> 
>>>   Again, your backup strategy depends on your needs, your budget, and
>>>   your risk tolerance.


It only deppends on your needs.  Your needs can include not surpassing 
certain budget amount, but definetly it hasn't nothing to do with "risk 
tolerance".  "Risk tolerance" is either a winner bet or a misinformation 
issue.

 
>>>   It doesn't make sense to spend $10k for a backup solution for $20k of
>>>   data. It does make sense to spend $10k to backup $100k.  
>>>


Plainly true... except for the last value: it makes sense to spend $X at 
most to backup $Y*p, where p is the probability of loosing that data 
(ie. if the probability is 1, so you're certain to loss the data, you 
can expend up to $100k to insure it -within the time frame that data 
produces $100k revenues).


>>Of course.  Your backup strategy surely deppends... *on the data value* 
>>and only on this.  From the very beginning I stated that for "home data" 
>>
> 
>     No, it doesn't. It depend on several things:
> 
>     (1) Value of data.


Plain data value

 
>     (2) Cost of downtime.


Data value too (in terms of lost revenue for the time the data is not 
accesible).

 
>     (3) Rate of change. (If your data set is completely worthless after
>     24 hours, but worth several million for the first hour, offsite
>     backups don't necessarily make sense etc.) 
> 


Data value too (in terms on how the value of data evolutions with time).


>     And probably some more I haven't thought of. 
> 


Probably: and they all will be expresable in terms of data value or will 
have no significance at all.
-- 
SALUD,
Jes�s
***
jesus_navarro@promofinarsa.es
***

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Wolfgang Weisselberg]

Hi, James!

James Hawtin (oolon@ankh.org) wrote 45 lines:

> The reasons resize scares me, is all in needs is someone to define MB or
> GB differently and your up S..T creek, is the block could 1k or 512
> bytes... Was that M option MiB or MB most people say MB whent hey mean
> MiB. Was there and "overhead" of a few blocks magically required when i
> asked for a 1GiB filesystem... etc.

If you are using ext2 or ext3, you might want to look at
e2fsadm...

-Wolfgang

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[James Hawtin]

On Thu, 7 Feb 2002, Wolfgang Weisselberg wrote:

> If you are using ext2 or ext3, you might want to look at
> e2fsadm...
> 
> -Wolfgang
> 

I am running reiserfs as it seems to find my needs.

James

Re: [linux-lvm] Backup costs (was: LVM reimplementationre)

[Marc MERLIN]

On Thu, Feb 07, 2002 at 10:03:12AM +0100, Jesus Manuel NAVARRO LOPEZ wrote:
> That's plainly untrue: AMANDA for one is free and very good for small 
> systems.

We use amanda at VA Software, and  also for sourceforge.net, but while it is
nice as long  as your backup sets  are smaller than half your  tape size, it
starts to choke  a bit when you  have backups that start  reaching your tape
size, and it fails miserably if any backup takes more than your tape.

It is disapointing  that in the last 3 years,  that major shortcoming hasn't
effectively been fixed, but  eh, it's free software, I get to  fix it if I'm
not happy :-)

As for autoloaders  and tape libraries, you  can make it work  with mtx, but
expect to have  to write some glue  and to go through non  trivial trial and
error.

Marc
-- 
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking
  
Home page: http://marc.merlins.org/   |   Finger marc_f@merlins.org for PGP key