* RE: [linux-lvm] snapshot questions
@ 2001-11-02 15:58 Kenny Gorman
2001-11-02 17:04 ` Kyle Hayes
0 siblings, 1 reply; 18+ messages in thread
From: Kenny Gorman @ 2001-11-02 15:58 UTC (permalink / raw)
To: linux-lvm
I will try to quicky summarize what _I_ do:
- Yes snapshots work.
- No, it's not a 1 to 1 disk usage ratio between the vg and the snapshot
vol. It's much less. I dont know how to test for fullness using
Linux/LVM yet.
- I use Oracle, your DB may be different.
1) put the tablespaces in hot backup mode
2) take a snapshot (mount the snapshot vol) of the volume/s that the
tablespace is on
3) take the db out of hot backup mode.
- I also do some checkpointing, etc in my script.
- the tape drive can then get the backup from the snapshot.
- You would restore the image from tape, to a new or the orig. volume
when you recover.
- no noticable effect on end users, it's a 24/7 thing. Restores work
fine.
It's really not all that scary. Sorry for not going into more technical
detail.
Kenny Gorman
DBA
Kenamea
-----Original Message-----
From: Kyle Hayes
Sent: Fri 11/2/2001 1:41 PM
To: linux-lvm@sistina.com
Cc:
Subject: Re: [linux-lvm] snapshot questions
Hmm, seems to be quite a range of opinion. However, I must say that
having so many negative options is not a good sign. Do snapshots
work at all?
If I get a corrupted filesystem in my snapshot, then it doesn't work.
If I have to bend over backwards and change all my apps, then it does
not work.
It is good to hear that it works more like Veritas and less like the
SuSE whitepaper claims. That paper claimed that the changes will go
to the snapshot partition. This implied that there would be a terrible
IO hit when you deleted the snapshot partition and the changes need to
be written back to the original partition.
I am confused by the claim (see below) that you need the same amount
of space for the snapshot partition as you had on the original LV.
From what I understood in the HOWTOs and the whitepapers, you only need
as much space as you will have changes during the lifetime of the
snapshot LV. And, if it works like Veritas, that implies that once
a PE was copied, any further data changes to it would not be copied.
This means that in situations like database systems, you'd actually
need less than the number of PEs that changed. Is this correct?
Under what circumstances do snapshots work? I am confused by the
claims that a snapshot partition is going to be "corrupt" as far as
the OS is concerned. In my scenario, here is what I am planning on
doing:
1) lock all tables against writes.
2) flush all logs and tables to disk.
3) create a snapshot partition for the filesystem (ext2 or probably)
on which the database data resides. These are not raw disks.
4) unlock the tables.
5) mount the snapshot partition and copy the database data out.
From what I have guessed so far (guesses, which makes me nervous),
if I make sure all changes are flushed to disk, then I am OK. However,
the responses I have received are not clear on what conditions caused
the problems.
I am also not clear on which kernel and what patches I need to make
sure that the filesystems (ext2,ext3 and reiserfs) will deal with the
VFS patches.
We run 24/7 systems and it is not an option to take down a system to
back it up. We can handle slightly degraded performance, but even
that is a problem.
On Friday 02 November 2001 12:03, Brent Harding wrote:
> The only real issue with snapshots is needing a partition reserved for
> it, as you get more and more data, this gets difficult, once half the
> hard disk is full, you can't copy any more. I'd say it sounds easier
to
> just use whatever backup program with cron to execute late at night,
> maybe after midnight, depends how far backup is when a file changes,
> it'll get included if it wasn't backed up yet, otherwise if you change
> something the system already backed up, it gets done on the next run.
This is not what I understood from the HOWTO's and the SuSE whitepaper.
They are quite clear that you only need enough space to contain the
changes that occur while the snapshot partition exists.
Best,
Kyle
--
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MicroTelco Services saves money on every Fax:
- Fax to email (FREE)
- Fax to PSTN based Fax (Up to 95% Savings)
- Fax Broadcasting: Send 100s of faxes to fax machines
and email addresses in the time it takes to send just one!
===========================================================
So send a fax today and let us know what you think!
For more info. visit: www.internetfaxjack.com
===========================================================
_______________________________________________
linux-lvm mailing list
linux-lvm@sistina.com
http://lists.sistina.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [linux-lvm] snapshot questions
2001-11-02 15:58 [linux-lvm] snapshot questions Kenny Gorman
@ 2001-11-02 17:04 ` Kyle Hayes
0 siblings, 0 replies; 18+ messages in thread
From: Kyle Hayes @ 2001-11-02 17:04 UTC (permalink / raw)
To: linux-lvm, Kenny Gorman
On Friday 02 November 2001 13:59, Kenny Gorman wrote:
> I will try to quicky summarize what _I_ do:
>
> - Yes snapshots work.
> - No, it's not a 1 to 1 disk usage ratio between the vg and the snapshot
> vol. It's much less. I dont know how to test for fullness using
> Linux/LVM yet.
> - I use Oracle, your DB may be different.
> 1) put the tablespaces in hot backup mode
> 2) take a snapshot (mount the snapshot vol) of the volume/s that the
> tablespace is on
> 3) take the db out of hot backup mode.
I use MySQL in a mode that uses files in the filesystem. Does your
Oracle installation use raw disk or files? If it is raw disk, then
I probably need to do something that will flush disk changes somehow.
I am hoping that the VFS hooks will make the filesystem flush its
buffers to disk so that I don't get a snapshot full of garbage. I
suppose that Oracle will flush all buffers to disk (thus making a
coherent disk image) when you put it into hot backup mode.
When I tell MySQL to flush its tables, it simply writes out everything it
can, closes and reopens the files. Thus it is up to the filesystem/OS to
get the data to the disk in a reasonable fashion.
Interestingly, this may imply (and this seems to be in line with some of
the responses I read) that journaling filesystems may actually be more
of a problem than non-journaling ones. Weird, but seems to make sense.
> - I also do some checkpointing, etc in my script.
We keep live remote replicas, but sometimes the replica needs to be
rebuild. In that case, it is very important to have a means of getting
a completely coherent database snapshot. Right now, this process is very
carefully done with lots of highly database specific code. Ick. It is a
pain to maintain and has a lot of edge cases that we find from time to
time :-) We'd like something a little simpler and more complete.
> - the tape drive can then get the backup from the snapshot.
In our case, we spool it off remotely, but I get the idea.
> - You would restore the image from tape, to a new or the orig. volume
> when you recover.
> - no noticable effect on end users, it's a 24/7 thing. Restores work
> fine.
So you have this in production?
> It's really not all that scary. Sorry for not going into more technical
> detail.
No problem. I am still figuring out how to deal with LVM. It looks like
the solution to all our problems, but I want to make sure before we spend
more time on it. Our existing solution works, but it costly to maintain.
Best,
Kyle
--
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MicroTelco Services saves money on every Fax:
- Fax to email (FREE)
- Fax to PSTN based Fax (Up to 95% Savings)
- Fax Broadcasting: Send 100s of faxes to fax machines
and email addresses in the time it takes to send just one!
===========================================================
So send a fax today and let us know what you think!
For more info. visit: www.internetfaxjack.com
===========================================================
^ permalink raw reply [flat|nested] 18+ messages in thread
* [linux-lvm] Snapshot questions
@ 2007-05-06 5:32 Ed Martin
0 siblings, 0 replies; 18+ messages in thread
From: Ed Martin @ 2007-05-06 5:32 UTC (permalink / raw)
To: linux-lvm
Hi,
I have been looking into doing snapshots with LVM and i have found the
documentation is very light on how they work. Sorry for all the
questions, they are probably very easy and i realize its a lot but the
documentation says almost nothing about this and i would really like to
know. Even just pointing me to something that answers this would be
nice. So far all i have found is tldp.org and that has about 1 page on
snapshots and there are about 100 other sites listing the same text
(almost word for word) So I'm going to ask my questions here and
hopefully get an idea on how it all works.
First what exactly is the size of a snapshot used for? Is the the total
difference (changed data) between the snapshot and the active LV that it
was created against? What happens if i have multiple snapshots? are the
snapshots changed data shared, so if i have LV1, make snap1 from LV1,
then change LV1 it uses space in snap1 correct? what if i then add snap2
created from LV1, does modifying LV1 remove free space from snap1 and
snap2, or just snap2?
If snapshots only use the space that has been changed between it and the
LV it was created from then does that imply that a new snapshot uses
zero space on the VG (if i made its size 1GB) and grow as data is
changed, or does it use 1GB of space on the VG immediately, basically i
want to know if the specified size of the snapshots is allowed to sum to
more then the space on the VG (so if the VG is 200GB, and i put a 100GB
LV on it, am i limited to 10 10GB snaps (to sum to the 100GB), or can i
put 10 50GB snaps on it (which would probably only have a very small
percent of usage) and let then get auto-deleted as space runs out?
I saw that snapshots are deleted when the LV is full, does this mean
when the snapshot differers by more then the changes made since it was
created? if so what snapshot gets deleted? is it just the one that ran
out of space? what if there was a snapshot that came before it that has
not run out of space?
What about fragmentation, does LVM handle 50 snapshots of a single VG
(against a single LV), or will that cause massive fragmentation? does
LVM have problems with high numbers of snapshots, or is 50+ considered
normal?
And for the last question, I'm building a backup server, right now I'm
taking a RAID, putting LVM on it, and I'm thinking about taking weekly
snapshots of the backup data to allow me to rollback (for as far back as
the disk can take me). does this sound like a good idea, or should i try
something else? what is bad about this plan if its a bad idea? do you
have any other recommendations to do something like that? the backup
server is going to be very low load too with all disk idle almost all
the time except when performing a backup (for a few hours every night)
Thanks in advance for answering any of these questions
--Ed Martin
^ permalink raw reply [flat|nested] 18+ messages in thread
* RE: [linux-lvm] snapshot questions
@ 2001-11-02 17:49 Kenny Gorman
2001-11-05 10:31 ` Jesus Manuel NAVARRO LOPEZ
0 siblings, 1 reply; 18+ messages in thread
From: Kenny Gorman @ 2001-11-02 17:49 UTC (permalink / raw)
To: khayes, linux-lvm
I use cooked file systems for Oracle not RAW. I am not sure MySQL has
an similar function to Oracle Hot Backup. Basically, oracle suspends
writes to the datafiles for the tablespaces in Hot Backup by keeping the
whole blocks in the redo logs. Since MySQL (to my knowledge) doesn't do
something like this, your datafiles will get corrupted when you
snapshot. I am not sure the 'flush to disk' idea will work... unless
(and maybe your saying this and I am not getting it) MySQL will suspend
writes until your snapshot is done. I suppose that would work...
In Oracle, the files that are part of the snapshot are not consistent
(corrupted). Thats just how it works. Then when you use Oracle to
recover, it puts together the coherent database image using the System
Change Number and the blocks in the redo logs (recovery structures) and
archive logs. It's pretty advanced in this regard.
We use Veritas in production with Sun. I am evaluating using Linux in
production. I am using a SuSE Linux box in production for another
project, and it's stable. Plan to start using snapshots soon in both
environments if the evaluation works out. Currently snapshots are in
use on testing/dev boxes.
-kg
-----Original Message-----
From: Kyle Hayes
Sent: Fri 11/2/2001 3:05 PM
To: linux-lvm@sistina.com; Kenny Gorman
Cc:
Subject: Re: [linux-lvm] snapshot questions
On Friday 02 November 2001 13:59, Kenny Gorman wrote:
> I will try to quicky summarize what _I_ do:
>
> - Yes snapshots work.
> - No, it's not a 1 to 1 disk usage ratio between the vg and the
snapshot
> vol. It's much less. I dont know how to test for fullness using
> Linux/LVM yet.
> - I use Oracle, your DB may be different.
> 1) put the tablespaces in hot backup mode
> 2) take a snapshot (mount the snapshot vol) of the volume/s that the
> tablespace is on
> 3) take the db out of hot backup mode.
I use MySQL in a mode that uses files in the filesystem. Does your
Oracle installation use raw disk or files? If it is raw disk, then
I probably need to do something that will flush disk changes somehow.
I am hoping that the VFS hooks will make the filesystem flush its
buffers to disk so that I don't get a snapshot full of garbage. I
suppose that Oracle will flush all buffers to disk (thus making a
coherent disk image) when you put it into hot backup mode.
When I tell MySQL to flush its tables, it simply writes out everything
it
can, closes and reopens the files. Thus it is up to the filesystem/OS
to
get the data to the disk in a reasonable fashion.
Interestingly, this may imply (and this seems to be in line with some of
the responses I read) that journaling filesystems may actually be more
of a problem than non-journaling ones. Weird, but seems to make sense.
> - I also do some checkpointing, etc in my script.
We keep live remote replicas, but sometimes the replica needs to be
rebuild. In that case, it is very important to have a means of getting
a completely coherent database snapshot. Right now, this process is
very
carefully done with lots of highly database specific code. Ick. It is
a
pain to maintain and has a lot of edge cases that we find from time to
time :-) We'd like something a little simpler and more complete.
> - the tape drive can then get the backup from the snapshot.
In our case, we spool it off remotely, but I get the idea.
> - You would restore the image from tape, to a new or the orig. volume
> when you recover.
> - no noticable effect on end users, it's a 24/7 thing. Restores work
> fine.
So you have this in production?
> It's really not all that scary. Sorry for not going into more
technical
> detail.
No problem. I am still figuring out how to deal with LVM. It looks
like
the solution to all our problems, but I want to make sure before we
spend
more time on it. Our existing solution works, but it costly to
maintain.
Best,
Kyle
--
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MicroTelco Services saves money on every Fax:
- Fax to email (FREE)
- Fax to PSTN based Fax (Up to 95% Savings)
- Fax Broadcasting: Send 100s of faxes to fax machines
and email addresses in the time it takes to send just one!
===========================================================
So send a fax today and let us know what you think!
For more info. visit: www.internetfaxjack.com
===========================================================
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [linux-lvm] snapshot questions
2001-11-02 17:49 [linux-lvm] snapshot questions Kenny Gorman
@ 2001-11-05 10:31 ` Jesus Manuel NAVARRO LOPEZ
0 siblings, 0 replies; 18+ messages in thread
From: Jesus Manuel NAVARRO LOPEZ @ 2001-11-05 10:31 UTC (permalink / raw)
To: linux-lvm
Hi, Kenny:
Kenny Gorman wrote:
>
> I use cooked file systems for Oracle not RAW. I am not sure MySQL has
> an similar function to Oracle Hot Backup. Basically, oracle suspends
> writes to the datafiles for the tablespaces in Hot Backup by keeping the
> whole blocks in the redo logs. Since MySQL (to my knowledge) doesn't do
> something like this, your datafiles will get corrupted when you
> snapshot. I am not sure the 'flush to disk' idea will work... unless
> (and maybe your saying this and I am not getting it) MySQL will suspend
> writes until your snapshot is done. I suppose that would work...
>
It can do it... at a cost: While Oracle will still accept clients to
write data (but it'll end up at the redo logs instead to the "real"
database), when you block for writing you clients they simply won't
write, which it can turn on an error at the client app side, or a freeze
till the lock goes away.
...but there's another snapshot-like approach you can take with MySQL:
hot replication. Under these circumnstances a "live" database will
create an "insert log" on one hand, and will sustain an open pipe to a
read-only copy (which it can be at other machine... only caveat is that
the conection pipe should be wide enough to cope with traffic
generated). The copy will in turn be actualized reading that insert log
(the insert log is only a binary-format for all operations that alter
the database, like insert, update, create table, delete table, etc.).
Thus you can programatically break the mirror in the moment you desire
and backup the backup database the way that best fit (and you can even
stop the engine: once you break the mirror it'll be "death" for
practical purpouses). Once the backup is done you redo the mirror and
your copy will "automagically" run the log till current state. In order
to restore you will need to restore from last backup and rerun the log
(you should examine it, of course, for post-mortem diagnostic purpouses)
from the backup checkpoint to-date (well, that's not magic on current
MySQL stable version: you will need to purge the update log to run it
from the backup check point, while you can avoid it if proper database
schema is in place so you can guarantee that a duplicated insertion will
fail).
--
SALUD,
Jes�s
***
jesus_navarro@promofinarsa.es
***
^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <D880FFC9CFDBA84FA13E9B166D06A123B647FB@daedalus.kenamea.co m>]
* RE: [linux-lvm] snapshot questions
[not found] <D880FFC9CFDBA84FA13E9B166D06A123B647FB@daedalus.kenamea.co m>
@ 2001-11-02 13:59 ` Brent Harding
2001-11-02 15:40 ` Kyle Hayes
0 siblings, 1 reply; 18+ messages in thread
From: Brent Harding @ 2001-11-02 13:59 UTC (permalink / raw)
To: linux-lvm
The only real issue with snapshots is needing a partition reserved for it,
as you get more and more data, this gets difficult, once half the hard disk
is full, you can't copy any more. I'd say it sounds easier to just use
whatever backup program with cron to execute late at night, maybe after
midnight, depends how far backup is when a file changes, it'll get included
if it wasn't backed up yet, otherwise if you change something the system
already backed up, it gets done on the next run.
At 11:17 AM 11/2/01 -0800, you wrote:
>Kyle,
>
>I have the opposite opinion as Jesus. File system level backups are the
>way to go. Maybe not on Linux (because of the robustness of the
>snapshot facility), but for sure using Veritas. See, file system level
>database backups are fast, scalable, and simple. However, if you can
>not count on the data consistency, then I would agree with Jesus.
>
>My personal opinion (as a DBA), is that the simplest approach to backup
>and recover is best. If you cant do the restore at 3:00am with one eye
>closed and make 0 mistakes, then it's not worth it (that why I dont use
>Oracle RMAN). I use snapshots (Veritas/Sun) for backups of my
>production DB's. It's really nice to not to ever worry about a backup
>window, just backup whenever!
>
>That said, I am _evaluating_ how good Linux/LVM backups are in this
>regard. So far, it's been fine. Also, as always, make sure you are, or
>have a good DBA around. 8-)
>
>2.5 cents.
>
>Kenny Gorman
>DBA
>Kenamea
>
>
>-----Original Message-----
>From: Jesus Manuel NAVARRO LOPEZ
>Sent: Fri 11/2/2001 10:52 AM
>To: linux-lvm@sistina.com
>Cc:
>Subject: Re: [linux-lvm] snapshot questions
>
>Hi, Kyle:
>
>Kyle Hayes wrote:
>>
>> We are considering using LVM to do database snapshots for backups.
>>
>
>It's so rare the case where this approach really works that it doesn't
>even deserve to be taken into consideration, so: DON'T EVER BACKUP A
>DATABASE THROUGH FILESYSTEM UTILITIES. The database has for sure
>methods to achieve this goal: use them.
>--
>SALUD,
>Jesús
>***
>jesus_navarro@promofinarsa.es
>***
>
>_______________________________________________
>linux-lvm mailing list
>linux-lvm@sistina.com
>http://lists.sistina.com/mailman/listinfo/linux-lvm
>read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
>
>
>
>
>Attachment Converted: "c:\download\winmail2.dat"
>
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [linux-lvm] snapshot questions
2001-11-02 13:59 ` Brent Harding
@ 2001-11-02 15:40 ` Kyle Hayes
2001-11-02 16:39 ` Andreas Dilger
2001-11-05 4:15 ` Jesus Manuel NAVARRO LOPEZ
0 siblings, 2 replies; 18+ messages in thread
From: Kyle Hayes @ 2001-11-02 15:40 UTC (permalink / raw)
To: linux-lvm
Hmm, seems to be quite a range of opinion. However, I must say that
having so many negative options is not a good sign. Do snapshots
work at all?
If I get a corrupted filesystem in my snapshot, then it doesn't work.
If I have to bend over backwards and change all my apps, then it does
not work.
It is good to hear that it works more like Veritas and less like the
SuSE whitepaper claims. That paper claimed that the changes will go
to the snapshot partition. This implied that there would be a terrible
IO hit when you deleted the snapshot partition and the changes need to
be written back to the original partition.
I am confused by the claim (see below) that you need the same amount
of space for the snapshot partition as you had on the original LV.
From what I understood in the HOWTOs and the whitepapers, you only need
as much space as you will have changes during the lifetime of the
snapshot LV. And, if it works like Veritas, that implies that once
a PE was copied, any further data changes to it would not be copied.
This means that in situations like database systems, you'd actually
need less than the number of PEs that changed. Is this correct?
Under what circumstances do snapshots work? I am confused by the
claims that a snapshot partition is going to be "corrupt" as far as
the OS is concerned. In my scenario, here is what I am planning on
doing:
1) lock all tables against writes.
2) flush all logs and tables to disk.
3) create a snapshot partition for the filesystem (ext2 or probably)
on which the database data resides. These are not raw disks.
4) unlock the tables.
5) mount the snapshot partition and copy the database data out.
From what I have guessed so far (guesses, which makes me nervous),
if I make sure all changes are flushed to disk, then I am OK. However,
the responses I have received are not clear on what conditions caused
the problems.
I am also not clear on which kernel and what patches I need to make
sure that the filesystems (ext2,ext3 and reiserfs) will deal with the
VFS patches.
We run 24/7 systems and it is not an option to take down a system to
back it up. We can handle slightly degraded performance, but even
that is a problem.
On Friday 02 November 2001 12:03, Brent Harding wrote:
> The only real issue with snapshots is needing a partition reserved for
> it, as you get more and more data, this gets difficult, once half the
> hard disk is full, you can't copy any more. I'd say it sounds easier to
> just use whatever backup program with cron to execute late at night,
> maybe after midnight, depends how far backup is when a file changes,
> it'll get included if it wasn't backed up yet, otherwise if you change
> something the system already backed up, it gets done on the next run.
This is not what I understood from the HOWTO's and the SuSE whitepaper.
They are quite clear that you only need enough space to contain the
changes that occur while the snapshot partition exists.
Best,
Kyle
--
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MicroTelco Services saves money on every Fax:
- Fax to email (FREE)
- Fax to PSTN based Fax (Up to 95% Savings)
- Fax Broadcasting: Send 100s of faxes to fax machines
and email addresses in the time it takes to send just one!
===========================================================
So send a fax today and let us know what you think!
For more info. visit: www.internetfaxjack.com
===========================================================
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [linux-lvm] snapshot questions
2001-11-02 15:40 ` Kyle Hayes
@ 2001-11-02 16:39 ` Andreas Dilger
2001-11-02 18:20 ` Kyle Hayes
2001-11-05 4:15 ` Jesus Manuel NAVARRO LOPEZ
1 sibling, 1 reply; 18+ messages in thread
From: Andreas Dilger @ 2001-11-02 16:39 UTC (permalink / raw)
To: linux-lvm
On Nov 02, 2001 13:41 -0800, Kyle Hayes wrote:
> I am confused by the claim (see below) that you need the same amount
> of space for the snapshot partition as you had on the original LV.
They are wrong.
> Under what circumstances do snapshots work? I am confused by the
> claims that a snapshot partition is going to be "corrupt" as far as
> the OS is concerned.
You need to ensure that the the data/filesystem is "synced" at the
time that you create the snapshot. With journaling filesystems,
you need to use the VFS locking. With other filesystems, you
should be OK with a "sync" (which is done for you in the kernel).
For oracle, you need to put your tablespaces into hot backup mode.
> 1) lock all tables against writes.
Is this the same as hot backup? At a company I used to work for, we
did live backups without any kind of outage. Oracle would commit
the writes into a journal (or whatever, don't know the correct term)
and then commit them once the table was out of hot backup mode. No
need to lock all writes.
Depending on your I/O and response requirements, you can do the backups
while your tables are in hot backup mode, without using snapshots at
all.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [linux-lvm] snapshot questions
2001-11-02 16:39 ` Andreas Dilger
@ 2001-11-02 18:20 ` Kyle Hayes
2001-11-05 4:38 ` Jesus Manuel NAVARRO LOPEZ
0 siblings, 1 reply; 18+ messages in thread
From: Kyle Hayes @ 2001-11-02 18:20 UTC (permalink / raw)
To: linux-lvm
On Friday 02 November 2001 14:39, Andreas Dilger wrote:
> On Nov 02, 2001 13:41 -0800, Kyle Hayes wrote:
> > I am confused by the claim (see below) that you need the same amount
> > of space for the snapshot partition as you had on the original LV.
>
> They are wrong.
Phwew!
> > Under what circumstances do snapshots work? I am confused by the
> > claims that a snapshot partition is going to be "corrupt" as far as
> > the OS is concerned.
>
> You need to ensure that the the data/filesystem is "synced" at the
> time that you create the snapshot. With journaling filesystems,
> you need to use the VFS locking. With other filesystems, you
> should be OK with a "sync" (which is done for you in the kernel).
> For oracle, you need to put your tablespaces into hot backup mode.
So, the act of creating a snapshot partition actually does the same
thing that running 'sync' would? I.e. all the dirty write buffers
are flushed to disk.
For the database partition we will probably use Ext2.
> > 1) lock all tables against writes.
>
> Is this the same as hot backup? At a company I used to work for, we
> did live backups without any kind of outage. Oracle would commit
> the writes into a journal (or whatever, don't know the correct term)
> and then commit them once the table was out of hot backup mode. No
> need to lock all writes.
The lock prevents other apps from changing anything. This ensures that
nothing will change (perhaps atime entries in the inode since reads can
still occur).
The flush tables action after that will cause MySQL to flush anything it
buffered to disk.
At that point, I have an unchanging, flushed (to the filesystem) version
of the tables. It is coherent at this point if the filesystem is OK.
> Depending on your I/O and response requirements, you can do the backups
> while your tables are in hot backup mode, without using snapshots at
> all.
With MySQL this is not possible at this time. I suspect they'll have it
relatively soon. They seem to add features based on demand and as it
expands in popularity, I suppose that better backup features will be
demanded.
The LVM solution seems like it will also be usable for other things
than databases. I'd like to be able to create a set of scripts and
tools for our use that can be used for more than databases.
On Friday 02 November 2001 15:50, Kenny Gorman wrote:
> I use cooked file systems for Oracle not RAW. I am not sure MySQL has
> an similar function to Oracle Hot Backup. Basically, oracle suspends
> writes to the datafiles for the tablespaces in Hot Backup by keeping the
> whole blocks in the redo logs. Since MySQL (to my knowledge) doesn't do
> something like this, your datafiles will get corrupted when you
> snapshot. I am not sure the 'flush to disk' idea will work... unless
> (and maybe your saying this and I am not getting it) MySQL will suspend
> writes until your snapshot is done. I suppose that would work...
The locking tables part does this. It prevents other threads from writing
to any table. I explain this above.
> In Oracle, the files that are part of the snapshot are not consistent
> (corrupted). Thats just how it works. Then when you use Oracle to
> recover, it puts together the coherent database image using the System
> Change Number and the blocks in the redo logs (recovery structures) and
> archive logs. It's pretty advanced in this regard.
You have the choice in MySQL of locking all tables, or a subset. If you
lock a subset, then you'll probably get a partially incoherent snapshot.
The lock plus flush actions are fast enough that I think we can use them
successfully for this purpose.
> We use Veritas in production with Sun. I am evaluating using Linux in
> production. I am using a SuSE Linux box in production for another
> project, and it's stable. Plan to start using snapshots soon in both
> environments if the evaluation works out. Currently snapshots are in
> use on testing/dev boxes.
I am doing all my testing on SuSE as well. We have deployed Red Hat
in the past, but have not been thrilled with the stability of the
product. By the time you get to a X.2 or so, they are fine. We had
problems with installation of binary drivers on Red Hat (they were our
own drivers!) too, so that also has soured us on it. We have since
released our drivers under the GPL, so that isn't as much of an issue,
but about 75% of the extra work in the configuration and installation
testing just tests Red Hat specific quirks. Sigh.
I really like the fact that SuSE bundles all this stuff already so
that I do not have to go hunting all over the place for an RPM that
might work. I love source but I don't have time to compile everything
myself.
Best,
Kyle
--
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MicroTelco Services saves money on every Fax:
- Fax to email (FREE)
- Fax to PSTN based Fax (Up to 95% Savings)
- Fax Broadcasting: Send 100s of faxes to fax machines
and email addresses in the time it takes to send just one!
===========================================================
So send a fax today and let us know what you think!
For more info. visit: www.internetfaxjack.com
===========================================================
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [linux-lvm] snapshot questions
2001-11-02 18:20 ` Kyle Hayes
@ 2001-11-05 4:38 ` Jesus Manuel NAVARRO LOPEZ
2001-11-05 10:50 ` Kyle Hayes
0 siblings, 1 reply; 18+ messages in thread
From: Jesus Manuel NAVARRO LOPEZ @ 2001-11-05 4:38 UTC (permalink / raw)
To: linux-lvm
Hi again, Kyle:
Kyle Hayes wrote:
>
[...]
>
> > Depending on your I/O and response requirements, you can do the backups
> > while your tables are in hot backup mode, without using snapshots at
> > all.
>
> With MySQL this is not possible at this time. I suspect they'll have it
> relatively soon. They seem to add features based on demand and as it
> expands in popularity, I suppose that better backup features will be
> demanded.
>
So, can I suppouse your RDBM is MySQL? If so, my previous assertion goes
from *very sensible* to a totally must. Don't go with snapshots, go
with dump. If it is safe to suppouse that there's not transaction-like
operations spanning more than one database, write-block and dump on a
per database basis, and pipe the dump to tar -z. This will be safest
method. Well, there's no fool proof backup with MySQL apart from
shuting down the engine and you can bet *there will be* database
unconsistancies disregarding what you do to avoid them (since even
write blocking doesn't guard against bad programming practices and you
can bet your database accessing programs will have more than a bug
regarding "pseudo transacting" through multiple table locking).
> The LVM solution seems like it will also be usable for other things
> than databases. I'd like to be able to create a set of scripts and
> tools for our use that can be used for more than databases.
>
What I would suggest is use snapshots for everything you want *except*
database backups. They can be acomplished easier and safer using
mysqldump.
>
> The locking tables part does this. It prevents other threads from writing
> to any table. I explain this above.
>
But it doesn't protect you against bad programming practices: *Any*
multiple table writing *must* be wrapped between write blocks, *and*
rollback code *must* be in place. I bet that there's at least one
operation on the code accesing those databases that doesn't follow this
must. In this case, sooner or later you will end up "cutting" a
multiple writing operation by half when you write block tables for the
backup. *If* you're very carefull still you will be able to restore to
a stable state using the mysqldump *and* a timestamped transaction log,
but this is a pain in the ass. Luckily enough MySQL is fairly stable,
so you won't need your backups but very rarely.
> > In Oracle, the files that are part of the snapshot are not consistent
> > (corrupted). Thats just how it works. Then when you use Oracle to
> > recover, it puts together the coherent database image using the System
> > Change Number and the blocks in the redo logs (recovery structures) and
> > archive logs. It's pretty advanced in this regard.
>
Yes it is, while quite curious: as you say, once in hot-backup mode
Oracle tables are backuped in an unconsistent manner, but it can recover
to a consistent state because of the fact that though tables change, SCN
markers do not, so redologs -holding the entire block contents, can do
the trick.
> You have the choice in MySQL of locking all tables, or a subset. If you
> lock a subset, then you'll probably get a partially incoherent snapshot.
>
Well yes. As it usually turns to be even if you can find common groups
of tables that can be locked since no operation crosses its boundaries,
things will change tomorrow without you noting it. The only safe
asumption is that you can block database by database instead of all
databases at a time... provided you can policy-force there won't be
cross-database operations.
> The lock plus flush actions are fast enough that I think we can use them
> successfully for this purpose.
>
They usually are... if there's no code flaws around there.
>
> I am doing all my testing on SuSE as well. We have deployed Red Hat
> in the past, but have not been thrilled with the stability of the
> product. By the time you get to a X.2 or so, they are fine. We had
That's why I'm still with RH6.2 and 2.2 kernels.
> problems with installation of binary drivers on Red Hat (they were our
> own drivers!) too, so that also has soured us on it. We have since
> released our drivers under the GPL, so that isn't as much of an issue,
> but about 75% of the extra work in the configuration and installation
> testing just tests Red Hat specific quirks. Sigh.
>
> I really like the fact that SuSE bundles all this stuff already so
> that I do not have to go hunting all over the place for an RPM that
> might work. I love source but I don't have time to compile everything
> myself.
>
Not being used to the SuSE way, I can't say for sure but it surprises my
that SuSE will do "thingies" regarding drivers on a different way RH
does, or that you have less problems "haunting" RPMs for SuSE than for
RH.
--
SALUD,
Jes�s
***
jesus_navarro@promofinarsa.es
***
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [linux-lvm] snapshot questions
2001-11-05 4:38 ` Jesus Manuel NAVARRO LOPEZ
@ 2001-11-05 10:50 ` Kyle Hayes
2001-11-05 14:21 ` Andreas Dilger
0 siblings, 1 reply; 18+ messages in thread
From: Kyle Hayes @ 2001-11-05 10:50 UTC (permalink / raw)
To: linux-lvm, Jesus Manuel NAVARRO LOPEZ
On Monday 05 November 2001 02:39, Jesus Manuel NAVARRO LOPEZ wrote:
> Hi again, Kyle:
>
> Kyle Hayes wrote:
>
>
> [...]
>
> > > Depending on your I/O and response requirements, you can do the
> > > backups while your tables are in hot backup mode, without using
> > > snapshots at all.
> >
> > With MySQL this is not possible at this time. I suspect they'll have
> > it relatively soon. They seem to add features based on demand and as
> > it expands in popularity, I suppose that better backup features will
> > be demanded.
>
> So, can I suppouse your RDBM is MySQL? If so, my previous assertion goes
> from *very sensible* to a totally must. Don't go with snapshots, go
> with dump. If it is safe to suppouse that there's not transaction-like
> operations spanning more than one database, write-block and dump on a
> per database basis, and pipe the dump to tar -z. This will be safest
> method. Well, there's no fool proof backup with MySQL apart from
> shuting down the engine and you can bet *there will be* database
> unconsistancies disregarding what you do to avoid them (since even
> write blocking doesn't guard against bad programming practices and you
> can bet your database accessing programs will have more than a bug
> regarding "pseudo transacting" through multiple table locking).
Mysqldump is not an option due to the amount of time that it takes
to dump multiple databases of multiple gigabytes. Far too slow.
We do not use transactions.
MySQL has a separate tool called myisamchk for MyISAM tables (some
variation on ISAM I suppose) that does a very good job of correcting
problem. Since MySQL only keeps index data in memory, only the indexes
tend to be corrupt. We've had very good luck with this tool repairing
databases that have been corrupted due to hardware crashes etc.
> > The LVM solution seems like it will also be usable for other things
> > than databases. I'd like to be able to create a set of scripts and
> > tools for our use that can be used for more than databases.
>
> What I would suggest is use snapshots for everything you want *except*
> database backups. They can be acomplished easier and safer using
> mysqldump.
As stated above, far too slow. We have remote replicas that we need to
rebuild from time to time (due to takedown for system maintenance etc.).
We do mysqldump backups on a few systems that are not time critical. It
locks all tables while doing a dump of the whole system. If you only
dump one database, it will then be out of sync with the others. We have
written a lot of very specialized program code to get consistent/coherent
snapshots of our databases. It is an ongoing struggle to do this and get
it right.
> > The locking tables part does this. It prevents other threads from
> > writing to any table. I explain this above.
>
> But it doesn't protect you against bad programming practices: *Any*
> multiple table writing *must* be wrapped between write blocks, *and*
> rollback code *must* be in place. I bet that there's at least one
> operation on the code accesing those databases that doesn't follow this
> must. In this case, sooner or later you will end up "cutting" a
> multiple writing operation by half when you write block tables for the
> backup. *If* you're very carefull still you will be able to restore to
> a stable state using the mysqldump *and* a timestamped transaction log,
> but this is a pain in the ass. Luckily enough MySQL is fairly stable,
> so you won't need your backups but very rarely.
Since we do not use transactions at all, we are able to relax the requirements
of "standard" database programs. We can do a snapshot of the system with
partially completed transactions and our software and checking programs
have no problems. We've written the code to expect that some things just
aren't complete yet. This is not a standard database application.
> > You have the choice in MySQL of locking all tables, or a subset. If
> > you lock a subset, then you'll probably get a partially incoherent
> > snapshot.
>
> Well yes. As it usually turns to be even if you can find common groups
> of tables that can be locked since no operation crosses its boundaries,
> things will change tomorrow without you noting it. The only safe
> asumption is that you can block database by database instead of all
> databases at a time... provided you can policy-force there won't be
> cross-database operations.
No operations cross table set boundaries. We wrote it that way.
> > I really like the fact that SuSE bundles all this stuff already so
> > that I do not have to go hunting all over the place for an RPM that
> > might work. I love source but I don't have time to compile everything
> > myself.
>
> Not being used to the SuSE way, I can't say for sure but it surprises my
> that SuSE will do "thingies" regarding drivers on a different way RH
> does, or that you have less problems "haunting" RPMs for SuSE than for
> RH.
SuSE simply comes with virtually everything we've ever needed. Red Hat
required too much hunting on the Internet for RPMs that might or
might not work.
If SuSE generally doesn't have the RPM on their CDs or web site, it is
more than likely that no one has an RPM. In that case, we use source.
The only problem we have with SuSE is that it is hard to strip it down
to a minimum for deployment on locked-down servers.
This is getting rather off topic for the LVM list :-)
To bring it back on topic. I have learned so far that:
1) if your apps think they left everything on the filesystem in a
coherent state and,
2) if your filesystem thinks that it left the disk in a coherent
state,
then you will be able to use the snapshot facility without a problem.
We can insure 1 with no problem. 2 seems to require either patches or
new versions of various code. I have not yet seen a full reply as to
what patches are necessary for which kernels.
I have seen that the AC kernels are being patched nearly up to the level
of what is in CVS. But, this is for 2.4.x. What about 2.2.x? Is 2.2.20
going to have the latest LVM patches?
Best,
Kyle
--
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MicroTelco Services saves money on every Fax:
- Fax to email (FREE)
- Fax to PSTN based Fax (Up to 95% Savings)
- Fax Broadcasting: Send 100s of faxes to fax machines
and email addresses in the time it takes to send just one!
===========================================================
So send a fax today and let us know what you think!
For more info. visit: www.internetfaxjack.com
===========================================================
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [linux-lvm] snapshot questions
2001-11-05 10:50 ` Kyle Hayes
@ 2001-11-05 14:21 ` Andreas Dilger
0 siblings, 0 replies; 18+ messages in thread
From: Andreas Dilger @ 2001-11-05 14:21 UTC (permalink / raw)
To: Kyle Hayes; +Cc: linux-lvm, Jesus Manuel NAVARRO LOPEZ
On Nov 05, 2001 08:51 -0800, Kyle Hayes wrote:
> We can insure 1 with no problem. 2 seems to require either patches or
> new versions of various code. I have not yet seen a full reply as to
> what patches are necessary for which kernels.
You need LVM-VFS patch on all kernels, you need ext3 or reiserfs or XFS
to get "real" VFS lock usage. For non-journaled fs (like ext2) it just
syncs, and you mount the snapshot - it will complain that it is dirty,
but fsck will verify that there are no errors.
> I have seen that the AC kernels are being patched nearly up to the level
> of what is in CVS. But, this is for 2.4.x. What about 2.2.x? Is 2.2.20
> going to have the latest LVM patches?
No, LVM was never part of 2.2, so you always need to patch.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [linux-lvm] snapshot questions
2001-11-02 15:40 ` Kyle Hayes
2001-11-02 16:39 ` Andreas Dilger
@ 2001-11-05 4:15 ` Jesus Manuel NAVARRO LOPEZ
1 sibling, 0 replies; 18+ messages in thread
From: Jesus Manuel NAVARRO LOPEZ @ 2001-11-05 4:15 UTC (permalink / raw)
To: linux-lvm
Hi, Kyle:
Kyle Hayes wrote:
>
> Hmm, seems to be quite a range of opinion. However, I must say that
> having so many negative options is not a good sign. Do snapshots
> work at all?
>
Well, I can't say if LVM snapshots do work, but I know for sure they can
be very useful on a wide bunch of situations.
[...]
>
> It is good to hear that it works more like Veritas and less like the
> SuSE whitepaper claims. That paper claimed that the changes will go
> to the snapshot partition. This implied that there would be a terrible
> IO hit when you deleted the snapshot partition and the changes need to
> be written back to the original partition.
>
Huh? Well, I haven't search the code, but the LVM How to stays that it
has to "...hold all the changes that are likely to happen to the
original volume during the lifetime of the snapshot", and it says too
"...You should remove snapshot volume when you have finished with them
because they take a copy of all data wirtten to the original volume".
On the other hand, rigth in between those two states it says: "Note that
the [snapshot] volume was mounted read-only. Snapshots can never be
written to, and the data in them cannot change." So there is obviously
something missed in here.
> I am confused by the claim (see below) that you need the same amount
> of space for the snapshot partition as you had on the original LV.
Again, from the LVM How To: "A snapshot volume can be as large or a
[sic] small as you like but it must be large enough to hold all the
changes that are likely to happen... [...] A snapshot logical volume
can be a maximum of 1.1x the size of the original volume".
[...]
> Under what circumstances do snapshots work? I am confused by the
> claims that a snapshot partition is going to be "corrupt" as far as
> the OS is concerned. In my scenario, here is what I am planning on
> doing:
>
> 1) lock all tables against writes.
>
> 2) flush all logs and tables to disk.
>
> 3) create a snapshot partition for the filesystem (ext2 or probably)
> on which the database data resides. These are not raw disks.
>
> 4) unlock the tables.
>
> 5) mount the snapshot partition and copy the database data out.
>
That's a "standard procedure" for RDBMs that don't support hot backups
with the exception that the "standard" would include a dump of the
database contents instead of the filesystem snapshot. *If* you know
there won't be operations spanning more than one database, you can
reduce comptention by just blocking/dumping one database at a time.
Then I would evaluate speed. If the dump doesn't need too much time
than the snapshot I would go with it because of portability and
(somehow) integrity (you will have a database operation completly driven
from the RDBM). Under some circumnstances size can be an issue too: if
you go with the snapshot you will have to do it big enough to cope with
the worst case scenario, and since it does involve creating and deleting
a filesystem you will find more dificult to reclaim that disk space if
needed. On the other hand, the dump will most probably need less space
(specially if you pipe it through gzip, or tar -z) and will need as much
space as really needed, which can be reclaimed as soon as the dump is
finished and properly stored somewhere else.
>
> We run 24/7 systems and it is not an option to take down a system to
> back it up. We can handle slightly degraded performance, but even
> that is a problem.
>
Again, regarding database operations you will end up most probably
recognizing that database backup is a database issue best coped with
database tools. Snapshots can be useful tool for many other situations
like mail (or other) spools, user data, etc.
(just my 2 cents)
--
SALUD,
Jes�s
***
jesus_navarro@promofinarsa.es
***
^ permalink raw reply [flat|nested] 18+ messages in thread
* RE: [linux-lvm] snapshot questions
@ 2001-11-02 13:16 Kenny Gorman
2001-11-05 3:27 ` Jesus Manuel NAVARRO LOPEZ
0 siblings, 1 reply; 18+ messages in thread
From: Kenny Gorman @ 2001-11-02 13:16 UTC (permalink / raw)
To: linux-lvm
[-- Attachment #1: Type: text/plain, Size: 1790 bytes --]
Kyle,
I have the opposite opinion as Jesus. File system level backups are the
way to go. Maybe not on Linux (because of the robustness of the
snapshot facility), but for sure using Veritas. See, file system level
database backups are fast, scalable, and simple. However, if you can
not count on the data consistency, then I would agree with Jesus.
My personal opinion (as a DBA), is that the simplest approach to backup
and recover is best. If you cant do the restore at 3:00am with one eye
closed and make 0 mistakes, then it's not worth it (that why I dont use
Oracle RMAN). I use snapshots (Veritas/Sun) for backups of my
production DB's. It's really nice to not to ever worry about a backup
window, just backup whenever!
That said, I am _evaluating_ how good Linux/LVM backups are in this
regard. So far, it's been fine. Also, as always, make sure you are, or
have a good DBA around. 8-)
2.5 cents.
Kenny Gorman
DBA
Kenamea
-----Original Message-----
From: Jesus Manuel NAVARRO LOPEZ
Sent: Fri 11/2/2001 10:52 AM
To: linux-lvm@sistina.com
Cc:
Subject: Re: [linux-lvm] snapshot questions
Hi, Kyle:
Kyle Hayes wrote:
>
> We are considering using LVM to do database snapshots for backups.
>
It's so rare the case where this approach really works that it doesn't
even deserve to be taken into consideration, so: DON'T EVER BACKUP A
DATABASE THROUGH FILESYSTEM UTILITIES. The database has for sure
methods to achieve this goal: use them.
--
SALUD,
Jesús
***
jesus_navarro@promofinarsa.es
***
_______________________________________________
linux-lvm mailing list
linux-lvm@sistina.com
http://lists.sistina.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
[-- Attachment #2: winmail.dat --]
[-- Type: application/ms-tnef, Size: 3475 bytes --]
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [linux-lvm] snapshot questions
2001-11-02 13:16 Kenny Gorman
@ 2001-11-05 3:27 ` Jesus Manuel NAVARRO LOPEZ
0 siblings, 0 replies; 18+ messages in thread
From: Jesus Manuel NAVARRO LOPEZ @ 2001-11-05 3:27 UTC (permalink / raw)
To: linux-lvm
Hi, Kenny:
Kenny Gorman wrote:
>
> Kyle,
>
> I have the opposite opinion as Jesus. File system level backups are the
> way to go. Maybe not on Linux (because of the robustness of the
> snapshot facility), but for sure using Veritas. See, file system level
> database backups are fast, scalable, and simple. However, if you can
> not count on the data consistency, then I would agree with Jesus.
>
Well, yes, that's the point: You can never count on data consistency
regarding database activities. As a general matter, data disk
consistency is not mapped one-to-one to data base consistency: The
filesystem can be glad enough regarding consistency on the middle of a
database transaction, while the database won't be so, in case data is to
be recovered from that backup. It *migth* probe that in any particular
case sysadmin and dba are working close enough so in one particular
installation you can sign on the "yes, you can safely do filesystem
backups", but that won't be the general case, not by far. Even if
write-to-disk is an atomic operation, you cannot know for sure that
writing down transaction logs will be so (even having filesystem
snapshots in place: unless the snapshotting is *very* carefully
programmed to be truly atomic *and* there's no write buffer around -OS
write buffer, or hardware one -both at the disks or the disk controller,
you can't count there won't be any write operation from the moment the
snapshot is asked to be started and the moment it is taken).
In the usual case, the sysadmin will do system backups (they're
simpler), he'll test it and they'll work, so they'll go into
production. Once in a sudden, those backups will be needed, and then,
misteriously, they'll bring the database to an unconsistant state.
Why? Because tests were done under ligth on no load at all so having
the backup storing half an atomic operation is the rarest case. Once in
production you can bet that -taking apart human mistakes, database
problems will appear under heavy load conditions which mostly will mean
that last backup will include unconsistant data (either because your
RDBM doesn't support true ACID, or because programmers didn't wrapped
that operation that needed it into a single transaction, or because the
operation would need to be insured as atomic but it couldn't from the
RDBM point of view because it expanded more than a single database,
or...).
> My personal opinion (as a DBA), is that the simplest approach to backup
> and recover is best. If you cant do the restore at 3:00am with one eye
> closed and make 0 mistakes, then it's not worth it (that why I dont use
> Oracle RMAN). I use snapshots (Veritas/Sun) for backups of my
> production DB's. It's really nice to not to ever worry about a backup
> window, just backup whenever!
>
Hmmm... The one and only way to achieve that goal is heavy automation,
heavy documentation and heavy training/testing your procedures. Even if
you can be confident on your snapshots (the only one I know of are Sun's
off-site hardware replication, but that's another story), they'll be
done with certain delay, so if you should return the database to the
latest consistant state possible, still you will need last transaction
log available. All in all, at the end you will find that either
database tools are the ones that best fit (since you will end up needing
them if only in some cases) or that you're lucky enough that -as DBA,
you can pass that shit down to the sysadmin (but that wouldn't be fair
play, at least).
Speaking about Oracle, I have to say I'm not proficient at it (to say
the least), but AFAIK, it has its own snapshot methods to do hot
backups, so you're most probably re-doing filesystem level what can be
done with more accuracy at RDBM level.
There's even some "politics" envolved here: the health of the databases
is a concern for the DBA; the RDBM will have is own tools to achieve the
goal, so it is *her* responsibility to have them in place. It will make
no difference if sysadmin and DBA is the same person, since this migth
change tomorrow, and when this happens both sysadmin and dba will be
glad not needing to take more responsibilities than those associated
with the post.
Being said this all, there *is* one situation when filesystem backup is
the tool to choice and this is it when you can turn down your RDBM prior
to the backup. Even then it obviously need to be sync'd with the DBA,
since she will be the one who will turn off and on the RDBM (and know
how to do this safely, or at least, being her responsibility to do so).
> That said, I am _evaluating_ how good Linux/LVM backups are in this
> regard. So far, it's been fine. Also, as always, make sure you are, or
> have a good DBA around. 8-)
>
Fair enough, no doubt.
--
SALUD,
Jes�s
***
jesus_navarro@promofinarsa.es
***
^ permalink raw reply [flat|nested] 18+ messages in thread
* [linux-lvm] snapshot questions
@ 2001-11-02 10:42 Kyle Hayes
2001-11-02 11:28 ` Andreas Dilger
2001-11-02 12:51 ` Jesus Manuel NAVARRO LOPEZ
0 siblings, 2 replies; 18+ messages in thread
From: Kyle Hayes @ 2001-11-02 10:42 UTC (permalink / raw)
To: linux-lvm
We are considering using LVM to do database snapshots for backups.
I have some questions that the HOWTO's etc. do not seem to answer.
My largest concerns are that we will end up with a read-only corrupt
filesystem. This seems to be a problem reported in the mailing lists.
1) how does the snapshot actually work? My understanding is that the
creation of a snapshot partition sets up a new LV on the same VG as
the "source" partition. This new partition is then used for writes to
the "source" partition. When the snapshot LV is removed, do I get
a disk IO hit as these changes are copied back to the "source" partition?
Or, does it work more like Veritas where the changed data go to the
"source" and the original values are copied to the snapshot partition?
The Veritas method is nice because dropping the snapshot partition
does not require any data copying.
2) are there any people using LVM for database snapshots? The
Whitepaper from SuSE has examples for Oracle and the HOWTO's
talk about it, but is it used in production? If so, what issues have you
found?
3) what is the overall stability of snapshot functions? I have been following
the pvmove thread with some alarm as I thought that LVM functionality was
stable at this point. Is there a place that I can see what functionality is
considered stable and what isn't? The mailing list archives do not paint a rosy
picture of snapshot functionality.
4) It seems that most of the LVM work is going into 2.4. We have not been
particularly pleased with the stability of 2.4 in general and would like to stay
with 2.2 for a while. How does the stability of LVM on 2.2 compare to 2.4?
5) I have seen references to the VFS lock patch in the mailing list. Is this
actually supported by any filesystems yet? Is it in the kernel proper?
Thanks.
Best,
Kyle
--
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MicroTelco Services saves money on every Fax:
- Fax to email (FREE)
- Fax to PSTN based Fax (Up to 95% Savings)
- Fax Broadcasting: Send 100s of faxes to fax machines
and email addresses in the time it takes to send just one!
===========================================================
So send a fax today and let us know what you think!
For more info. visit: www.internetfaxjack.com
===========================================================
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [linux-lvm] snapshot questions
2001-11-02 10:42 Kyle Hayes
@ 2001-11-02 11:28 ` Andreas Dilger
2001-11-02 12:51 ` Jesus Manuel NAVARRO LOPEZ
1 sibling, 0 replies; 18+ messages in thread
From: Andreas Dilger @ 2001-11-02 11:28 UTC (permalink / raw)
To: Kyle Hayes; +Cc: linux-lvm
On Nov 02, 2001 08:43 -0800, Kyle Hayes wrote:
> 1) how does the snapshot actually work? My understanding is that the
> creation of a snapshot partition sets up a new LV on the same VG as
> the "source" partition. This new partition is then used for writes to
> the "source" partition. When the snapshot LV is removed, do I get
> a disk IO hit as these changes are copied back to the "source" partition?
> Or, does it work more like Veritas where the changed data go to the
> "source" and the original values are copied to the snapshot partition?
> The Veritas method is nice because dropping the snapshot partition
> does not require any data copying.
It is like the veritas method - changes go into the original LV, and the
old data is copied into the snapshot(s).
> 2) are there any people using LVM for database snapshots? The
> Whitepaper from SuSE has examples for Oracle and the HOWTO's
> talk about it, but is it used in production? If so, what issues have you
> found?
You probably need to put your tablespaces into "backup" mode temporarily
when doing the snapshot. Otherwise there is no guarantee that what is
currently on disk is any good.
> 4) It seems that most of the LVM work is going into 2.4. We have not
> been particularly pleased with the stability of 2.4 in general and would
> like to stay with 2.2 for a while. How does the stability of LVM on 2.2
> compare to 2.4?
It should be pretty much identical. There are compatibility hooks for
2.2 and 2.4, and they both use an otherwise identical code base.
> 5) I have seen references to the VFS lock patch in the mailing list.
> Is this actually supported by any filesystems yet? Is it in the
> kernel proper?
Yes, reiserfs, ext3, and XFS all support the lock interface to the VFS.
Note that the stock 2.4 kernel has the VFS methods for this, but is
missing the code to do it. There is the capability to support this for
ext3 on 2.2, but it is not in the stock code. I think the same is true
for reiserfs.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [linux-lvm] snapshot questions
2001-11-02 10:42 Kyle Hayes
2001-11-02 11:28 ` Andreas Dilger
@ 2001-11-02 12:51 ` Jesus Manuel NAVARRO LOPEZ
1 sibling, 0 replies; 18+ messages in thread
From: Jesus Manuel NAVARRO LOPEZ @ 2001-11-02 12:51 UTC (permalink / raw)
To: linux-lvm
Hi, Kyle:
Kyle Hayes wrote:
>
> We are considering using LVM to do database snapshots for backups.
>
It's so rare the case where this approach really works that it doesn't
even deserve to be taken into consideration, so: DON'T EVER BACKUP A
DATABASE THROUGH FILESYSTEM UTILITIES. The database has for sure
methods to achieve this goal: use them.
--
SALUD,
Jes�s
***
jesus_navarro@promofinarsa.es
***
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2007-05-06 7:05 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2001-11-02 15:58 [linux-lvm] snapshot questions Kenny Gorman
2001-11-02 17:04 ` Kyle Hayes
-- strict thread matches above, loose matches on Subject: below --
2007-05-06 5:32 [linux-lvm] Snapshot questions Ed Martin
2001-11-02 17:49 [linux-lvm] snapshot questions Kenny Gorman
2001-11-05 10:31 ` Jesus Manuel NAVARRO LOPEZ
[not found] <D880FFC9CFDBA84FA13E9B166D06A123B647FB@daedalus.kenamea.co m>
2001-11-02 13:59 ` Brent Harding
2001-11-02 15:40 ` Kyle Hayes
2001-11-02 16:39 ` Andreas Dilger
2001-11-02 18:20 ` Kyle Hayes
2001-11-05 4:38 ` Jesus Manuel NAVARRO LOPEZ
2001-11-05 10:50 ` Kyle Hayes
2001-11-05 14:21 ` Andreas Dilger
2001-11-05 4:15 ` Jesus Manuel NAVARRO LOPEZ
2001-11-02 13:16 Kenny Gorman
2001-11-05 3:27 ` Jesus Manuel NAVARRO LOPEZ
2001-11-02 10:42 Kyle Hayes
2001-11-02 11:28 ` Andreas Dilger
2001-11-02 12:51 ` Jesus Manuel NAVARRO LOPEZ
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).