* [linux-lvm] LVM Snapshots for remote archiving. @ 2004-01-27 18:05 Chris Beck 2004-01-27 18:46 ` Greg Freemyer 0 siblings, 1 reply; 6+ messages in thread From: Chris Beck @ 2004-01-27 18:05 UTC (permalink / raw) To: linux-lvm Hi! I've been looking through the archives and I can't see anything addressing my issue. Probably because I didn't look hard enough. I want to use 24-hour snap shots as an archival tool. I have 2 identical file servers, one primary and one as an off-site mirror. I'd like the primary system to generate a transaction log that rolls over every 24-hours and gets transmitted to the remote site. After 7 days on the remote site, the log gets triggered so the remote is always 7 days out of sync but with the last 6 days logs ready to go. I could have it auto update on arrival, but I was thinking that allowing a week for someone to realize that they deleted something vital would be a good thing(TM) - standard archiving stuff I guess. Do you think that lvm snapshot volumes are a simple and convenient way of doing this? Does this make sense at all? Thanks, Chris -- Chris Beck / Y.A.B.A. / Fungal Genomics CFSG / Concordia University "La loi dans sa majestueuse égalité, interdit à tous, aux riches comme aux pauvres de dormir sous les ponts, de coucher dans la rue et de voler du pain." -- Anatole France (Les Lys Rouge - 1894) ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [linux-lvm] LVM Snapshots for remote archiving. 2004-01-27 18:05 [linux-lvm] LVM Snapshots for remote archiving Chris Beck @ 2004-01-27 18:46 ` Greg Freemyer 2004-01-28 11:44 ` Chris Beck 0 siblings, 1 reply; 6+ messages in thread From: Greg Freemyer @ 2004-01-27 18:46 UTC (permalink / raw) To: linux-lvm On Tue, 2004-01-27 at 18:03, Chris Beck wrote: > Hi! I've been looking through the archives and I can't see anything > addressing my issue. Probably because I didn't look hard enough. > > I want to use 24-hour snap shots as an archival tool. > I have 2 identical file servers, one primary and one as an off-site > mirror. I'd like the primary system to generate a transaction log that > rolls over every 24-hours and gets transmitted to the remote site. > After 7 days on the remote site, the log gets triggered so the remote is > always 7 days out of sync but with the last 6 days logs ready to go. I > could have it auto update on arrival, but I was thinking that allowing a > week for someone to realize that they deleted something vital would be a > good thing(TM) - standard archiving stuff I guess. > > Do you think that lvm snapshot volumes are a simple and convenient way > of doing this? Does this make sense at all? > > Thanks, > Chris The above transaction log is new to me. I have seen that done with databases, but not with generic fileservers. If I was trying to accomplish what I think are your goals, I would use rdiff-backup. I don't know the syntax offhand, but the process would be: Create snapshot mount snapshot as /snap rdiff-backup /snap //backup_server/snap rdiff-backup --delete_older_than 7-days //backup_server rdiff-backup uses technology similar to rsync to ensure only the deltas are sent between the servers. (ie. first check datestamps etc. to see if file changed. If so, use md5sums to isolate what parts of the file changed and send them between the servers.) It maintains a current copy of the source files, and a series of diffs to go backwards to older versions of the files. In the above it would keep a max of 7 diffs per file. If you wanted to restore from 7 days prior, rdiff-backup would internally have to apply all 7 diffs, one after another. rdiff-backup has lots of other features/capabilities. Also, it uses ssh to encrypt all data between the servers, so it is also fairly secure. The only problems I have with rdiff-backup are: 1) ACL and EA support is only in the Unstable release, but they seem to be working fine 2) rdiff-backup is written in python and when an unhandled error occurs, it dumps out a stack trace. To see what the problem is you have to go thru the stack trace and the code. Not to bad if you have some programming skills. If not, the mailing list is fairly responsive. There is a wiki at http://rdiff-backup.solutionsfirst.com.au/index.php?RdiffBackupWiki Greg -- Greg Freemyer ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [linux-lvm] LVM Snapshots for remote archiving. 2004-01-27 18:46 ` Greg Freemyer @ 2004-01-28 11:44 ` Chris Beck 2004-01-28 12:33 ` Greg Freemyer 0 siblings, 1 reply; 6+ messages in thread From: Chris Beck @ 2004-01-28 11:44 UTC (permalink / raw) To: linux-lvm I guess I am misunderstanding how a snapshot volume works, I kinda thought it would be this mythical transaction log - disk writes need to be stored somewhere while the original volume is frozen and I was hoping that I could move the whole thing around like any other file. I was hoping to avoid rsync/rdiff because the snapshot volume itself was a list of differences over the last 24 hours. It is whispered that Greg Freemyer was heard, on or about 1/27/2004 6:29 PM to say: >On Tue, 2004-01-27 at 18:03, Chris Beck wrote: > > >>Hi! I've been looking through the archives and I can't see anything >>addressing my issue. Probably because I didn't look hard enough. >> >>I want to use 24-hour snap shots as an archival tool. >>I have 2 identical file servers, one primary and one as an off-site >>mirror. I'd like the primary system to generate a transaction log that >>rolls over every 24-hours and gets transmitted to the remote site. >>After 7 days on the remote site, the log gets triggered so the remote is >>always 7 days out of sync but with the last 6 days logs ready to go. I >>could have it auto update on arrival, but I was thinking that allowing a >>week for someone to realize that they deleted something vital would be a >>good thing(TM) - standard archiving stuff I guess. >> >>Do you think that lvm snapshot volumes are a simple and convenient way >>of doing this? Does this make sense at all? >> >>Thanks, >>Chris >> >> > >The above transaction log is new to me. I have seen that done with >databases, but not with generic fileservers. > >If I was trying to accomplish what I think are your goals, I would use >rdiff-backup. > >I don't know the syntax offhand, but the process would be: > >Create snapshot >mount snapshot as /snap >rdiff-backup /snap //backup_server/snap >rdiff-backup --delete_older_than 7-days //backup_server > >rdiff-backup uses technology similar to rsync to ensure only the deltas >are sent between the servers. (ie. first check datestamps etc. to see >if file changed. If so, use md5sums to isolate what parts of the file >changed and send them between the servers.) > >It maintains a current copy of the source files, and a series of diffs >to go backwards to older versions of the files. In the above it would >keep a max of 7 diffs per file. If you wanted to restore from 7 days >prior, rdiff-backup would internally have to apply all 7 diffs, one >after another. > >rdiff-backup has lots of other features/capabilities. > >Also, it uses ssh to encrypt all data between the servers, so it is also >fairly secure. > >The only problems I have with rdiff-backup are: > >1) ACL and EA support is only in the Unstable release, but they seem to >be working fine > >2) rdiff-backup is written in python and when an unhandled error occurs, >it dumps out a stack trace. To see what the problem is you have to go >thru the stack trace and the code. Not to bad if you have some >programming skills. If not, the mailing list is fairly responsive. > >There is a wiki at >http://rdiff-backup.solutionsfirst.com.au/index.php?RdiffBackupWiki > >Greg > > -- Chris Beck / Y.A.B.A. / Fungal Genomics CFSG / Concordia University "La loi dans sa majestueuse égalité, interdit à tous, aux riches comme aux pauvres de dormir sous les ponts, de coucher dans la rue et de voler du pain." -- Anatole France (Les Lys Rouge - 1894) ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [linux-lvm] LVM Snapshots for remote archiving. 2004-01-28 11:44 ` Chris Beck @ 2004-01-28 12:33 ` Greg Freemyer 2004-01-28 14:06 ` Chris Beck 0 siblings, 1 reply; 6+ messages in thread From: Greg Freemyer @ 2004-01-28 12:33 UTC (permalink / raw) To: linux-lvm On Wed, 2004-01-28 at 11:43, Chris Beck wrote: > I guess I am misunderstanding how a snapshot volume works, I kinda > thought it would be this mythical transaction log - disk writes need to > be stored somewhere while the original volume is frozen and I was hoping > that I could move the whole thing around like any other file. > > I was hoping to avoid rsync/rdiff because the snapshot volume itself was > a list of differences over the last 24 hours. > > It is whispered that Greg Freemyer was heard, on or about 1/27/2004 6:29 > PM to say: > A snapshot is based on copy on write technology. I guess you could use it to create the transaction log you are talking about, but it is not what it is designed to do, and I believe significant effort would be involved. Basically with a snapshot, a blank disk volume is created with a flag table with one entry per block of master volume data. All of the flag entries are initialized as "unmodified". When a write is performed to the master volume, the flag table is checked. If the flag for that particular block is set to "unmodified", then a copy of the unmodified block is made to the snapshot area prior to the write being performed. The flag is then set to modified. Future writes to that specific block will have no effect on the flag table, nor on the snapshot volume. For your transaction log, the flag table is significant, but the actual snapshot volume is not. It is just the old unmodified content. To get a true set of differences over a 24-hour period, you would have to create a snapshot at time T, and another at time T+24. (Easily done I think). Then have code that went thru the flag table for snapshot T and retrieved the blocks from snapshot T+24 that were marked as modified. (LVM will automatically get the block from the appropriate source: T+24 or Master volume) Once done, the snapshot T would be deleted, and snapshot T+24 would be maintained. At the 48 hour mark, snapshot T+48 would be created and the process repeated. I don't think that is a trivial issue, and I think the code would have to implemented in the kernel. New code would also have to be developed to apply the above blocks. HTH Greg -- Greg Freemyer ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [linux-lvm] LVM Snapshots for remote archiving. 2004-01-28 12:33 ` Greg Freemyer @ 2004-01-28 14:06 ` Chris Beck 2004-01-29 11:52 ` Greg Freemyer 0 siblings, 1 reply; 6+ messages in thread From: Chris Beck @ 2004-01-28 14:06 UTC (permalink / raw) To: linux-lvm It is whispered that Greg Freemyer was heard, on or about 1/28/2004 12:19 PM to say: >A snapshot is based on copy on write technology. I guess you could use >it to create the transaction log you are talking about, but it is not >what it is designed to do, and I believe significant effort would be >involved. > >Basically with a snapshot, a blank disk volume is created with a flag >table with one entry per block of master volume data. All of the flag >entries are initialized as "unmodified". > >When a write is performed to the master volume, the flag table is >checked. If the flag for that particular block is set to "unmodified", >then a copy of the unmodified block is made to the snapshot area prior >to the write being performed. The flag is then set to modified. > >Future writes to that specific block will have no effect on the flag >table, nor on the snapshot volume. > > Oh well, I guess I was thinking that it was the reverse: changes were written to the new volume, which was then merged into the old one once the snapshot was finished. >For your transaction log, the flag table is significant, but the actual >snapshot volume is not. It is just the old unmodified content. > >To get a true set of differences over a 24-hour period, you would have >to create a snapshot at time T, and another at time T+24. (Easily done >I think). > >Then have code that went thru the flag table for snapshot T and >retrieved the blocks from snapshot T+24 that were marked as modified. >(LVM will automatically get the block from the appropriate source: T+24 >or Master volume) > >Once done, the snapshot T would be deleted, and snapshot T+24 would be >maintained. At the 48 hour mark, snapshot T+48 would be created and the >process repeated. > >I don't think that is a trivial issue, and I think the code would have >to implemented in the kernel. > > Why the kernel? ... surely reading the flag tables could be done in user space? I think you have the necessary logic spot on. >New code would also have to be developed to apply the above blocks. > > Hmmm. I can mess around in perl and java as well as the next guy, but my c-fu is weak, very weak. How much would it cost to sponsor someone to do this do you think? -- Chris Beck / Y.A.B.A. / Fungal Genomics CFSG / Concordia University "La loi dans sa majestueuse égalité, interdit à tous, aux riches comme aux pauvres de dormir sous les ponts, de coucher dans la rue et de voler du pain." -- Anatole France (Les Lys Rouge - 1894) ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [linux-lvm] LVM Snapshots for remote archiving. 2004-01-28 14:06 ` Chris Beck @ 2004-01-29 11:52 ` Greg Freemyer 0 siblings, 0 replies; 6+ messages in thread From: Greg Freemyer @ 2004-01-29 11:52 UTC (permalink / raw) To: linux-lvm On Wed, 2004-01-28 at 14:04, Chris Beck wrote: > >I don't think that is a trivial issue, and I think the code would have > >to implemented in the kernel. > > > > > Why the kernel? ... surely reading the flag tables could be done in user > space? I think you have the necessary logic spot on. I don't think the COW flag table is accessible from userspace, but I could be wrong. FYI: I am a LVM user, not a developer. > > >New code would also have to be developed to apply the above blocks. > > > > > Hmmm. I can mess around in perl and java as well as the next guy, but > my c-fu is weak, very weak. How much would it cost to sponsor someone > to do this do you think? I don't have any idea. If your serious, I would recommend doing it with LVM 2 (requires DM in the kernel), since DM is part of 2.6.x vanilla and 2.6.x will not likely ever support LVM 1.x. In that case, I would post on the dm-devel list. There seem to be a few non-Sistina people that are very knowledgable and post regularily. http://lists.sistina.com/mailman/listinfo/dm-devel FYI: Since Sistina is in the process of being bought by Redhat, I doubt they would consider a new project (or sub-project) right now. HTH Greg -- Greg Freemyer ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2004-01-29 11:52 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2004-01-27 18:05 [linux-lvm] LVM Snapshots for remote archiving Chris Beck 2004-01-27 18:46 ` Greg Freemyer 2004-01-28 11:44 ` Chris Beck 2004-01-28 12:33 ` Greg Freemyer 2004-01-28 14:06 ` Chris Beck 2004-01-29 11:52 ` Greg Freemyer
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.