How to backup of large md raid volumes?

How to backup of large md raid volumes?

[Ram Ramesh]

Any one have a method to backup large volumes like md raid6 (16TB)? 
Since the backup will not fit in one disk (in many cases and mine too) I 
am wondering, if there is a known/easy technique to backup using 
multiple usb hard drives. I googed and found a few fancy backup 
utils/systems like Amanda etc. They are overkill for me. I am choosing 
not to back up simply because of the complexity of setup as the data in 
my RAID volume is NOT precious and can be replaced with a week of effort.

If any one can think of some thing simple, please point me to it and I 
will do the reading to figure out.

Thanks
Ramesh

Re: How to backup of large md raid volumes?

[Adam Goryachev]

On 19/05/17 07:37, Ram Ramesh wrote:
> Any one have a method to backup large volumes like md raid6 (16TB)? 
> Since the backup will not fit in one disk (in many cases and mine too) 
> I am wondering, if there is a known/easy technique to backup using 
> multiple usb hard drives. I googed and found a few fancy backup 
> utils/systems like Amanda etc. They are overkill for me. I am choosing 
> not to back up simply because of the complexity of setup as the data 
> in my RAID volume is NOT precious and can be replaced with a week of 
> effort.
>
> If any one can think of some thing simple, please point me to it and I 
> will do the reading to figure out.
>
I suppose it depends on what you want to achieve. There are various 
options such as mirroring to another raid6 array (ie, RAID60) but really 
that isn't a backup, it's another replica.
I use backuppc for my backups, it works well under linux with rsync, I'm 
sure there are many various options (including amanda which I've used in 
the past). Ultimately, it depends on your requirements, backups vary 
significantly depending on needs/etc.

Regards,
Adam


-- 
Adam Goryachev Website Managers www.websitemanagers.com.au

Re: How to backup of large md raid volumes?

[Ram Ramesh]

On 05/18/2017 08:34 PM, Adam Goryachev wrote:
> On 19/05/17 07:37, Ram Ramesh wrote:
>> Any one have a method to backup large volumes like md raid6 (16TB)? 
>> Since the backup will not fit in one disk (in many cases and mine 
>> too) I am wondering, if there is a known/easy technique to backup 
>> using multiple usb hard drives. I googed and found a few fancy backup 
>> utils/systems like Amanda etc. They are overkill for me. I am 
>> choosing not to back up simply because of the complexity of setup as 
>> the data in my RAID volume is NOT precious and can be replaced with a 
>> week of effort.
>>
>> If any one can think of some thing simple, please point me to it and 
>> I will do the reading to figure out.
>>
> I suppose it depends on what you want to achieve. There are various 
> options such as mirroring to another raid6 array (ie, RAID60) but 
> really that isn't a backup, it's another replica.
> I use backuppc for my backups, it works well under linux with rsync, 
> I'm sure there are many various options (including amanda which I've 
> used in the past). Ultimately, it depends on your requirements, 
> backups vary significantly depending on needs/etc.
>
> Regards,
> Adam
>
>
Here is a summary of what I like to do. I want to backup files on to 
(multiple) disks that will be loaded on to a USB dock. Simple one to one 
copy is all I am looking for. I am not interested in full vs. 
incremental or keeping versions of files for restore. My data is just 
movies and songs. All I want is a SW that understands links (to avoid 
duplicates) and copy files in batch on to multiple disks. I want content 
of each (backup) disk to be independent. This way if one backup disk 
dies, I have all other files unaffected by this failure. The only reason 
I did not try multi-volume tar is the lack of independence across disks.

I suppose this is a backup question rather than RAID question. I asked 
here because the size of RAID volumes make it impossible to back up to a 
single drive and this is a more common problem RAID world rather than 
general user forum in a backup mailing list.

Ramesh

Re: How to backup of large md raid volumes?

[Victor Helmholtz]

> On 18 May 2017, at 22:37, Ram Ramesh <rramesh2400@gmail.com> wrote:
> 
> Any one have a method to backup large volumes like md raid6 (16TB)? Since the backup will not fit in one disk (in many cases and mine too) I am wondering, if there is a known/easy technique to backup using multiple usb hard drives. I googed and found a few fancy backup utils/systems like Amanda etc. They are overkill for me. I am choosing not to back up simply because of the complexity of setup as the data in my RAID volume is NOT precious and can be replaced with a week of effort.
> 
> If any one can think of some thing simple, please point me to it and I will do the reading to figure out.
> 
> Thanks
> Ramesh
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

If you are looking for the simplest solution you can use tar with --multi-volume option. I use it to create tape backups, but it can be used with hard drives as well. Detailed description of this tar option is available at https://www.gnu.org/software/tar/manual/html_section/tar_79.html

Regards
Victor

Re: How to backup of large md raid volumes?

[Victor Helmholtz]

> On 19 May 2017, at 04:39, Ram Ramesh <rramesh2400@gmail.com> wrote:
> 
> On 05/18/2017 08:34 PM, Adam Goryachev wrote:
>> On 19/05/17 07:37, Ram Ramesh wrote:
>>> Any one have a method to backup large volumes like md raid6 (16TB)? Since the backup will not fit in one disk (in many cases and mine too) I am wondering, if there is a known/easy technique to backup using multiple usb hard drives. I googed and found a few fancy backup utils/systems like Amanda etc. They are overkill for me. I am choosing not to back up simply because of the complexity of setup as the data in my RAID volume is NOT precious and can be replaced with a week of effort.
>>> 
>>> If any one can think of some thing simple, please point me to it and I will do the reading to figure out.
>>> 
>> I suppose it depends on what you want to achieve. There are various options such as mirroring to another raid6 array (ie, RAID60) but really that isn't a backup, it's another replica.
>> I use backuppc for my backups, it works well under linux with rsync, I'm sure there are many various options (including amanda which I've used in the past). Ultimately, it depends on your requirements, backups vary significantly depending on needs/etc.
>> 
>> Regards,
>> Adam
>> 
>> 
> Here is a summary of what I like to do. I want to backup files on to (multiple) disks that will be loaded on to a USB dock. Simple one to one copy is all I am looking for. I am not interested in full vs. incremental or keeping versions of files for restore. My data is just movies and songs. All I want is a SW that understands links (to avoid duplicates) and copy files in batch on to multiple disks. I want content of each (backup) disk to be independent. This way if one backup disk dies, I have all other files unaffected by this failure. The only reason I did not try multi-volume tar is the lack of independence across disks.
> 
> I suppose this is a backup question rather than RAID question. I asked here because the size of RAID volumes make it impossible to back up to a single drive and this is a more common problem RAID world rather than general user forum in a backup mailing list.
> 
> Ramesh
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

I have never tried it myself, but my understanding is that in multi-volume GNU tar archive each volume is independent. Tar manual states:
> Each volume is itself a valid GNU tar archive, so it can be read without any special options. Consequently any file member residing entirely on one volume can be extracted or otherwise operated upon without needing the other volume. Sure enough, to extract a split member you would need all volumes its parts reside on.

Regards
Victor

Re: How to backup of large md raid volumes?

[Roman Mamedov]

On Thu, 18 May 2017 22:39:59 -0500
Ram Ramesh <rramesh2400@gmail.com> wrote:

> Here is a summary of what I like to do. I want to backup files on to 
> (multiple) disks that will be loaded on to a USB dock. Simple one to one 
> copy is all I am looking for. I am not interested in full vs. 
> incremental or keeping versions of files for restore. My data is just 
> movies and songs. All I want is a SW that understands links (to avoid 
> duplicates) and copy files in batch on to multiple disks. I want content 
> of each (backup) disk to be independent. This way if one backup disk 
> dies, I have all other files unaffected by this failure. The only reason 
> I did not try multi-volume tar is the lack of independence across disks.

What I'm doing is storing files on my primary storage in folders according to
where they are backed up to.

E.g. "[back up of this is to be stored] On USB disk A", "B", etc.

In your case that would be:

/storage/OnDiskA/Movies/......
/storage/OnDiskA/Music/......

/storage/OnDiskB/Movies/......


-- 
With respect,
Roman

Re: How to backup of large md raid volumes?

[Roman Mamedov]

On Thu, 18 May 2017 22:39:59 -0500
Ram Ramesh <rramesh2400@gmail.com> wrote:

> Here is a summary of what I like to do. I want to backup files on to 
> (multiple) disks that will be loaded on to a USB dock. Simple one to one 
> copy is all I am looking for. I am not interested in full vs. 
> incremental or keeping versions of files for restore. My data is just 
> movies and songs. All I want is a SW that understands links (to avoid 
> duplicates) and copy files in batch on to multiple disks. I want content 
> of each (backup) disk to be independent. This way if one backup disk 
> dies, I have all other files unaffected by this failure. The only reason 
> I did not try multi-volume tar is the lack of independence across disks.

What I'm doing is storing files on my primary storage in folders according to
where they are backed up to.

E.g. "[back up of this is to be stored] On USB disk A", "B", etc.

In your case that would be:

/storage/OnDiskA/Movies/......
/storage/OnDiskA/Music/......

/storage/OnDiskB/Movies/......
/storage/OnDiskB/Software/......

(here I hit some wicked hotkey which causes the message to be saved and sent
prematurely :D)

/storage/OnDiskC/Photos/......

etc.

After that you can set up simple rsync to synchronize each folder to its
corresponding disk (though mine are network locations of various kind, not
transient USB devices, so I can set up rsync by crontab).

And for convenience in day-to-day use, you can merge all those folders into
one using one of the Union FS tools, such as mhddfs: https://romanrm.net/mhddfs

Running it with /storage/DiskA/,/storage/DiskB/,/storage/DiskC/ and mounting
them into /storage/all/, you would have 

/storage/all/Movies/
/storage/all/Music/
/storage/all/Software/
/storage/all/Photos/

conveniently within one directory, i.e. directory trees from all the "Disk"
dirs overlaid onto each other. You do not have to use mhddfs to write through
it (since with one large storage it doesn't help you manage space), can just
keep using the individual dirs for precise allocation of what you want backed
up where.

Perhaps there are backup tools which would let you split backups without also
separating them in primary storage, but this is the way I found to be the
simplest (regular rsync), most transparent (can be inspected with any file
manager) and easy to work with (just drag files and dirs around).

-- 
With respect,
Roman

Re: How to backup of large md raid volumes?

[Reindl Harald]

Am 18.05.2017 um 23:37 schrieb Ram Ramesh:
> Any one have a method to backup large volumes like md raid6 (16TB)? 
> Since the backup will not fit in one disk (in many cases and mine too) I 
> am wondering, if there is a known/easy technique to backup using 
> multiple usb hard drives. I googed and found a few fancy backup 
> utils/systems like Amanda etc. They are overkill for me. I am choosing 
> not to back up simply because of the complexity of setup as the data in 
> my RAID volume is NOT precious and can be replaced with a week of effort.
> 
> If any one can think of some thing simple, please point me to it and I 
> will do the reading to figure out

https://www.amazon.com/16TB-Desktop-External-Drive-WDBLWE0160JCH-NESN/dp/B01B6BN1CU

499$ is not that much for a 16 TB external backup disk and whn you 
format it with BTRFS and enable compression you most likely have a lot 
fo space and your backup finally is just a single rsync command

Re: How to backup of large md raid volumes?

[Wols Lists]

On 19/05/17 06:49, Reindl Harald wrote:
> 
> 
> Am 18.05.2017 um 23:37 schrieb Ram Ramesh:
>> Any one have a method to backup large volumes like md raid6 (16TB)?
>> Since the backup will not fit in one disk (in many cases and mine too)
>> I am wondering, if there is a known/easy technique to backup using
>> multiple usb hard drives. I googed and found a few fancy backup
>> utils/systems like Amanda etc. They are overkill for me. I am choosing
>> not to back up simply because of the complexity of setup as the data
>> in my RAID volume is NOT precious and can be replaced with a week of
>> effort.
>>
>> If any one can think of some thing simple, please point me to it and I
>> will do the reading to figure out
> 
> https://www.amazon.com/16TB-Desktop-External-Drive-WDBLWE0160JCH-NESN/dp/B01B6BN1CU
> 
> 
> 499$ is not that much for a 16 TB external backup disk and whn you
> format it with BTRFS and enable compression you most likely have a lot
> fo space and your backup finally is just a single rsync command

My only reaction to that is it's not a 16TB disk. It's a 16TB raid-0
array. We had a similar setup ages ago (when 500GB was huge) with a
500GB external USB drive. One of the 250GB drives failed and the entire
array was lost.

I'd be inclined to try and get an e-SATA or USB-3 dock, and just back up
on to independent 8TB drives. Mind you, at £70 for a 3TB Barracuda, $500
for 16TB of disk doesn't sound pricey at all ...

One thing to watch out for, it sounds like the OP has mostly music and
videos, so compression is likely to INCREASE the disk space required...

Cheers,
Wol

Re: How to backup of large md raid volumes?

[Reindl Harald]

Am 20.05.2017 um 20:00 schrieb Wols Lists:
> On 19/05/17 06:49, Reindl Harald wrote:
>>
>>
>> Am 18.05.2017 um 23:37 schrieb Ram Ramesh:
>>> Any one have a method to backup large volumes like md raid6 (16TB)?
>>> Since the backup will not fit in one disk (in many cases and mine too)
>>> I am wondering, if there is a known/easy technique to backup using
>>> multiple usb hard drives. I googed and found a few fancy backup
>>> utils/systems like Amanda etc. They are overkill for me. I am choosing
>>> not to back up simply because of the complexity of setup as the data
>>> in my RAID volume is NOT precious and can be replaced with a week of
>>> effort.
>>>
>>> If any one can think of some thing simple, please point me to it and I
>>> will do the reading to figure out
>>
>> https://www.amazon.com/16TB-Desktop-External-Drive-WDBLWE0160JCH-NESN/dp/B01B6BN1CU
>>
>>
>> 499$ is not that much for a 16 TB external backup disk and whn you
>> format it with BTRFS and enable compression you most likely have a lot
>> fo space and your backup finally is just a single rsync command
> 
> My only reaction to that is it's not a 16TB disk. It's a 16TB raid-0
> array. We had a similar setup ages ago (when 500GB was huge) with a
> 500GB external USB drive. One of the 250GB drives failed and the entire
> array was lost

well, it's a *backup* drive - when it's gone it's gone

we are using a bunch of them with 4 TB since 2011 for offsite-backups of 
the whole company cluster and they are a RAID0 too

Re: How to backup of large md raid volumes?

[Reindl Harald]

Am 20.05.2017 um 20:00 schrieb Wols Lists:
> I'd be inclined to try and get an e-SATA or USB-3 dock, and just back up
> on to independent 8TB drives. Mind you, at £70 for a 3TB Barracuda, $500
> for 16TB of disk doesn't sound pricey at all ...

yes but it don't make much fun if you need to spread your backups to 
several disks when you can have just one with all

> One thing to watch out for, it sounds like the OP has mostly music and
> videos, so compression is likely to INCREASE the disk space required...

yo are confusing "compress-force=zlib" with "compress=zlib" - read the 
BTRFS manuals, it's practically the same as comp-lzo for openvpn - 
uncompressable data are not compressed

Re: How to backup of large md raid volumes?

[Ram Ramesh]

On 05/21/2017 12:04 PM, Chris Schanzle wrote:
> On 05/18/2017 11:39 PM, Ram Ramesh wrote:
>> On 05/18/2017 08:34 PM, Adam Goryachev wrote:
>>> On 19/05/17 07:37, Ram Ramesh wrote:
>>>> Any one have a method to backup large volumes like md raid6 (16TB)? 
>>>> Since the backup will not fit in one disk (in many cases and mine 
>>>> too) I am wondering, if there is a known/easy technique to backup 
>>>> using multiple usb hard drives. I googed and found a few fancy 
>>>> backup utils/systems like Amanda etc. They are overkill for me. I 
>>>> am choosing not to back up simply because of the complexity of 
>>>> setup as the data in my RAID volume is NOT precious and can be 
>>>> replaced with a week of effort.
>>>>
>>>> If any one can think of some thing simple, please point me to it 
>>>> and I will do the reading to figure out.
>>>>
>>> I suppose it depends on what you want to achieve. There are various 
>>> options such as mirroring to another raid6 array (ie, RAID60) but 
>>> really that isn't a backup, it's another replica.
>>> I use backuppc for my backups, it works well under linux with rsync, 
>>> I'm sure there are many various options (including amanda which I've 
>>> used in the past). Ultimately, it depends on your requirements, 
>>> backups vary significantly depending on needs/etc.
>>>
>>> Regards,
>>> Adam
>>>
>>>
>> Here is a summary of what I like to do. I want to backup files on to 
>> (multiple) disks that will be loaded on to a USB dock. Simple one to 
>> one copy is all I am looking for. I am not interested in full vs. 
>> incremental or keeping versions of files for restore. My data is just 
>> movies and songs. All I want is a SW that understands links (to avoid 
>> duplicates) and copy files in batch on to multiple disks. I want 
>> content of each (backup) disk to be independent. This way if one 
>> backup disk dies, I have all other files unaffected by this failure. 
>> The only reason I did not try multi-volume tar is the lack of 
>> independence across disks.
>>
>> I suppose this is a backup question rather than RAID question. I 
>> asked here because the size of RAID volumes make it impossible to 
>> back up to a single drive and this is a more common problem RAID 
>> world rather than general user forum in a backup mailing list.
>>
>> Ramesh
>
> Thanks for the additional info on your requirements.
>
> Rsync will be your best/fastest tool, as writing 16 TB of data over 
> USB will likely take days for a full backup...and that seems silly 
> when you probably already have 90% of the data already.
>
> If one drive failures are a concern, why not use RAID5 for your group 
> of USB backup disks?  Might take you a bit more effort 
> starting/stopping the array, but I'm sure you'll enjoy the large 
> single volume/filesystem in the long run.   Use a fairly large 
> --chunk= size for your large files.  I have had very positive 
> experience with XFS for large multimedia filesystems (mythtv user and 
> 20TB backup servers).
> If the drives you choose to use are desktop grade, be sure to increase 
> the timeouts.  Search this list archives for Chris Murphy's subject: 
> "Re: URE, link resets, user hostile defaults".
>
The only problem with raid is that all disks need to be present to 
create md. I only have the ability to load two disks and the disks are 
fairly small compared to RAID l (3TB vs. 16TB) So RAID5 won't work. 
Besides, I do not want to use RAID to backup RAID as my worry is md 
going bad and preventing access to my files. It is not disk that I am 
worried about mdadm layout and requirement that many disk to be alive to 
get even one file.