corrupted RAID1: unsuccessful recovery / help needed

corrupted RAID1: unsuccessful recovery / help needed

[Lukas Pirl]

TL;DR: RAID1 does not recover, I guess the interesting part in the stack 
trace is:

   Call Trace:
   [<ffffffffc0377e60>] __del_reloc_root+0x30/0x100 [btrfs]
   [<ffffffffc0377f55>] free_reloc_roots+0x25/0x40 [btrfs]
   [<ffffffffc037e47e>] merge_reloc_roots+0x18e/0x240 [btrfs]
   [<ffffffffc037f1f4>] btrfs_recover_relocation+0x374/0x420 [btrfs]
   [<ffffffffc0328f6d>] open_ctree+0x1b7d/0x23e0 [btrfs]
   [<ffffffffc02ff72e>] btrfs_mount+0x94e/0xa70 [btrfs]
   [<ffffffff813bc0c5>] ? find_next_bit+0x15/0x20
   [<ffffffff811f1078>] mount_fs+0x38/0x160
   …

Hello list.

I'd appreciate some help for repairing a corrupted RAID1.

Setup:
* Linux 4.2.0-12, Btrfs v3.17, `btrfs fi show`:
   uuid: 5be372f5-5492-4f4b-b641-c14f4ad8ae23
   Total devices 6 FS bytes used 2.87TiB
   devid 1 size 931.51GiB used 636.00GiB path /dev/mapper/WD-WCC4J7AFLTSZ
   devid 2 size 931.51GiB used 634.03GiB path /dev/mapper/WD-WCAU45343103
   devid 3 size   1.82TiB used   1.53TiB path /dev/mapper/WD-WCAVY6423276
   devid 4 size   1.82TiB used   1.53TiB path /dev/mapper/WD-WCAZAF872578
   devid 6 size   1.82TiB used   1.05TiB path /dev/mapper/WD-WMC4M0H3Z5UK
   *** Some devices missing
* disks are dm-crypted

What happened:
* devid 5 started to die (slowly)
* added a new disk (devid 6) and tried `btrfs device delete`
* failed with kernel crashes (guess:) due to heavy IO errors
* removed devid 5 from /dev (deactivated in dm-crypt)
* tried `btrfs balance`
   * interrupted multiple times due to kernel crashes
     (probably due to semi-corrupted file system?)
* file system did not mount anymore after a required hard-reset
* no successful recovery so far:
   if not read-only, kernel IO blocks eventually (hard-reset required)
* tried:
   * `-o degraded`
     -> IO freeze, kernel log: http://pastebin.com/Rzrp7XeL
   * `-o degraded,recovery`
     -> IO freeze, kernel log: http://pastebin.com/VemHfnuS
   * `-o degraded,recovery,ro`
     -> file system accessible, system stable
* going rw again does not fix the problem

I did not btrfs-zero-log so far because my oops did not look very
similar to the one in the Wiki and I did not want to risk to make
recovery harder.

Thanks,

Lukas

Re: corrupted RAID1: unsuccessful recovery / help needed

[Duncan]

Lukas Pirl posted on Mon, 26 Oct 2015 19:19:50 +1300 as excerpted:

> TL;DR: RAID1 does not recover, I guess the interesting part in the stack
> trace is: [elided, I'm not a dev so it's little help to me]
> 
> I'd appreciate some help for repairing a corrupted RAID1.
> 
> Setup:
> * Linux 4.2.0-12, Btrfs v3.17, 
> `btrfs fi show`:
>    uuid: 5be372f5-5492-4f4b-b641-c14f4ad8ae23
>    Total devices 6 FS bytes used 2.87TiB
>    devid 1 size 931.51GiB used 636.00GiB path /dev/mapper/[...]
>    devid 2 size 931.51GiB used 634.03GiB path /dev/mapper/
>    devid 3 size   1.82TiB used   1.53TiB path /dev/mapper/
>    devid 4 size   1.82TiB used   1.53TiB path /dev/mapper/
>    devid 6 size   1.82TiB used   1.05TiB path /dev/mapper/
>    *** Some devices missing
> * disks are dm-crypted

FWIW... Older btrfs userspace such as your v3.17 is "OK" for normal 
runtime use, assuming you don't need any newer features, as in normal 
runtime, it's the kernel code doing the real work and userspace for the 
most part simply makes the appropriate kernel calls to do that work.

But, once you get into a recovery situation like the one you're in now, 
current userspace becomes much more important, as the various things 
you'll do to attempt recovery rely far more on userspace code directly 
accessing the filesystem, and it's only the newest userspace code that 
has the latest fixes.

So for a recovery situation, the newest userspace release (4.2.2 at 
present) as well as a recent kernel is recommended, and depending on the 
problem, you may at times need to run integration or apply patches on top 
of that.

> What happened:
> * devid 5 started to die (slowly)
> * added a new disk (devid 6) and tried `btrfs device delete`
> * failed with kernel crashes (guess:) due to heavy IO errors
> * removed devid 5 from /dev (deactivated in dm-crypt)
> * tried `btrfs balance`
>    * interrupted multiple times due to kernel crashes
>      (probably due to semi-corrupted file system?)
> * file system did not mount anymore after a required hard-reset
> * no successful recovery so far:
>    if not read-only, kernel IO blocks eventually (hard-reset required)
> * tried:
>    * `-o degraded`
>      -> IO freeze, kernel log: http://pastebin.com/Rzrp7XeL
>    * `-o degraded,recovery`
>      -> IO freeze, kernel log: http://pastebin.com/VemHfnuS
>    * `-o degraded,recovery,ro`
>      -> file system accessible, system stable
> * going rw again does not fix the problem
> 
> I did not btrfs-zero-log so far because my oops did not look very
> similar to the one in the Wiki and I did not want to risk to make
> recovery harder.

General note about btrfs and btrfs raid.  Given that btrfs itself remains 
a "stabilizing, but not yet fully mature and stable filesystem", while 
btrfs raid will often let you recover from a bad device, sometimes that 
recovery is in the form of letting you mount ro, so you can access the 
data and copy it elsewhere, before blowing away the filesystem and 
starting over.

Back to the problem at hand.  Current btrfs has a known limitation when 
operating in degraded mode.  That being, a btrfs raid may be write-
mountable only once, degraded, after which it can only be read-only 
mounted.  This is because under certain circumstances in degraded mode, 
btrfs will fall back from its normal raid mode to single mode chunk 
allocation for new writes, and once there's single-mode chunks on the 
filesystem, btrfs mount isn't currently smart enough to check that all 
chunks are actually available on present devices, and simply jumps to the 
conclusion that there's single mode chunks on the missing device(s) as 
well, so refuses to mount writable after that in ordered to prevent 
further damage to the filesystem and preserve the ability to mount at 
least ro, to copy off what isn't damaged.

There's a patch in the pipeline for this problem, that checks individual 
chunks instead of leaping to conclusions based on the presence of single-
mode chunks on a degraded filesystem with missing devices.  If that's 
your only problem (which the backtraces might reveal but I as a non-dev 
btrfs user can't tell), the patches should let you mount writable.

But that patch isn't in kernel 4.2.  You'll need at least kernel 4.3-rc, 
and possibly btrfs integration, or to cherrypick the patches onto 4.2.


Meanwhile, in keeping with the admin's rule on backups, by definition, if 
you valued the data more than the time and resources necessary for a 
backup, by definition, you have a backup available, otherwise, by 
definition, you valued the data less than the time and resources 
necessary to back it up.

Therefore, no worries.  Regardless of the fate of the data, you saved 
what your actions declared of most valuable to you, either the data, or 
the hassle and resources cost of the backup you didn't do.  As such, if 
you don't have a backup (or if you do but it's outdated), the data at 
risk of loss is by definition of very limited value.

That said, it appears you don't even have to worry about loss of that 
very limited value data, since mounting degraded,recovery,ro gives you 
stable access to it, and you can use the opportunity provided to copy it 
elsewhere, at least to the extent that the data we already know is of 
limited value is even worth the hassle of doing that.

Which is exactly what I'd do.  Actually, I've had to resort to btrfs 
restore[1] a couple times when the filesystem wouldn't mount at all, so 
the fact that you can mount it degraded,recovery,ro, already puts you 
ahead of the game. =:^)

So yeah, first thing, since you have the opportunity, unless your backups 
are sufficiently current that it's not worth the trouble, copy off the 
data while you can.

Then, unless you wish to keep the filesystem around in case the devs want 
to use it to improve btrfs' recovery system, I'd just blow it away and 
start over, restoring the data from backup once you have a fresh 
filesystem to restore to.  That's the simplest and fastest way to a fully 
working system once again, and what I did here after using btrfs restore 
to recover the delta between current and my backups.

---
[1] Btrfs restore: Yes, I have backups, but I don't always keep them 
current.  To the extent that I risk losing the difference between current 
and backup, my actions obviously define that difference as not worth the 
hassle cost of more frequent backups vs. the risk.  But while my actions 
define that delta data as of relatively low value, it's not of /no/ 
value, and to the extent btrfs restore allows me to recover it, I 
appreciate that I can do so, avoiding the loss of the delta between my 
backup and what was current.  Of course, that lowers the risk of loss 
even further, letting me put off updating the backups even longer if I 
wanted, but I haven't actually done so.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: corrupted RAID1: unsuccessful recovery / help needed

[Lukas Pirl]

TL;DR: thanks but recovery still preferred over recreation.

Hello Duncan and thanks for your reply!

On 10/26/2015 09:31 PM, Duncan wrote:
> FWIW... Older btrfs userspace such as your v3.17 is "OK" for normal
> runtime use, assuming you don't need any newer features, as in normal
> runtime, it's the kernel code doing the real work and userspace for the
> most part simply makes the appropriate kernel calls to do that work.
 >
> But, once you get into a recovery situation like the one you're in now,
> current userspace becomes much more important, as the various things
> you'll do to attempt recovery rely far more on userspace code directly
> accessing the filesystem, and it's only the newest userspace code that
> has the latest fixes.
>
> So for a recovery situation, the newest userspace release (4.2.2 at
> present) as well as a recent kernel is recommended, and depending on the
> problem, you may at times need to run integration or apply patches on top
> of that.

I am willing to update before trying further repairs. Is e.g. "balance" 
also influenced by the userspace tools or does the kernel the actual work?

> General note about btrfs and btrfs raid.  Given that btrfs itself remains
> a "stabilizing, but not yet fully mature and stable filesystem", while
> btrfs raid will often let you recover from a bad device, sometimes that
> recovery is in the form of letting you mount ro, so you can access the
> data and copy it elsewhere, before blowing away the filesystem and
> starting over.

If there is one subvolume that contains all other (read only) snapshots 
and there is insufficient storage to copy them all separately:
Is there an elegant way to preserve those when moving the data across disks?

> Back to the problem at hand.  Current btrfs has a known limitation when
> operating in degraded mode.  That being, a btrfs raid may be write-
> mountable only once, degraded, after which it can only be read-only
> mounted.  This is because under certain circumstances in degraded mode,
> btrfs will fall back from its normal raid mode to single mode chunk
> allocation for new writes, and once there's single-mode chunks on the
> filesystem, btrfs mount isn't currently smart enough to check that all
> chunks are actually available on present devices, and simply jumps to the
> conclusion that there's single mode chunks on the missing device(s) as
> well, so refuses to mount writable after that in ordered to prevent
> further damage to the filesystem and preserve the ability to mount at
> least ro, to copy off what isn't damaged.
>
> There's a patch in the pipeline for this problem, that checks individual
> chunks instead of leaping to conclusions based on the presence of single-
> mode chunks on a degraded filesystem with missing devices.  If that's
> your only problem (which the backtraces might reveal but I as a non-dev
> btrfs user can't tell), the patches should let you mount writable.

Interesting, thanks for the insights.

> But that patch isn't in kernel 4.2.  You'll need at least kernel 4.3-rc,
> and possibly btrfs integration, or to cherrypick the patches onto 4.2.

Well, before digging into that, a hint that this is actually the case 
would be appreciated. :)

> Meanwhile, in keeping with the admin's rule on backups, by definition, if
> you valued the data more than the time and resources necessary for a
> backup, by definition, you have a backup available, otherwise, by
> definition, you valued the data less than the time and resources
> necessary to back it up.
>
> Therefore, no worries.  Regardless of the fate of the data, you saved
> what your actions declared of most valuable to you, either the data, or
> the hassle and resources cost of the backup you didn't do.  As such, if
> you don't have a backup (or if you do but it's outdated), the data at
> risk of loss is by definition of very limited value.
>
> That said, it appears you don't even have to worry about loss of that
> very limited value data, since mounting degraded,recovery,ro gives you
> stable access to it, and you can use the opportunity provided to copy it
> elsewhere, at least to the extent that the data we already know is of
> limited value is even worth the hassle of doing that.
>
> Which is exactly what I'd do.  Actually, I've had to resort to btrfs
> restore[1] a couple times when the filesystem wouldn't mount at all, so
> the fact that you can mount it degraded,recovery,ro, already puts you
> ahead of the game. =:^)
>
> So yeah, first thing, since you have the opportunity, unless your backups
> are sufficiently current that it's not worth the trouble, copy off the
> data while you can.
>
> Then, unless you wish to keep the filesystem around in case the devs want
> to use it to improve btrfs' recovery system, I'd just blow it away and
> start over, restoring the data from backup once you have a fresh
> filesystem to restore to.  That's the simplest and fastest way to a fully
> working system once again, and what I did here after using btrfs restore
> to recover the delta between current and my backups.

Thanks for all the elaborations. I guess there are also other valid 
definitions of making backups out there – some that determine the amount 
and the types of redundancy by additionally taking factors like the 
anticipated risk of a failure or the severity of a failure into 
consideration.

However, you are perfectly correct with your advice to 
create/update/verify all backups as it is (still) possible.

Besides that, I'd still be willing to restore the file system and to 
provide additional information to devs.

Cheers,

Lukas

Re: corrupted RAID1: unsuccessful recovery / help needed

[Duncan]

Lukas Pirl posted on Fri, 30 Oct 2015 10:43:41 +1300 as excerpted:

> Is e.g. "balance" also influenced by the userspace tools or does
> the kernel the actual work?

btrfs balance is done "online", that is, on the (writable-)mounted 
filesystem, and the kernel does the real work.  It's the tools that work 
on the unmounted filesystem, btrfs check, btrfs restore, btrfs rescue, 
etc, where the userspace code does the real work, and thus where being 
current and having all the latests userspace fixes is vital.

If you can't mount writable, you can't balance.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: corrupted RAID1: unsuccessful recovery / help needed

[Duncan]

Lukas Pirl posted on Fri, 30 Oct 2015 10:43:41 +1300 as excerpted:

> If there is one subvolume that contains all other (read only) snapshots
> and there is insufficient storage to copy them all separately:
> Is there an elegant way to preserve those when moving the data across
> disks?

AFAIK, no elegant way without a writable mount.

Tho I'm not sure, btrfs send, to a btrfs elsewhere using receive, may 
work, since you did specify read-only snapshots, which is what send 
normally works with in ordered to avoid changes to the snapshot while 
it's sending it.  My own use-case doesn't involve either snapshots or 
send/receive, however, so I'm not sure if send can work with a read-only 
filesystem or not, but I think its normal method of operation is to 
create those read-only snapshots itself, which of course would require a 
writable filesystem, so I'm guessing it won't work unless you can 
convince it to use the read-only mounts as-is.

The less elegant way would involve manual deduplication.  Copy one 
snapshot, then another, and dedup what hasn't changed between the two, 
then add a third and dedup again. ...  Depending on the level of dedup 
(file vs block level) and the level of change in your filesystem, this 
should ultimately take about the same level of space as a full backup 
plus a series of incrementals.


Meanwhile, this does reinforce the point that snapshots don't replace 
full backups, that being the reason I don't use them here, since if the 
filesystem goes bad, it'll very likely take all the snapshots with it.

Snapshots do tend to be pretty convenient, arguably /too/ convenient and 
near-zero-cost to make, as people then tend to just do scheduled 
snapshots, without thinking about their overhead and maintenance costs on 
the filesystem, until they already have problems.  I'm not sure if you 
are a regular list reader and have thus seen my normal spiel on btrfs 
snapshot scaling and recommended limits to avoid problems or not, so if 
not, here's a slightly condensed version...

Btrfs has scaling issues that appear when trying to manage too many 
snapshots.  These tend to appear first in tools like balance and check, 
where time to process a filesystem goes up dramatically as the number of 
snapshots increases, to the point where it can become entirely 
impractical to manage at all somewhere near the 100k snapshots range, and 
is already dramatically affecting runtime at 10k snapshots.

As a result, I recommend keeping per-subvol snapshots to 250-ish, which 
will allow snapshotting four subvolumes while still keeping total 
filesystem snapshots to 1000, or eight subvolumes at a filesystem total 
of 2000 snapshots, levels where the scaling issues should remain well 
within control.  And 250-ish snapshots per subvolume is actually very 
reasonable even with half-hour scheduled snapshotting, provided a 
reasonable scheduled snapshot thinning program is also implemented, 
cutting say to hourly after six hours, six-hourly after a day, 12 hourly 
after 2 days, daily after a week, and weekly after four weeks to a 
quarter (13 weeks).  Out beyond a quarter or two, certainly within a 
year, longer term backups to other media should be done, and snapshots 
beyond that can be removed entirely, freeing up the space the old 
snapshots kept locked down and helping to keep the btrfs healthy and 
functioning well within its practical scalability limits.

Because a balance that takes a month to complete because it's dealing 
with a few hundred k snapshots is in practice (for most people) not 
worthwhile to do at all, and also in practice, a year or even six months 
out, are you really going to care about the precise half-hour snapshot, 
or is the next daily or weekly snapshot going to be just as good, and a 
whole lot easier to find among a couple hundred snapshots than hundreds 
of thousands?

If you have far too many snapshots, perhaps this sort of thinning 
strategy will as well allow you to copy and dedup only key snapshots, say 
weekly plus daily for the last week, doing the backup thing manually, as 
well, modifying the thinning strategy accordingly if necessary to get it 
to fit.  Tho using the copy and dedup strategy above will still require 
at least double the full space of a single copy, plus the space necessary 
for each deduped snapshot copy you keep, since the dedup occurs after the 
copy.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: corrupted RAID1: unsuccessful recovery / help needed

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 5189 bytes --]

On Fri, Oct 30, 2015 at 10:58:47AM +0000, Duncan wrote:
> Lukas Pirl posted on Fri, 30 Oct 2015 10:43:41 +1300 as excerpted:
> 
> > If there is one subvolume that contains all other (read only) snapshots
> > and there is insufficient storage to copy them all separately:
> > Is there an elegant way to preserve those when moving the data across
> > disks?

   If they're read-only snapshots already, then yes:

sent=
for sub in *; do
   btrfs send $sent $sub | btrfs receive /where/ever
   sent="$sent -c$sub"
done

   That will preserve the shared extents between the subvols on the
receiving FS.

   If they're not read-only, then snapshotting each one again as RO
before sending would be the approach, but if your FS is itself RO,
that's not going to be possible, and you need to look at Duncan's
email.

   Hugo.

> AFAIK, no elegant way without a writable mount.
> 
> Tho I'm not sure, btrfs send, to a btrfs elsewhere using receive, may 
> work, since you did specify read-only snapshots, which is what send 
> normally works with in ordered to avoid changes to the snapshot while 
> it's sending it.  My own use-case doesn't involve either snapshots or 
> send/receive, however, so I'm not sure if send can work with a read-only 
> filesystem or not, but I think its normal method of operation is to 
> create those read-only snapshots itself, which of course would require a 
> writable filesystem, so I'm guessing it won't work unless you can 
> convince it to use the read-only mounts as-is.
> 
> The less elegant way would involve manual deduplication.  Copy one 
> snapshot, then another, and dedup what hasn't changed between the two, 
> then add a third and dedup again. ...  Depending on the level of dedup 
> (file vs block level) and the level of change in your filesystem, this 
> should ultimately take about the same level of space as a full backup 
> plus a series of incrementals.
> 
> 
> Meanwhile, this does reinforce the point that snapshots don't replace 
> full backups, that being the reason I don't use them here, since if the 
> filesystem goes bad, it'll very likely take all the snapshots with it.
> 
> Snapshots do tend to be pretty convenient, arguably /too/ convenient and 
> near-zero-cost to make, as people then tend to just do scheduled 
> snapshots, without thinking about their overhead and maintenance costs on 
> the filesystem, until they already have problems.  I'm not sure if you 
> are a regular list reader and have thus seen my normal spiel on btrfs 
> snapshot scaling and recommended limits to avoid problems or not, so if 
> not, here's a slightly condensed version...
> 
> Btrfs has scaling issues that appear when trying to manage too many 
> snapshots.  These tend to appear first in tools like balance and check, 
> where time to process a filesystem goes up dramatically as the number of 
> snapshots increases, to the point where it can become entirely 
> impractical to manage at all somewhere near the 100k snapshots range, and 
> is already dramatically affecting runtime at 10k snapshots.
> 
> As a result, I recommend keeping per-subvol snapshots to 250-ish, which 
> will allow snapshotting four subvolumes while still keeping total 
> filesystem snapshots to 1000, or eight subvolumes at a filesystem total 
> of 2000 snapshots, levels where the scaling issues should remain well 
> within control.  And 250-ish snapshots per subvolume is actually very 
> reasonable even with half-hour scheduled snapshotting, provided a 
> reasonable scheduled snapshot thinning program is also implemented, 
> cutting say to hourly after six hours, six-hourly after a day, 12 hourly 
> after 2 days, daily after a week, and weekly after four weeks to a 
> quarter (13 weeks).  Out beyond a quarter or two, certainly within a 
> year, longer term backups to other media should be done, and snapshots 
> beyond that can be removed entirely, freeing up the space the old 
> snapshots kept locked down and helping to keep the btrfs healthy and 
> functioning well within its practical scalability limits.
> 
> Because a balance that takes a month to complete because it's dealing 
> with a few hundred k snapshots is in practice (for most people) not 
> worthwhile to do at all, and also in practice, a year or even six months 
> out, are you really going to care about the precise half-hour snapshot, 
> or is the next daily or weekly snapshot going to be just as good, and a 
> whole lot easier to find among a couple hundred snapshots than hundreds 
> of thousands?
> 
> If you have far too many snapshots, perhaps this sort of thinning 
> strategy will as well allow you to copy and dedup only key snapshots, say 
> weekly plus daily for the last week, doing the backup thing manually, as 
> well, modifying the thinning strategy accordingly if necessary to get it 
> to fit.  Tho using the copy and dedup strategy above will still require 
> at least double the full space of a single copy, plus the space necessary 
> for each deduped snapshot copy you keep, since the dedup occurs after the 
> copy.
> 

-- 
Hugo Mills             | Beware geeks bearing GIFs
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: corrupted RAID1: unsuccessful recovery / help needed

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 4325 bytes --]

On 2015-10-30 06:58, Duncan wrote:
> Lukas Pirl posted on Fri, 30 Oct 2015 10:43:41 +1300 as excerpted:
>
>> If there is one subvolume that contains all other (read only) snapshots
>> and there is insufficient storage to copy them all separately:
>> Is there an elegant way to preserve those when moving the data across
>> disks?
>
> AFAIK, no elegant way without a writable mount.
>
> Tho I'm not sure, btrfs send, to a btrfs elsewhere using receive, may
> work, since you did specify read-only snapshots, which is what send
> normally works with in ordered to avoid changes to the snapshot while
> it's sending it.  My own use-case doesn't involve either snapshots or
> send/receive, however, so I'm not sure if send can work with a read-only
> filesystem or not, but I think its normal method of operation is to
> create those read-only snapshots itself, which of course would require a
> writable filesystem, so I'm guessing it won't work unless you can
> convince it to use the read-only mounts as-is.
Unless something has significantly changed since I last looked, send 
only works on existing snapshots and doesn't create any directly itself, 
and as such should work fine to send snapshots from a read-only 
filesystem.  In theory, you could use it to send all the snapshots at 
once, although that would probably take a long time, so you'll probably 
have to use a loop like the fragment of shell-script that Hugo suggested 
in his response.  That should result in an (almost) identical level of 
sharing.
>
> The less elegant way would involve manual deduplication.  Copy one
> snapshot, then another, and dedup what hasn't changed between the two,
> then add a third and dedup again. ...  Depending on the level of dedup
> (file vs block level) and the level of change in your filesystem, this
> should ultimately take about the same level of space as a full backup
> plus a series of incrementals.
If you're using duperemove (which is the only maintained dedupe tool I 
know of for BTRFS), then this will likely take a long time for any 
reasonable amount of data, and probably take up more space on the 
destination drive than it does on the source (while duperemove does 
block-based deduplication, it uses large chunks by default).
>
> Meanwhile, this does reinforce the point that snapshots don't replace
> full backups, that being the reason I don't use them here, since if the
> filesystem goes bad, it'll very likely take all the snapshots with it.
FWIW, while I don't use them directly myself as a backup, they are 
useful when doing a backup to get a guaranteed stable version of the 
filesystem being backed-up (this is also one of the traditional use 
cases for LVM snapshots, although those have a lot of different issues 
to deal with).  For local backups (I also do cloud-storage based remote 
backups, but local is what matters in this case because it's where I 
actually use send/receive and snapshots) I use two different methods 
depending on the amount of storage I have:
1. If I'm relatively limited on local storage (like in my laptop where 
the secondary internal disk is only 64G), I use a temporary snapshot to 
generate a SquashFS image of the system, which I then store on the 
secondary drive.
2. If I have a lot of spare space (like on my desktop where I have 4x 
1TB HDD's and 2x 128G SSD's), I make a snapshot of the filesystem, then 
use send/receive to transfer that to a backup filesystem on a separate 
disk.  I then keep the original snapshot around on the filesystem so I 
can do incremental send/receive to speed up future backups.
In both cases, I can directly boot my most recent backups if need be, 
and in the second case, I can actually use it to trivially regenerate 
the backed-up filesystems (by simply doing a send/receive in the 
opposite direction).

Beyond providing a stable system-image for backups, the only valid use 
case for snapshots in my opinion is to provide the equivalent to MS 
Windows' 'Restore Point' feature (which I'm pretty sure is done 
currently by RHEL and SLES if they are installed on BTRFS) and possibly 
'File History' for people who for some reason can't use real VCS or just 
need to store the last few revision (which is itself done by stuff like 
'snapper').



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: corrupted RAID1: unsuccessful recovery / help needed

[Lukas Pirl]

Thanks for the answers Duncan, Hugo and Austin.

I'll try a recovery as soon as I can and might (probably) come back to
the list then.

On 10/30/2015 11:58 PM, Duncan wrote:
> Meanwhile, this does reinforce the point that snapshots don't
> replace full backups, that being the reason I don't use them here,
> since if the filesystem goes bad, it'll very likely take all the
> snapshots with it.

It is very true that snapshots do not replace full backups (in case of
software or hardware failures). However, they are a very suited
strategy to recover from unintentional deletions (most likely human,
but could also be software failures)

Best,

Lukas

-- 
+49 174 940 74 71
+64 21 150 86 86
GPG: http://lukas-pirl.de/media/attachments/lukas_pirl.pub