No space on empty, degraded raid10

No space on empty, degraded raid10

[Or Tal]

Hi,

I've created a new raid10 array from 4, 4TB drives in order to migrate
old data to it.
As I didn't have enough sata ports, I:
- disconnected one of the raid10 disks to free a sata port,
- connected an old disk I wanted to migrate,
- mounted the array with -o degraded
- copied the data it it.

After about 2MB I got a "no space left on device message."
btrfs fi df showed strange things - much less space in every category
(about 8GB?) and none of then was full.

Ubuntu 14.10 beta - linux 3.16.0-14

Best regards,
Or Tal.

Re: No space on empty, degraded raid10

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1259 bytes --]

On 2014-09-07 16:38, Or Tal wrote:
> Hi,
> 
> I've created a new raid10 array from 4, 4TB drives in order to migrate
> old data to it.
> As I didn't have enough sata ports, I:
> - disconnected one of the raid10 disks to free a sata port,
> - connected an old disk I wanted to migrate,
> - mounted the array with -o degraded
> - copied the data it it.
> 
> After about 2MB I got a "no space left on device message."
> btrfs fi df showed strange things - much less space in every category
> (about 8GB?) and none of then was full.
> 
> Ubuntu 14.10 beta - linux 3.16.0-14
Yeah, RAID10 doesn't really work in degraded mode (even if you have two
disks that have stripes from the same copy).  The approach that would be
needed for what you want to do is:
 1. Make a BTRFS RAID1 filesystem with _3_ new drives
 2. Connect one of the old disks
 3. Transfer data from old disk to new filesystem
 4. After repeating steps 2 and 3 for each old disk, connect the final
new disk, add it to the filesystem, and rebalance with '-dconvert=raid10
-mconvert=raid10'

Also, I've found out the hard way that system chunks really should be
RAID1, _NOT_ RAID10, otherwise it's very likely that the filesystem
won't mount at all if you lose 2 disks.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2455 bytes --]

Re: No space on empty, degraded raid10

[Russell Coker]

On Mon, 8 Sep 2014, Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:
> Also, I've found out the hard way that system chunks really should be
> RAID1, NOT RAID10, otherwise it's very likely that the filesystem
> won't mount at all if you lose 2 disks.

Why would that be different?

In a RAID-1 you expect system problems if 2 disks fail, why would RAID-10 be 
different?

Also it would be nice if there was a N-way mirror option for system data.  As 
such data is tiny (32MB on the 120G filesystem in my workstation) the space 
used by having a copy on every disk in the array shouldn't matter.

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

Re: No space on empty, degraded raid10

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1280 bytes --]

On 2014-09-11 02:40, Russell Coker wrote:
> On Mon, 8 Sep 2014, Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:
>> Also, I've found out the hard way that system chunks really should be
>> RAID1, NOT RAID10, otherwise it's very likely that the filesystem
>> won't mount at all if you lose 2 disks.
> 
> Why would that be different?
> 
> In a RAID-1 you expect system problems if 2 disks fail, why would RAID-10 be 
> different?
That's still the case, but in a RAID1 with four disks, of the six
different pairs of two disks you could lose, only one will make the
filesystem un-mountable, whereas for a four disk RAID10, there are two
different pairs of two disks you could lose to make the filesystem
un-mountable.  In haven't run the numbers for higher numbers of disks,
but things are likely not better, because if you lose both copies of the
same stripe, things will fail.
> 
> Also it would be nice if there was a N-way mirror option for system data.  As 
> such data is tiny (32MB on the 120G filesystem in my workstation) the space 
> used by having a copy on every disk in the array shouldn't matter.
> 
N-way mirroring is in the queue for after RAID5/6 work; ideally, once it
is ready, mkfs should default to one copy per disk in the filesystem.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2455 bytes --]

Re: No space on empty, degraded raid10

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 802 bytes --]

On Thu, Sep 11, 2014 at 07:19:00AM -0400, Austin S Hemmelgarn wrote:
> On 2014-09-11 02:40, Russell Coker wrote:
> > Also it would be nice if there was a N-way mirror option for system data.  As 
> > such data is tiny (32MB on the 120G filesystem in my workstation) the space 
> > used by having a copy on every disk in the array shouldn't matter.
> 
> N-way mirroring is in the queue for after RAID5/6 work; ideally, once it
> is ready, mkfs should default to one copy per disk in the filesystem.

   Why change the default from 2-copies, which it's been for years?

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
             --- Ceci est un travail pour l'Australien. ---              

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: No space on empty, degraded raid10

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1023 bytes --]

On 2014-09-11 07:38, Hugo Mills wrote:
> On Thu, Sep 11, 2014 at 07:19:00AM -0400, Austin S Hemmelgarn wrote:
>> On 2014-09-11 02:40, Russell Coker wrote:
>>> Also it would be nice if there was a N-way mirror option for system data.  As 
>>> such data is tiny (32MB on the 120G filesystem in my workstation) the space 
>>> used by having a copy on every disk in the array shouldn't matter.
>>
>> N-way mirroring is in the queue for after RAID5/6 work; ideally, once it
>> is ready, mkfs should default to one copy per disk in the filesystem.
> 
>    Why change the default from 2-copies, which it's been for years?

Sorry about the ambiguity in my statement, I meant that the default for
system chunks should be one copy per disk in the filesystem.  If you
don't have a copy of the system chunks, then you essentially don't have
a filesystem, and that means that BTRFS RAID6 can't provide true
resilience against 2 disks failing catastrophically unless there are at
least 3 copies of the system chunks.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2455 bytes --]

Re: No space on empty, degraded raid10

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1388 bytes --]

On Thu, Sep 11, 2014 at 08:06:21AM -0400, Austin S Hemmelgarn wrote:
> On 2014-09-11 07:38, Hugo Mills wrote:
> > On Thu, Sep 11, 2014 at 07:19:00AM -0400, Austin S Hemmelgarn wrote:
> >> On 2014-09-11 02:40, Russell Coker wrote:
> >>> Also it would be nice if there was a N-way mirror option for system data.  As 
> >>> such data is tiny (32MB on the 120G filesystem in my workstation) the space 
> >>> used by having a copy on every disk in the array shouldn't matter.
> >>
> >> N-way mirroring is in the queue for after RAID5/6 work; ideally, once it
> >> is ready, mkfs should default to one copy per disk in the filesystem.
> > 
> >    Why change the default from 2-copies, which it's been for years?
> 
> Sorry about the ambiguity in my statement, I meant that the default for
> system chunks should be one copy per disk in the filesystem.  If you
> don't have a copy of the system chunks, then you essentially don't have
> a filesystem, and that means that BTRFS RAID6 can't provide true
> resilience against 2 disks failing catastrophically unless there are at
> least 3 copies of the system chunks.

   Aah, OK. That makes perfect sense, then.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
    --- Some days, it's just not worth gnawing through the straps ---    

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]