raid0 confusion question

raid0 confusion question

[Tim Cuthbertson]

I thought raid0 "striped" the data across two or more devices to
increase total capacity, for example when adding a new device to an
existing filesystem. But that is not apparently what I ended up with.

Before:
btrfs device usage /mnt/backup/
/dev/sdc1, ID: 1
   Device size:           300.00GiB
   Device slack:              0.00B
   Data,single:           226.01GiB
   Metadata,DUP:            8.00GiB
   System,DUP:             64.00MiB
   Unallocated:            65.93GiB

/dev/sdc2, ID: 2
   Device size:           300.00GiB
   Device slack:              0.00B
   Data,single:             1.00GiB
   Unallocated:           299.00GiB

Then, I ran command:
btrfs balance start -dconvert=raid0 -mconvert=raid1 /mnt/backup

And what I ended up with seems to be double the amount of data used,
like what I think would happen with raid1, not raid0:

btrfs device usage /mnt/backup/
/dev/sdc1, ID: 1
   Device size:           300.00GiB
   Device slack:              0.00B
   Data,RAID0:            228.00GiB
   Metadata,RAID1:          5.00GiB
   System,RAID1:           64.00MiB
   Unallocated:            66.94GiB

/dev/sdc2, ID: 2
   Device size:           300.00GiB
   Device slack:              0.00B
   Data,RAID0:            228.00GiB
   Metadata,RAID1:          5.00GiB
   System,RAID1:           64.00MiB
   Unallocated:            66.94GiB

Or, am I misinterpreting what I am seeing? Thank you.

# uname -a
Linux tux 5.10.7-arch1-1 #1 SMP PREEMPT Wed, 13 Jan 2021 12:02:01
+0000 x86_64 GNU/Linux
# btrfs --version
btrfs-progs v5.9
# btrfs fi show
Label: none  uuid: c0f4c8e2-b580-4c0d-9562-abdb933b9625
        Total devices 1 FS bytes used 13.11GiB
        devid    1 size 449.51GiB used 14.01GiB path /dev/sda3

Label: none  uuid: 4fe39403-7ba1-4f22-972f-5041e3b6ff6f
        Total devices 1 FS bytes used 37.36GiB
        devid    1 size 600.00GiB used 40.02GiB path /dev/sdb1

Label: none  uuid: 1751eeca-c1a2-47bb-906b-c7199b09eb6d
        Total devices 2 FS bytes used 229.57GiB
        devid    1 size 300.00GiB used 233.06GiB path /dev/sdc1
        devid    2 size 300.00GiB used 233.06GiB path /dev/sdc2

# btrfs fi df /mnt/backup
Data, RAID0: total=456.00GiB, used=226.65GiB
System, RAID1: total=64.00MiB, used=64.00KiB
Metadata, RAID1: total=5.00GiB, used=2.92GiB
GlobalReserve, single: total=401.84MiB, used=0.00B

Re: raid0 confusion question

[Zygo Blaxell]

On Sat, Jan 16, 2021 at 03:08:00PM -0600, Tim Cuthbertson wrote:
> I thought raid0 "striped" the data across two or more devices to
> increase total capacity, for example when adding a new device to an
> existing filesystem. But that is not apparently what I ended up with.
> 
> Before:
> btrfs device usage /mnt/backup/
> /dev/sdc1, ID: 1
>    Device size:           300.00GiB
>    Device slack:              0.00B
>    Data,single:           226.01GiB
>    Metadata,DUP:            8.00GiB
>    System,DUP:             64.00MiB
>    Unallocated:            65.93GiB
> 
> /dev/sdc2, ID: 2
>    Device size:           300.00GiB
>    Device slack:              0.00B
>    Data,single:             1.00GiB
>    Unallocated:           299.00GiB
> 
> Then, I ran command:
> btrfs balance start -dconvert=raid0 -mconvert=raid1 /mnt/backup
> 
> And what I ended up with seems to be double the amount of data used,
> like what I think would happen with raid1, not raid0:
> 
> btrfs device usage /mnt/backup/
> /dev/sdc1, ID: 1
>    Device size:           300.00GiB
>    Device slack:              0.00B
>    Data,RAID0:            228.00GiB
>    Metadata,RAID1:          5.00GiB
>    System,RAID1:           64.00MiB
>    Unallocated:            66.94GiB
> 
> /dev/sdc2, ID: 2
>    Device size:           300.00GiB
>    Device slack:              0.00B
>    Data,RAID0:            228.00GiB
>    Metadata,RAID1:          5.00GiB
>    System,RAID1:           64.00MiB
>    Unallocated:            66.94GiB
> 
> Or, am I misinterpreting what I am seeing? Thank you.

btrfs divides disks into 1 GiB slices (on disks of this size), and the
joins the slices together to make chunks with a RAID profile.  Data
and metadata is then stored inside the chunks.

btrfs dev usage will show you the size of the chunks, not the amount of
(meta)data inside the chunks.

> # uname -a
> Linux tux 5.10.7-arch1-1 #1 SMP PREEMPT Wed, 13 Jan 2021 12:02:01
> +0000 x86_64 GNU/Linux
> # btrfs --version
> btrfs-progs v5.9
> # btrfs fi show
> Label: none  uuid: c0f4c8e2-b580-4c0d-9562-abdb933b9625
>         Total devices 1 FS bytes used 13.11GiB
>         devid    1 size 449.51GiB used 14.01GiB path /dev/sda3
> 
> Label: none  uuid: 4fe39403-7ba1-4f22-972f-5041e3b6ff6f
>         Total devices 1 FS bytes used 37.36GiB
>         devid    1 size 600.00GiB used 40.02GiB path /dev/sdb1
> 
> Label: none  uuid: 1751eeca-c1a2-47bb-906b-c7199b09eb6d
>         Total devices 2 FS bytes used 229.57GiB
>         devid    1 size 300.00GiB used 233.06GiB path /dev/sdc1
>         devid    2 size 300.00GiB used 233.06GiB path /dev/sdc2

btrfs fi show also reports chunk sizes (228 + 5 + 0.06 = 233.06).

The difference between the btrfs device size and the amount of data
allocated is called "unallocated" in 'btrfs dev usage' and 'btrfs
fi usage'.

The difference between btrfs device size and the physical device size
is called "slack" in 'btrfs dev usage' output (it does not appear in
'fi usage' output).

> # btrfs fi df /mnt/backup
> Data, RAID0: total=456.00GiB, used=226.65GiB
> System, RAID1: total=64.00MiB, used=64.00KiB
> Metadata, RAID1: total=5.00GiB, used=2.92GiB
> GlobalReserve, single: total=401.84MiB, used=0.00B

In 'btrfs fi df' output, 'total' is the size of chunks allocated, 'used'
is the amount of space used within the chunks (GlobalReserve is deducted
from metadata in RAM, it doesn't physically exist on any disk).

If you had shown 'btrfs fi usage' here, it might be clearer.
'fi usage' combines 'dev usage' with 'fi df', and it indicates
how much data is stored in each profile separately from how much
chunk space is allocated on each disk.

Plain 'df' should be showing the same amount of available space,
maybe a few GB different due to the metadata balance.

It is still storing 226 GiB of data, but it has allocated a larger number
of chunks (i.e. one chunk out for each chunk in, but the input chunks
are 1GB and the output chunks are 2x1GB, so each chunk is half full).
1GB of data at the beginning of the new chunk and then 1GB of empty space,
give or take a few blocks.

It's probably harmless, but if you want to waste a lot of iops for
nothing, you can balance the data again and it should pack the data
into chunks more tightly.  I wouldn't bother--the space is allocated
for data, so if you add more data to the filesystem it will just
fill in those chunks.  If you do a data balance, it will repack the
data into the data chunks so that allocated space is closer to used
space, but then later as you add more data to the filesystem new data
chunks will have to be created again.  There's no fragmentation concern
since the free space areas are likely all contiguous 1 GiB or larger.

Same thing has happened with metadata, 5GiB allocated for ~3GiB used.
Definitely do NOT balance the metadata (only balance metadata to change
RAID profiles) because you'll need that extra 2GiB to be preallocated
for metadata as the disk fills up.