understanding btrfs fi df

understanding btrfs fi df

[Hendrik Friedel]

Hello,

I am struggling to understand the output of btrfs fi df:

btrfs fi df /mnt/__Complete_Disk/
Data, RAID5: total=3.85TiB, used=3.85TiB
System, RAID5: total=32.00MiB, used=576.00KiB
Metadata, RAID5: total=6.46GiB, used=5.14GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

I have three disks:

btrfs fi show
Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
         Total devices 3 FS bytes used 3.85TiB
         devid    1 size 2.73TiB used 2.73TiB path /dev/sdf
         devid    2 size 2.73TiB used 2.24TiB path /dev/sdd
         devid    3 size 2.73TiB used 2.73TiB path /dev/sde


So I would expect around 5.4TiB to be available. In fact I am currently 
using 4.4TiB and the drive seems to be full (I cannot store more data).

I went through Marcs Blog post on this, but the hints there did not 
help. In particular:
  btrfs balance start -dusage=95 /mnt/__Complete_Disk/ (started with 55)

What's wrong here?

Regards,
Hendrik

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus

Re: understanding btrfs fi df

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2734 bytes --]

On Sun, Aug 16, 2015 at 04:33:52PM +0200, Hendrik Friedel wrote:
> Hello,
> 
> I am struggling to understand the output of btrfs fi df:
> 
> btrfs fi df /mnt/__Complete_Disk/
> Data, RAID5: total=3.85TiB, used=3.85TiB
> System, RAID5: total=32.00MiB, used=576.00KiB
> Metadata, RAID5: total=6.46GiB, used=5.14GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> I have three disks:
> 
> btrfs fi show
> Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
>         Total devices 3 FS bytes used 3.85TiB
>         devid    1 size 2.73TiB used 2.73TiB path /dev/sdf
>         devid    2 size 2.73TiB used 2.24TiB path /dev/sdd
>         devid    3 size 2.73TiB used 2.73TiB path /dev/sde

   Here's your problem -- you've got a RAID 5 filesystem, which has a
minimum allocation of 2 devices, but only one device has free space on
it for allocation, so no more chunks can be allocated. I'm not sure
how it ended up in this situation, but that's what's happened.

   From btrfs fi df, you've filled up all the existing data
allocation, so the usual filtered balance isn't going to help, because
there's no space within the existing allocation to compact the storage
into.

   You will need to free up some space on devices 1 and 3, and then
run a full balance. There are two ways I can see of doing this:

1) add a small (4-8 GiB, say) block device, do a full balance, and
then delete the device again:

# btrfs dev add /dev/sdx /mnt/__Complete_Disk
# btrfs balance start /mnt/__Complete_Disk
# btrfs dev del /dev/sdx /mnt/__Complete_Disk

2) convert a few chunks of the existing FS, on devices 1 and 3, to
single, which should free up some space across all the devices, and
then run a full balance, and finally convert the single back to RAID
5:

# btrfs balance start -dconvert=single,devid=1,limit=4 /mnt/__Complete_Disk
# btrfs balance start -dconvert=single,devid=3,limit=4 /mnt/__Complete_Disk
# btrfs balance start -dprofiles=raid5 /mnt/__Complete_Disk
# btrfs balance start -dconvert=raid5,soft /mnt/__Complete_Disk

   Hugo.

> So I would expect around 5.4TiB to be available. In fact I am
> currently using 4.4TiB and the drive seems to be full (I cannot
> store more data).
> 
> I went through Marcs Blog post on this, but the hints there did not
> help. In particular:
>  btrfs balance start -dusage=95 /mnt/__Complete_Disk/ (started with 55)
> 
> What's wrong here?
> 
> Regards,
> Hendrik
> 
> ---
> Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
> https://www.avast.com/antivirus
> 

-- 
Hugo Mills             | ©1973 Unclear Research Ltd
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: understanding btrfs fi df

[Hendrik Friedel]

Hello Hugo,

thanks for your hint.

On 16.08.2015 16:57, Hugo Mills wrote:
>     Here's your problem -- you've got a RAID 5 filesystem, which has a
> minimum allocation of 2 devices, but only one device has free space on
> it for allocation, so no more chunks can be allocated. I'm not sure
> how it ended up in this situation, but that's what's happened.
>
> 2) convert a few chunks of the existing FS, on devices 1 and 3, to
> single, which should free up some space across all the devices, and
> then run a full balance, and finally convert the single back to RAID
> 5:
>
> # btrfs balance start -dconvert=single,devid=1,limit=4/mnt/__Complete_Disk
> # btrfs balance start -dconvert=single,devid=3,limit=4/mnt/__Complete_Disk
> # btrfs balance start -dprofiles=raid5/mnt/__Complete_Disk
> # btrfs balance start -dconvert=raid5,soft/mnt/__Complete_Disk


I did that. In order to run the first two commands I had to free some 
space though, which was no problem.

Now the output is:
root@homeserver:/media# btrfs fi df /mnt/__Complete_Disk 

Data, RAID5: total=3.79TiB, used=3.78TiB
System, RAID5: total=32.00MiB, used=416.00KiB
Metadata, RAID5: total=6.46GiB, used=4.85GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

root@homeserver:/media# btrfs fi show
Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
         Total devices 3 FS bytes used 3.79TiB
         devid    1 size 2.73TiB used 2.43TiB path /dev/sdf
         devid    2 size 2.73TiB used 2.43TiB path /dev/sdd
         devid    3 size 2.73TiB used 1.38TiB path /dev/sde

How can only 1.38TiB be used on devid 3?


Greetings,
Hendrik

-- 
Hendrik Friedel
Auf dem Brink 12
28844 Weyhe
Tel. 04203 8394854
Mobil 0178 1874363

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus

Re: understanding btrfs fi df

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2196 bytes --]

On Wed, Aug 19, 2015 at 06:10:06PM +0200, Hendrik Friedel wrote:
> Hello Hugo,
> 
> thanks for your hint.
> 
> On 16.08.2015 16:57, Hugo Mills wrote:
> >    Here's your problem -- you've got a RAID 5 filesystem, which has a
> >minimum allocation of 2 devices, but only one device has free space on
> >it for allocation, so no more chunks can be allocated. I'm not sure
> >how it ended up in this situation, but that's what's happened.
> >
> >2) convert a few chunks of the existing FS, on devices 1 and 3, to
> >single, which should free up some space across all the devices, and
> >then run a full balance, and finally convert the single back to RAID
> >5:
> >
> ># btrfs balance start -dconvert=single,devid=1,limit=4/mnt/__Complete_Disk
> ># btrfs balance start -dconvert=single,devid=3,limit=4/mnt/__Complete_Disk
> ># btrfs balance start -dprofiles=raid5/mnt/__Complete_Disk
> ># btrfs balance start -dconvert=raid5,soft/mnt/__Complete_Disk
> 
> 
> I did that. In order to run the first two commands I had to free
> some space though, which was no problem.
> 
> Now the output is:
> root@homeserver:/media# btrfs fi df /mnt/__Complete_Disk
> 
> Data, RAID5: total=3.79TiB, used=3.78TiB
> System, RAID5: total=32.00MiB, used=416.00KiB
> Metadata, RAID5: total=6.46GiB, used=4.85GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> root@homeserver:/media# btrfs fi show
> Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
>         Total devices 3 FS bytes used 3.79TiB
>         devid    1 size 2.73TiB used 2.43TiB path /dev/sdf
>         devid    2 size 2.73TiB used 2.43TiB path /dev/sdd
>         devid    3 size 2.73TiB used 1.38TiB path /dev/sde
> 
> How can only 1.38TiB be used on devid 3?

   It shouldn't happen, as I understand how the process works. Can you
show the output of "btrfs fi df /mnt/__Complete_Disk"? Let's just
check that everything is indeed RAID-5 still.

   After this, we're probably going to have to look at the device and
chunk trees to work out what's going on.

   Hugo.

-- 
Hugo Mills             | "I lost my leg in 1942. Some bastard stole it in a
hugo@... carfax.org.uk | pub in Pimlico."
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: understanding btrfs fi df

[Hendrik Friedel]

Hi Hugo,

thanks for your help.

 >> Now the output is:
>> root@homeserver:/media# btrfs fi df /mnt/__Complete_Disk
>>
>> Data, RAID5: total=3.79TiB, used=3.78TiB
>> System, RAID5: total=32.00MiB, used=416.00KiB
>> Metadata, RAID5: total=6.46GiB, used=4.85GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>> root@homeserver:/media# btrfs fi show
>> Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
>>          Total devices 3 FS bytes used 3.79TiB
>>          devid    1 size 2.73TiB used 2.43TiB path /dev/sdf
>>          devid    2 size 2.73TiB used 2.43TiB path /dev/sdd
>>          devid    3 size 2.73TiB used 1.38TiB path /dev/sde
>>
>> How can only 1.38TiB be used on devid 3?
>
>     It shouldn't happen, as I understand how the process works. Can you
> show the output of "btrfs fi df /mnt/__Complete_Disk"? Let's just
> check that everything is indeed RAID-5 still.

Here we go:

btrfs fi df /mnt/__Complete_Disk

Data, RAID5: total=3.79TiB, used=3.78TiB
System, RAID5: total=32.00MiB, used=416.00KiB
Metadata, RAID5: total=6.46GiB, used=4.85GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


Greetings,
Hendrik

Greetings,
Hendrik


-- 
Hendrik Friedel
Auf dem Brink 12
28844 Weyhe
Tel. 04203 8394854
Mobil 0178 1874363

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus

Re: understanding btrfs fi df

[Hendrik Friedel]

Hello Hugo,

>>     It shouldn't happen, as I understand how the process works. Can you
>> show the output of "btrfs fi df /mnt/__Complete_Disk"? Let's just
>> check that everything is indeed RAID-5 still.
>
> Here we go:
>
> btrfs fi df /mnt/__Complete_Disk
>
> Data, RAID5: total=3.79TiB, used=3.78TiB
> System, RAID5: total=32.00MiB, used=416.00KiB
> Metadata, RAID5: total=6.46GiB, used=4.85GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B

can you help me on this? As far as I see, it's all Raid5.
So,
 >>   After this, we're probably going to have to look at the device and
 >> chunk trees to work out what's going on.

Can you help me on this?

Regards,
Hendrik






---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus