* How to find/reclaim missing space in volume
@ 2023-06-05 16:26 Marc MERLIN
2023-06-05 16:47 ` Roman Mamedov
2023-06-05 17:00 ` Andrei Borzenkov
0 siblings, 2 replies; 13+ messages in thread
From: Marc MERLIN @ 2023-06-05 16:26 UTC (permalink / raw)
To: linux-btrfs
I have this:
sauron [mc]# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pool2 1.1T 853G 212G 81% /mnt/btrfs_pool2
sauron [mc]# btrfs fi show .
Label: 'btrfs_pool2' uuid: fde3da31-67e9-4f88-b90d-6c3f6becd56a
Total devices 1 FS bytes used 847.89GiB
devid 1 size 1.04TiB used 890.02GiB path /dev/mapper/pool2
sauron [mc]# btrfs fi df .
Data, single: total=878.00GiB, used=843.85GiB
System, DUP: total=8.00MiB, used=128.00KiB
Metadata, DUP: total=6.00GiB, used=4.04GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
sauron:/mnt/btrfs_pool2# du -sh *
599G varchange2
598G varchange2_ggm_daily_ro.20230605_07:57:43
4.0K varchange2_last
599G varchange2_ro.20230605_08:01:30
599G varchange2_ro.20230605_09:01:43
I'm confused, the volumes above are snapshots with mostly the same data
(made within the last 2 hours) and I didn't delete any data in the FS
(they are mostly identical and used for btfrs send/receive)
Why do they add up ot 600GB, but btrfs says 847FB is used?
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How to find/reclaim missing space in volume
2023-06-05 16:26 How to find/reclaim missing space in volume Marc MERLIN
@ 2023-06-05 16:47 ` Roman Mamedov
2023-06-05 17:00 ` Andrei Borzenkov
1 sibling, 0 replies; 13+ messages in thread
From: Roman Mamedov @ 2023-06-05 16:47 UTC (permalink / raw)
To: Marc MERLIN; +Cc: linux-btrfs
On Mon, 5 Jun 2023 09:26:36 -0700
Marc MERLIN <marc@merlins.org> wrote:
> I'm confused, the volumes above are snapshots with mostly the same data
> (made within the last 2 hours) and I didn't delete any data in the FS
> (they are mostly identical and used for btfrs send/receive)
>
> Why do they add up ot 600GB, but btrfs says 847FB is used?
- deleted snapshots that are still in the process of deletion
- deleted files that are still open by running apps
- files in subvolumes other than the currently mounted one (assuming mounting
a non-root subvolume)
- a huge amount of no longer used metadata chunks (does not apply here,
considering the fi df output)
- the infamous extent booking peculiarity of Btrfs, assuming you have large
files that are partially rewritten in-place (such as VM images or databases)
--
With respect,
Roman
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How to find/reclaim missing space in volume
2023-06-05 16:26 How to find/reclaim missing space in volume Marc MERLIN
2023-06-05 16:47 ` Roman Mamedov
@ 2023-06-05 17:00 ` Andrei Borzenkov
2023-06-06 1:46 ` Marc MERLIN
1 sibling, 1 reply; 13+ messages in thread
From: Andrei Borzenkov @ 2023-06-05 17:00 UTC (permalink / raw)
To: Marc MERLIN, linux-btrfs
On 05.06.2023 19:26, Marc MERLIN wrote:
> I have this:
> sauron [mc]# df -h .
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/pool2 1.1T 853G 212G 81% /mnt/btrfs_pool2
> sauron [mc]# btrfs fi show .
> Label: 'btrfs_pool2' uuid: fde3da31-67e9-4f88-b90d-6c3f6becd56a
> Total devices 1 FS bytes used 847.89GiB
> devid 1 size 1.04TiB used 890.02GiB path /dev/mapper/pool2
> sauron [mc]# btrfs fi df .
> Data, single: total=878.00GiB, used=843.85GiB
> System, DUP: total=8.00MiB, used=128.00KiB
> Metadata, DUP: total=6.00GiB, used=4.04GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
btrfs filesystem usage -T is usually more useful than both the above
commands.
>
> sauron:/mnt/btrfs_pool2# du -sh *
> 599G varchange2
> 598G varchange2_ggm_daily_ro.20230605_07:57:43
> 4.0K varchange2_last
> 599G varchange2_ro.20230605_08:01:30
> 599G varchange2_ro.20230605_09:01:43
>
> I'm confused, the volumes above are snapshots with mostly the same data
> (made within the last 2 hours) and I didn't delete any data in the FS
> (they are mostly identical and used for btfrs send/receive)
>
> Why do they add up ot 600GB, but btrfs says 847FB is used?
>
Each subvolume references 600G but it does not mean they are the same
600G. If quota is enabled, "btrfs quota show" may provide some more
information, otherwise "btrfs filesystem du" shows shared and exclusive
space (you need to pass all subvolumes in question to correctly compute
shared vs exclusive).
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How to find/reclaim missing space in volume
2023-06-05 17:00 ` Andrei Borzenkov
@ 2023-06-06 1:46 ` Marc MERLIN
2023-06-06 4:47 ` Andrei Borzenkov
0 siblings, 1 reply; 13+ messages in thread
From: Marc MERLIN @ 2023-06-06 1:46 UTC (permalink / raw)
To: Andrei Borzenkov; +Cc: linux-btrfs
On Mon, Jun 05, 2023 at 08:00:02PM +0300, Andrei Borzenkov wrote:
> On 05.06.2023 19:26, Marc MERLIN wrote:
> > I have this:
> > sauron [mc]# df -h .
> > Filesystem Size Used Avail Use% Mounted on
> > /dev/mapper/pool2 1.1T 853G 212G 81% /mnt/btrfs_pool2
> > sauron [mc]# btrfs fi show .
> > Label: 'btrfs_pool2' uuid: fde3da31-67e9-4f88-b90d-6c3f6becd56a
> > Total devices 1 FS bytes used 847.89GiB
> > devid 1 size 1.04TiB used 890.02GiB path /dev/mapper/pool2
> > sauron [mc]# btrfs fi df .
> > Data, single: total=878.00GiB, used=843.85GiB
> > System, DUP: total=8.00MiB, used=128.00KiB
> > Metadata, DUP: total=6.00GiB, used=4.04GiB
> > GlobalReserve, single: total=512.00MiB, used=0.00B
> >
>
> btrfs filesystem usage -T is usually more useful than both the above
> commands.
sauron:/mnt/btrfs_pool2# btrfs fi usage -T .
Overall:
Device size: 1.04TiB
Device allocated: 890.02GiB
Device unallocated: 177.73GiB
Device missing: 0.00B
Used: 851.85GiB
Free (estimated): 211.93GiB (min: 123.07GiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Data Metadata System
Id Path single DUP DUP Unallocated
-- ----------------- --------- -------- --------- -----------
1 /dev/mapper/pool2 878.00GiB 12.00GiB 16.00MiB 177.73GiB
-- ----------------- --------- -------- --------- -----------
Total 878.00GiB 6.00GiB 8.00MiB 177.73GiB
Used 843.79GiB 4.03GiB 128.00KiB
> > sauron:/mnt/btrfs_pool2# du -sh *
> > 599G varchange2
> > 598G varchange2_ggm_daily_ro.20230605_07:57:43
> > 4.0K varchange2_last
> > 599G varchange2_ro.20230605_08:01:30
> > 599G varchange2_ro.20230605_09:01:43
> >
> > I'm confused, the volumes above are snapshots with mostly the same data
> > (made within the last 2 hours) and I didn't delete any data in the FS
> > (they are mostly identical and used for btfrs send/receive)
> >
> > Why do they add up ot 600GB, but btrfs says 847FB is used?
> >
>
> Each subvolume references 600G but it does not mean they are the same 600G.
> If quota is enabled, "btrfs quota show" may provide some more information,
> otherwise "btrfs filesystem du" shows shared and exclusive space (you need
> to pass all subvolumes in question to correctly compute shared vs
> exclusive).
Right, I did check/know that the snapshots shared the same data, but it
doens't hurt to confirm:
sauron:/mnt/btrfs_pool2# btrfs filesystem du -s *
Total Exclusive Set shared Filename
597.57GiB 20.00KiB 588.75GiB varchange2
597.57GiB 4.00KiB 588.75GiB varchange2_ggm_daily_ro.20230605_09:59:26
597.57GiB 0.00B 588.75GiB varchange2_last
597.57GiB 4.00KiB 588.75GiB varchange2_minly.20230605_17:30:33
597.57GiB 0.00B 588.75GiB varchange2_minly.20230605_17:35:32
597.57GiB 0.00B 588.75GiB varchange2_minly.20230605_17:40:32
597.57GiB 4.00KiB 588.75GiB varchange2_minly.20230605_17:45:32
597.57GiB 4.00KiB 588.75GiB varchange2_minly.20230605_17:50:32
597.57GiB 4.00KiB 588.75GiB varchange2_minly.20230605_17:55:32
597.57GiB 0.00B 588.75GiB varchange2_minly.20230605_18:00:32
597.57GiB 0.00B 588.75GiB varchange2_minly.20230605_18:05:32
597.57GiB 8.00KiB 588.75GiB varchange2_minly.20230605_18:10:32
597.57GiB 16.00KiB 588.75GiB varchange2_ro.20230605_10:01:40
597.57GiB 12.00KiB 588.75GiB varchange2_ro.20230605_11:01:31
597.57GiB 4.00KiB 588.75GiB varchange2_ro.20230605_13:01:28
597.57GiB 4.00KiB 588.75GiB varchange2_ro.20230605_14:01:30
597.57GiB 4.00KiB 588.75GiB varchange2_ro.20230605_15:01:29
597.57GiB 4.00KiB 588.75GiB varchange2_ro.20230605_16:01:32
597.57GiB 4.00KiB 588.75GiB varchange2_ro.20230605_17:01:31
597.57GiB 0.00B 588.75GiB varchange2_ro.20230605_18:02:02
sauron:/mnt/btrfs_pool2# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pool2 1.1T 853G 212G 81% /mnt/btrfs_pool2
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How to find/reclaim missing space in volume
2023-06-06 1:46 ` Marc MERLIN
@ 2023-06-06 4:47 ` Andrei Borzenkov
2023-06-06 16:41 ` Marc MERLIN
2023-06-06 18:19 ` Graham Cobb
0 siblings, 2 replies; 13+ messages in thread
From: Andrei Borzenkov @ 2023-06-06 4:47 UTC (permalink / raw)
To: Marc MERLIN; +Cc: linux-btrfs
On 06.06.2023 04:46, Marc MERLIN wrote:
> On Mon, Jun 05, 2023 at 08:00:02PM +0300, Andrei Borzenkov wrote:
>> On 05.06.2023 19:26, Marc MERLIN wrote:
>>> I have this:
>>> sauron [mc]# df -h .
>>> Filesystem Size Used Avail Use% Mounted on
>>> /dev/mapper/pool2 1.1T 853G 212G 81% /mnt/btrfs_pool2
>>> sauron [mc]# btrfs fi show .
>>> Label: 'btrfs_pool2' uuid: fde3da31-67e9-4f88-b90d-6c3f6becd56a
>>> Total devices 1 FS bytes used 847.89GiB
>>> devid 1 size 1.04TiB used 890.02GiB path /dev/mapper/pool2
>>> sauron [mc]# btrfs fi df .
>>> Data, single: total=878.00GiB, used=843.85GiB
>>> System, DUP: total=8.00MiB, used=128.00KiB
>>> Metadata, DUP: total=6.00GiB, used=4.04GiB
>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>>
>>
>> btrfs filesystem usage -T is usually more useful than both the above
>> commands.
> sauron:/mnt/btrfs_pool2# btrfs fi usage -T .
> Overall:
> Device size: 1.04TiB
> Device allocated: 890.02GiB
> Device unallocated: 177.73GiB
> Device missing: 0.00B
> Used: 851.85GiB
> Free (estimated): 211.93GiB (min: 123.07GiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 512.00MiB (used: 0.00B)
>
> Data Metadata System
> Id Path single DUP DUP Unallocated
> -- ----------------- --------- -------- --------- -----------
> 1 /dev/mapper/pool2 878.00GiB 12.00GiB 16.00MiB 177.73GiB
> -- ----------------- --------- -------- --------- -----------
> Total 878.00GiB 6.00GiB 8.00MiB 177.73GiB
> Used 843.79GiB 4.03GiB 128.00KiB
>
>>> sauron:/mnt/btrfs_pool2# du -sh *
>>> 599G varchange2
>>> 598G varchange2_ggm_daily_ro.20230605_07:57:43
>>> 4.0K varchange2_last
>>> 599G varchange2_ro.20230605_08:01:30
>>> 599G varchange2_ro.20230605_09:01:43
>>>
>>> I'm confused, the volumes above are snapshots with mostly the same data
>>> (made within the last 2 hours) and I didn't delete any data in the FS
>>> (they are mostly identical and used for btfrs send/receive)
>>>
>>> Why do they add up ot 600GB, but btrfs says 847FB is used?
>>>
>>
>> Each subvolume references 600G but it does not mean they are the same 600G.
>> If quota is enabled, "btrfs quota show" may provide some more information,
>> otherwise "btrfs filesystem du" shows shared and exclusive space (you need
>> to pass all subvolumes in question to correctly compute shared vs
>> exclusive).
>
> Right, I did check/know that the snapshots shared the same data, but it
> doens't hurt to confirm:
>
> sauron:/mnt/btrfs_pool2# btrfs filesystem du -s *
> Total Exclusive Set shared Filename
> 597.57GiB 20.00KiB 588.75GiB varchange2
> 597.57GiB 4.00KiB 588.75GiB varchange2_ggm_daily_ro.20230605_09:59:26
> 597.57GiB 0.00B 588.75GiB varchange2_last
> 597.57GiB 4.00KiB 588.75GiB varchange2_minly.20230605_17:30:33
> 597.57GiB 0.00B 588.75GiB varchange2_minly.20230605_17:35:32
> 597.57GiB 0.00B 588.75GiB varchange2_minly.20230605_17:40:32
> 597.57GiB 4.00KiB 588.75GiB varchange2_minly.20230605_17:45:32
> 597.57GiB 4.00KiB 588.75GiB varchange2_minly.20230605_17:50:32
> 597.57GiB 4.00KiB 588.75GiB varchange2_minly.20230605_17:55:32
> 597.57GiB 0.00B 588.75GiB varchange2_minly.20230605_18:00:32
> 597.57GiB 0.00B 588.75GiB varchange2_minly.20230605_18:05:32
> 597.57GiB 8.00KiB 588.75GiB varchange2_minly.20230605_18:10:32
> 597.57GiB 16.00KiB 588.75GiB varchange2_ro.20230605_10:01:40
> 597.57GiB 12.00KiB 588.75GiB varchange2_ro.20230605_11:01:31
> 597.57GiB 4.00KiB 588.75GiB varchange2_ro.20230605_13:01:28
> 597.57GiB 4.00KiB 588.75GiB varchange2_ro.20230605_14:01:30
> 597.57GiB 4.00KiB 588.75GiB varchange2_ro.20230605_15:01:29
> 597.57GiB 4.00KiB 588.75GiB varchange2_ro.20230605_16:01:32
> 597.57GiB 4.00KiB 588.75GiB varchange2_ro.20230605_17:01:31
> 597.57GiB 0.00B 588.75GiB varchange2_ro.20230605_18:02:02
> sauron:/mnt/btrfs_pool2# df -h .
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/pool2 1.1T 853G 212G 81% /mnt/btrfs_pool2
>
>
Well, I have had it once, there were deleted but not freed subvolumes
https://lore.kernel.org/linux-btrfs/ecd46a18-1655-ec22-957b-de659af01bee@gmx.com/T/
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How to find/reclaim missing space in volume
2023-06-06 4:47 ` Andrei Borzenkov
@ 2023-06-06 16:41 ` Marc MERLIN
2023-06-06 18:25 ` Roman Mamedov
2023-06-06 18:19 ` Graham Cobb
1 sibling, 1 reply; 13+ messages in thread
From: Marc MERLIN @ 2023-06-06 16:41 UTC (permalink / raw)
To: Andrei Borzenkov; +Cc: linux-btrfs, Qu Wenruo, wqu
On Tue, Jun 06, 2023 at 07:47:03AM +0300, Andrei Borzenkov wrote:
> > sauron:/mnt/btrfs_pool2# df -h .
> > Filesystem Size Used Avail Use% Mounted on
> > /dev/mapper/pool2 1.1T 853G 212G 81% /mnt/btrfs_pool2
>
> Well, I have had it once, there were deleted but not freed subvolumes
>
> https://lore.kernel.org/linux-btrfs/ecd46a18-1655-ec22-957b-de659af01bee@gmx.com/T/
This sounds like it could be the same, thanks.
I started with
sauron:/mnt/btrfs_pool2# btrfs subvolume sync `pwd`
Unfortunately it's been stuck overnight.
Looks like I may have to reboot first (not ideal either)
Qu, any thing I should capture before reboot? Running 6.2.8 rigt now.
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How to find/reclaim missing space in volume
2023-06-06 4:47 ` Andrei Borzenkov
2023-06-06 16:41 ` Marc MERLIN
@ 2023-06-06 18:19 ` Graham Cobb
1 sibling, 0 replies; 13+ messages in thread
From: Graham Cobb @ 2023-06-06 18:19 UTC (permalink / raw)
To: Andrei Borzenkov, Marc MERLIN; +Cc: linux-btrfs
On 06/06/2023 05:47, Andrei Borzenkov wrote:
> On 06.06.2023 04:46, Marc MERLIN wrote:
>> On Mon, Jun 05, 2023 at 08:00:02PM +0300, Andrei Borzenkov wrote:
>>> On 05.06.2023 19:26, Marc MERLIN wrote:
>>>> I have this:
>>>> sauron [mc]# df -h .
>>>> Filesystem Size Used Avail Use% Mounted on
>>>> /dev/mapper/pool2 1.1T 853G 212G 81% /mnt/btrfs_pool2
>>>> sauron [mc]# btrfs fi show .
>>>> Label: 'btrfs_pool2' uuid: fde3da31-67e9-4f88-b90d-6c3f6becd56a
>>>> Total devices 1 FS bytes used 847.89GiB
>>>> devid 1 size 1.04TiB used 890.02GiB path /dev/mapper/pool2
>>>> sauron [mc]# btrfs fi df .
>>>> Data, single: total=878.00GiB, used=843.85GiB
>>>> System, DUP: total=8.00MiB, used=128.00KiB
>>>> Metadata, DUP: total=6.00GiB, used=4.04GiB
>>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>>>
>>>
>>> btrfs filesystem usage -T is usually more useful than both the above
>>> commands.
>> sauron:/mnt/btrfs_pool2# btrfs fi usage -T .
>> Overall:
>> Device size: 1.04TiB
>> Device allocated: 890.02GiB
>> Device unallocated: 177.73GiB
>> Device missing: 0.00B
>> Used: 851.85GiB
>> Free (estimated): 211.93GiB (min: 123.07GiB)
>> Data ratio: 1.00
>> Metadata ratio: 2.00
>> Global reserve: 512.00MiB (used: 0.00B)
>>
>> Data Metadata System
>> Id Path single DUP DUP Unallocated
>> -- ----------------- --------- -------- --------- -----------
>> 1 /dev/mapper/pool2 878.00GiB 12.00GiB 16.00MiB 177.73GiB
>> -- ----------------- --------- -------- --------- -----------
>> Total 878.00GiB 6.00GiB 8.00MiB 177.73GiB
>> Used 843.79GiB 4.03GiB 128.00KiB
>>
>>>> sauron:/mnt/btrfs_pool2# du -sh *
>>>> 599G varchange2
>>>> 598G varchange2_ggm_daily_ro.20230605_07:57:43
>>>> 4.0K varchange2_last
>>>> 599G varchange2_ro.20230605_08:01:30
>>>> 599G varchange2_ro.20230605_09:01:43
>>>>
>>>> I'm confused, the volumes above are snapshots with mostly the same data
>>>> (made within the last 2 hours) and I didn't delete any data in the FS
>>>> (they are mostly identical and used for btfrs send/receive)
>>>>
>>>> Why do they add up ot 600GB, but btrfs says 847FB is used?
>>>>
>>>
>>> Each subvolume references 600G but it does not mean they are the same
>>> 600G.
>>> If quota is enabled, "btrfs quota show" may provide some more
>>> information,
>>> otherwise "btrfs filesystem du" shows shared and exclusive space (you
>>> need
>>> to pass all subvolumes in question to correctly compute shared vs
>>> exclusive).
>>
>> Right, I did check/know that the snapshots shared the same data, but it
>> doens't hurt to confirm:
But your output doesn't confirm that. The problem is that the concept of
"shared" is not useful as it tells you nothing about how many times each
file is shared. The files in the last snapshot could all be completely
different from the first and you would still see that output.
You need to do pairwise comparisons. A check between the first and the
last would tell you if there was much change.
My "extents-lists" scripts (https://github.com/GrahamCobb/extents-lists)
were my hack to do such comparisons - there may be better ways to do it
nowadays. If you use those tools you could try:
extents-expr -s /mnt/btrfs_pool2/varchange2_ro.20230605_10:01:40 ^
/mnt/btrfs_pool2/varchange2_ro.20230605_18:02:02
That would show you the amount of data shared between the two specified
directories.
Graham
>>
>> sauron:/mnt/btrfs_pool2# btrfs filesystem du -s *
>> Total Exclusive Set shared Filename
>> 597.57GiB 20.00KiB 588.75GiB varchange2
>> 597.57GiB 4.00KiB 588.75GiB
>> varchange2_ggm_daily_ro.20230605_09:59:26
>> 597.57GiB 0.00B 588.75GiB varchange2_last
>> 597.57GiB 4.00KiB 588.75GiB varchange2_minly.20230605_17:30:33
>> 597.57GiB 0.00B 588.75GiB varchange2_minly.20230605_17:35:32
>> 597.57GiB 0.00B 588.75GiB varchange2_minly.20230605_17:40:32
>> 597.57GiB 4.00KiB 588.75GiB varchange2_minly.20230605_17:45:32
>> 597.57GiB 4.00KiB 588.75GiB varchange2_minly.20230605_17:50:32
>> 597.57GiB 4.00KiB 588.75GiB varchange2_minly.20230605_17:55:32
>> 597.57GiB 0.00B 588.75GiB varchange2_minly.20230605_18:00:32
>> 597.57GiB 0.00B 588.75GiB varchange2_minly.20230605_18:05:32
>> 597.57GiB 8.00KiB 588.75GiB varchange2_minly.20230605_18:10:32
>> 597.57GiB 16.00KiB 588.75GiB varchange2_ro.20230605_10:01:40
>> 597.57GiB 12.00KiB 588.75GiB varchange2_ro.20230605_11:01:31
>> 597.57GiB 4.00KiB 588.75GiB varchange2_ro.20230605_13:01:28
>> 597.57GiB 4.00KiB 588.75GiB varchange2_ro.20230605_14:01:30
>> 597.57GiB 4.00KiB 588.75GiB varchange2_ro.20230605_15:01:29
>> 597.57GiB 4.00KiB 588.75GiB varchange2_ro.20230605_16:01:32
>> 597.57GiB 4.00KiB 588.75GiB varchange2_ro.20230605_17:01:31
>> 597.57GiB 0.00B 588.75GiB varchange2_ro.20230605_18:02:02
>> sauron:/mnt/btrfs_pool2# df -h .
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/mapper/pool2 1.1T 853G 212G 81% /mnt/btrfs_pool2
>>
>>
>
> Well, I have had it once, there were deleted but not freed subvolumes
>
> https://lore.kernel.org/linux-btrfs/ecd46a18-1655-ec22-957b-de659af01bee@gmx.com/T/
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How to find/reclaim missing space in volume
2023-06-06 16:41 ` Marc MERLIN
@ 2023-06-06 18:25 ` Roman Mamedov
2023-06-07 2:12 ` Qu Wenruo
0 siblings, 1 reply; 13+ messages in thread
From: Roman Mamedov @ 2023-06-06 18:25 UTC (permalink / raw)
To: Marc MERLIN; +Cc: Andrei Borzenkov, linux-btrfs, Qu Wenruo, wqu
On Tue, 6 Jun 2023 09:41:39 -0700
Marc MERLIN <marc@merlins.org> wrote:
> This sounds like it could be the same, thanks.
>
> I started with
> sauron:/mnt/btrfs_pool2# btrfs subvolume sync `pwd`
>
> Unfortunately it's been stuck overnight.
So it does not look the same after all.
Do you see a lot of IO to the device (iostat, iotop)? Is it a fast/slow one?
Anything in dmesg?
--
With respect,
Roman
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How to find/reclaim missing space in volume
2023-06-06 18:25 ` Roman Mamedov
@ 2023-06-07 2:12 ` Qu Wenruo
2023-06-07 19:17 ` Marc MERLIN
0 siblings, 1 reply; 13+ messages in thread
From: Qu Wenruo @ 2023-06-07 2:12 UTC (permalink / raw)
To: Roman Mamedov, Marc MERLIN; +Cc: Andrei Borzenkov, linux-btrfs, Qu Wenruo
On 2023/6/7 02:25, Roman Mamedov wrote:
> On Tue, 6 Jun 2023 09:41:39 -0700
> Marc MERLIN <marc@merlins.org> wrote:
>
>> This sounds like it could be the same, thanks.
>>
>> I started with
>> sauron:/mnt/btrfs_pool2# btrfs subvolume sync `pwd`
>>
>> Unfortunately it's been stuck overnight.
>
> So it does not look the same after all.
>
> Do you see a lot of IO to the device (iostat, iotop)? Is it a fast/slow one?
>
> Anything in dmesg?
>
For the ghost subvolumes, it can be confirmed with "btrfs ins dump-tree
-t root", then looking for ROOT_ITEMs with "refs 0".
I'm not sure if it is the case.
Another possible cause is extent bookends, this needs something like
btrfs quota to confirm, and not much we can do if there are snapshots.
(If no snapshots, it's possible to defrag and free up such bookend extents).
Thanks,
Qu
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How to find/reclaim missing space in volume
2023-06-07 2:12 ` Qu Wenruo
@ 2023-06-07 19:17 ` Marc MERLIN
2023-06-07 19:32 ` Hans van Kranenburg
0 siblings, 1 reply; 13+ messages in thread
From: Marc MERLIN @ 2023-06-07 19:17 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Roman Mamedov, Andrei Borzenkov, linux-btrfs
On Wed, Jun 07, 2023 at 10:12:30AM +0800, Qu Wenruo wrote:
> For the ghost subvolumes, it can be confirmed with "btrfs ins dump-tree
> -t root", then looking for ROOT_ITEMs with "refs 0".
>
> I'm not sure if it is the case.
>
> Another possible cause is extent bookends, this needs something like
> btrfs quota to confirm, and not much we can do if there are snapshots.
> (If no snapshots, it's possible to defrag and free up such bookend extents).
unfortunately the system rebooted overnight, so I lost the state where
sync was hung.
On the plus side, this seems to have fixed the issue:
sauron:/mnt/btrfs_pool2# btrfs fi usage -T .
Overall:
Device size: 1.04TiB
Device allocated: 857.02GiB
Device unallocated: 210.73GiB
Device missing: 0.00B
Used: 589.30GiB
Free (estimated): 470.78GiB (min: 365.42GiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Data Metadata System
Id Path single DUP DUP Unallocated
-- ----------------- --------- -------- --------- -----------
1 /dev/mapper/pool2 845.00GiB 12.00GiB 16.00MiB 210.73GiB
-- ----------------- --------- -------- --------- -----------
Total 845.00GiB 6.00GiB 8.00MiB 210.73GiB
Used 584.95GiB 2.18GiB 112.00KiB
sauron:/mnt/btrfs_pool2# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pool2 1.1T 590G 471G 56% /mnt/btrfs_pool2
I know it's hard to say after the fact, but have you seen issues where snapshot delete
space cleanup would get stuck and
btrfs subvolume sync `pwd`
would hang forever (until reboot) ?
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How to find/reclaim missing space in volume
2023-06-07 19:17 ` Marc MERLIN
@ 2023-06-07 19:32 ` Hans van Kranenburg
2023-06-07 20:02 ` Marc MERLIN
0 siblings, 1 reply; 13+ messages in thread
From: Hans van Kranenburg @ 2023-06-07 19:32 UTC (permalink / raw)
To: Marc MERLIN, Qu Wenruo; +Cc: Roman Mamedov, Andrei Borzenkov, linux-btrfs
Hi,
On 6/7/23 21:17, Marc MERLIN wrote:
> On Wed, Jun 07, 2023 at 10:12:30AM +0800, Qu Wenruo wrote:
>> For the ghost subvolumes, it can be confirmed with "btrfs ins dump-tree
>> -t root", then looking for ROOT_ITEMs with "refs 0".
>>
>> I'm not sure if it is the case.
>>
>> Another possible cause is extent bookends, this needs something like
>> btrfs quota to confirm, and not much we can do if there are snapshots.
>> (If no snapshots, it's possible to defrag and free up such bookend extents).
>
> unfortunately the system rebooted overnight, so I lost the state where
> sync was hung.
> On the plus side, this seems to have fixed the issue:
Just a random hint... One possible situation in which a deleted
subvolume can't be freed up for real yet, is when there is a process
that still has an open file in it.
> sauron:/mnt/btrfs_pool2# btrfs fi usage -T .
> Overall:
> Device size: 1.04TiB
> Device allocated: 857.02GiB
> Device unallocated: 210.73GiB
> Device missing: 0.00B
> Used: 589.30GiB
> Free (estimated): 470.78GiB (min: 365.42GiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 512.00MiB (used: 0.00B)
>
> Data Metadata System
> Id Path single DUP DUP Unallocated
> -- ----------------- --------- -------- --------- -----------
> 1 /dev/mapper/pool2 845.00GiB 12.00GiB 16.00MiB 210.73GiB
> -- ----------------- --------- -------- --------- -----------
> Total 845.00GiB 6.00GiB 8.00MiB 210.73GiB
> Used 584.95GiB 2.18GiB 112.00KiB
> sauron:/mnt/btrfs_pool2# df -h .
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/pool2 1.1T 590G 471G 56% /mnt/btrfs_pool2
>
> I know it's hard to say after the fact, but have you seen issues where snapshot delete
> space cleanup would get stuck and
> btrfs subvolume sync `pwd`
> would hang forever (until reboot) ?
Hans
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How to find/reclaim missing space in volume
2023-06-07 19:32 ` Hans van Kranenburg
@ 2023-06-07 20:02 ` Marc MERLIN
2023-06-07 20:31 ` Hans van Kranenburg
0 siblings, 1 reply; 13+ messages in thread
From: Marc MERLIN @ 2023-06-07 20:02 UTC (permalink / raw)
To: Hans van Kranenburg
Cc: Qu Wenruo, Roman Mamedov, Andrei Borzenkov, linux-btrfs
On Wed, Jun 07, 2023 at 09:32:13PM +0200, Hans van Kranenburg wrote:
> > On the plus side, this seems to have fixed the issue:
>
> Just a random hint... One possible situation in which a deleted
> subvolume can't be freed up for real yet, is when there is a process
> that still has an open file in it.
this is a fair guess.
Too late now, but is this something that would show up in
lsof -n | grep volume ?
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How to find/reclaim missing space in volume
2023-06-07 20:02 ` Marc MERLIN
@ 2023-06-07 20:31 ` Hans van Kranenburg
0 siblings, 0 replies; 13+ messages in thread
From: Hans van Kranenburg @ 2023-06-07 20:31 UTC (permalink / raw)
To: Marc MERLIN; +Cc: Qu Wenruo, Roman Mamedov, Andrei Borzenkov, linux-btrfs
On 6/7/23 22:02, Marc MERLIN wrote:
> On Wed, Jun 07, 2023 at 09:32:13PM +0200, Hans van Kranenburg wrote:
>>> On the plus side, this seems to have fixed the issue:
>>
>> Just a random hint... One possible situation in which a deleted
>> subvolume can't be freed up for real yet, is when there is a process
>> that still has an open file in it.
>
> this is a fair guess.
> Too late now, but is this something that would show up in
> lsof -n | grep volume ?
Yes, it should.
I just quickly tested this, by doing something like...
btrfs sub create foo
cat >> foo/bar
.. and then while the file is still open ..
btrfs sub del foo/
-$ lsof -n |grep foo
cat 577355 knorrie 1w REG 0,0 32 257 /home/knorrie/p/foo/bar
So it doesn't show as DEL, but still as REG, even.
Hans
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2023-06-07 20:31 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-06-05 16:26 How to find/reclaim missing space in volume Marc MERLIN
2023-06-05 16:47 ` Roman Mamedov
2023-06-05 17:00 ` Andrei Borzenkov
2023-06-06 1:46 ` Marc MERLIN
2023-06-06 4:47 ` Andrei Borzenkov
2023-06-06 16:41 ` Marc MERLIN
2023-06-06 18:25 ` Roman Mamedov
2023-06-07 2:12 ` Qu Wenruo
2023-06-07 19:17 ` Marc MERLIN
2023-06-07 19:32 ` Hans van Kranenburg
2023-06-07 20:02 ` Marc MERLIN
2023-06-07 20:31 ` Hans van Kranenburg
2023-06-06 18:19 ` Graham Cobb
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox