* support request: btrfs df reports drive is out of space, cannot find what occupies it
@ 2024-04-19 8:40 Skirnir Torvaldsson
2024-04-19 11:19 ` Andrei Borzenkov
0 siblings, 1 reply; 5+ messages in thread
From: Skirnir Torvaldsson @ 2024-04-19 8:40 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1.1: Type: text/plain, Size: 1809 bytes --]
Dear btrfs experts,
Could you please help me sort out the following situation:
btrfs df reports my 100Gb device is almost out of space (which agrees
with the results produced by the standard "df"):
root@next:/home/support# btrfs fi df /
Data, single: total=82.00GiB, used=78.23GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.00GiB, used=153.70MiB
GlobalReserve, single: total=68.45MiB, used=0.00B
root@next:/home/support# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 96G 79G 16G 84% /
However when I try to locate files to delete with du that's what I get:
|root||@next||:/home/support# du -hd1 /|
|70M /boot|
|0| |/dev|
|2||.2G /.snapshots|
|14M /bin|
|4||.5M /etc|
|2||.5M /home|
|348M /lib|
|4||.0K /lib64|
|0| |/media|
|0| |/mnt|
|0| |/opt|
|0| |/proc|
|40K /root|
|2||.7M /run|
|12M /sbin|
|0| |/srv|
|0| |/sys|
|0| |/tmp|
|566M /usr|
|5||.0G /var|
|29G /next|
|38G /|
|
|
|I.e. almost 40Gb just gone somewhere. Am I doing something wrong? Is
there a problem or a piece of theory I'm missing? Kindly advice.|
|
|
|+++++++++++++++++++++++++++++++++++++|
|root@next:~||# uname -a|
|Linux next 5.10.0-28-amd64 ||#1 SMP Debian 5.10.209-2 (2024-01-31)
x86_64 GNU/Linux|
|root@next:~||# btrfs --version|
|btrfs-progs v5.10.1|
|root@next:~||# btrfs fi show|
|Label: ||'NEXT_ROOTFS'| |uuid: abc71bdb-c570-461d-a28a-54294a646089|
|||Total devices 1 FS bytes used 78.37GiB|
|||devid 1 size 95.46GiB used 84.06GiB path ||/dev/sda3|
|root@next:~||# btrfs fi df /|
|Data, single: total=82.00GiB, used=78.22GiB|
|System, DUP: total=32.00MiB, used=16.00KiB|
|Metadata, DUP: total=1.00GiB, used=153.64MiB|
|GlobalReserve, single: total=68.45MiB, used=0.00B|
|
|
[-- Attachment #1.2: Type: text/html, Size: 8568 bytes --]
[-- Attachment #2: dmesg.log.gz --]
[-- Type: application/x-gzip, Size: 20313 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: support request: btrfs df reports drive is out of space, cannot find what occupies it
2024-04-19 8:40 support request: btrfs df reports drive is out of space, cannot find what occupies it Skirnir Torvaldsson
@ 2024-04-19 11:19 ` Andrei Borzenkov
2024-04-19 13:18 ` Skirnir Torvaldsson
0 siblings, 1 reply; 5+ messages in thread
From: Andrei Borzenkov @ 2024-04-19 11:19 UTC (permalink / raw)
To: Skirnir Torvaldsson; +Cc: linux-btrfs
On Fri, Apr 19, 2024 at 11:40 AM Skirnir Torvaldsson
<skirnir.torvaldsson@gmail.com> wrote:
>
> Dear btrfs experts,
>
> Could you please help me sort out the following situation:
>
> btrfs df reports my 100Gb device is almost out of space (which agrees with the results produced by the standard "df"):
>
> root@next:/home/support# btrfs fi df /
> Data, single: total=82.00GiB, used=78.23GiB
> System, DUP: total=32.00MiB, used=16.00KiB
> Metadata, DUP: total=1.00GiB, used=153.70MiB
> GlobalReserve, single: total=68.45MiB, used=0.00B
> root@next:/home/support# df -h /
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda3 96G 79G 16G 84% /
>
> However when I try to locate files to delete with du that's what I get:
>
> root@next:/home/support# du -hd1 /
> 70M /boot
> 0 /dev
> 2.2G /.snapshots
> 14M /bin
> 4.5M /etc
> 2.5M /home
> 348M /lib
> 4.0K /lib64
> 0 /media
> 0 /mnt
> 0 /opt
> 0 /proc
> 40K /root
> 2.7M /run
> 12M /sbin
> 0 /srv
> 0 /sys
> 0 /tmp
> 566M /usr
> 5.0G /var
> 29G /next
> 38G /
>
> I.e. almost 40Gb just gone somewhere.
Huh?
2.2G + 5.0G + 29G + 38G == 75.2G out of 78G reported for DATA. What
40G are you talking about?
If you have some other mount points, you could start with explaining
your storage layout first.
> Am I doing something wrong? Is there a problem or a piece of theory I'm missing? Kindly advice.
>
> +++++++++++++++++++++++++++++++++++++
> root@next:~# uname -a
> Linux next 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux
> root@next:~# btrfs --version
> btrfs-progs v5.10.1
> root@next:~# btrfs fi show
> Label: 'NEXT_ROOTFS' uuid: abc71bdb-c570-461d-a28a-54294a646089
> Total devices 1 FS bytes used 78.37GiB
> devid 1 size 95.46GiB used 84.06GiB path /dev/sda3
>
> root@next:~# btrfs fi df /
> Data, single: total=82.00GiB, used=78.22GiB
> System, DUP: total=32.00MiB, used=16.00KiB
> Metadata, DUP: total=1.00GiB, used=153.64MiB
> GlobalReserve, single: total=68.45MiB, used=0.00B
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: support request: btrfs df reports drive is out of space, cannot find what occupies it
2024-04-19 11:19 ` Andrei Borzenkov
@ 2024-04-19 13:18 ` Skirnir Torvaldsson
2024-04-19 13:41 ` Andrei Borzenkov
0 siblings, 1 reply; 5+ messages in thread
From: Skirnir Torvaldsson @ 2024-04-19 13:18 UTC (permalink / raw)
To: Andrei Borzenkov; +Cc: linux-btrfs
With all due respect, 38G is a grand total, so it is 38G out of 78G
reported data.
There is one thing I've noticed that troubles me:
root@next:/home/support/btrfs-list-2.3# ./btrfs-list
NAME TYPE REFER EXCL MOUNTPOINT
NEXT_ROOTFS fs - 78.37G (single/dup,
15.03G/95.46G free, 15.74%)
[main] mainvol 16.00k 16.00k /
@System subvol 16.00k 16.00k /.snapshots
@Logs subvol 16.00k 16.00k /next/logs/.snapshots
@Logs/current subvol 4.70G 4.70G /next/logs
@AppData subvol 16.00k 16.00k /next/appdata/.snapshots
@AppData/current subvol 370.14M 370.14M /next/appdata
@AppData/var subvol 16.00k 16.00k
@Databases subvol 16.00k 16.00k /next/databases/.snapshots
@Databases/current subvol 754.95M 754.95M /next/databases
@MessageBus subvol 16.00k 16.00k /next/mbus/.snapshots
@MessageBus/current subvol 67.70G 67.70G /next/mbus
@Updates subvol 16.00k 16.00k /next/updates/.snapshots
@Updates/current subvol 1.81G 1.81G /next/updates
@SystemData subvol 16.00k 16.00k
/next/systemdata/.snapshots
@SystemData/current subvol 1.21G 1.21G /next/systemdata
@System/prev subvol 1.48G 1.48G
@System/current subvol 443.27M 443.27M /
root@next:/home/support/btrfs-list-2.3# du -hd1 /next/mbus
0 /next/mbus/.snapshots
1.4G /next/mbus/redpanda
1.4G /next/mbus
So, the MessageBus subvolume is occupying 67Gb (?), however I fail to
understand how come this space is not accounted for by du and how I can
clean it and limit it in future.
> On Fri, Apr 19, 2024 at 11:40 AM Skirnir Torvaldsson
> <skirnir.torvaldsson@gmail.com> wrote:
>> Dear btrfs experts,
>>
>> Could you please help me sort out the following situation:
>>
>> btrfs df reports my 100Gb device is almost out of space (which agrees with the results produced by the standard "df"):
>>
>> root@next:/home/support# btrfs fi df /
>> Data, single: total=82.00GiB, used=78.23GiB
>> System, DUP: total=32.00MiB, used=16.00KiB
>> Metadata, DUP: total=1.00GiB, used=153.70MiB
>> GlobalReserve, single: total=68.45MiB, used=0.00B
>> root@next:/home/support# df -h /
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/sda3 96G 79G 16G 84% /
>>
>> However when I try to locate files to delete with du that's what I get:
>>
>> root@next:/home/support# du -hd1 /
>> 70M /boot
>> 0 /dev
>> 2.2G /.snapshots
>> 14M /bin
>> 4.5M /etc
>> 2.5M /home
>> 348M /lib
>> 4.0K /lib64
>> 0 /media
>> 0 /mnt
>> 0 /opt
>> 0 /proc
>> 40K /root
>> 2.7M /run
>> 12M /sbin
>> 0 /srv
>> 0 /sys
>> 0 /tmp
>> 566M /usr
>> 5.0G /var
>> 29G /next
>> 38G /
>>
>> I.e. almost 40Gb just gone somewhere.
> Huh?
>
> 2.2G + 5.0G + 29G + 38G == 75.2G out of 78G reported for DATA. What
> 40G are you talking about?
>
> If you have some other mount points, you could start with explaining
> your storage layout first.
>
>> Am I doing something wrong? Is there a problem or a piece of theory I'm missing? Kindly advice.
>>
>> +++++++++++++++++++++++++++++++++++++
>> root@next:~# uname -a
>> Linux next 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux
>> root@next:~# btrfs --version
>> btrfs-progs v5.10.1
>> root@next:~# btrfs fi show
>> Label: 'NEXT_ROOTFS' uuid: abc71bdb-c570-461d-a28a-54294a646089
>> Total devices 1 FS bytes used 78.37GiB
>> devid 1 size 95.46GiB used 84.06GiB path /dev/sda3
>>
>> root@next:~# btrfs fi df /
>> Data, single: total=82.00GiB, used=78.22GiB
>> System, DUP: total=32.00MiB, used=16.00KiB
>> Metadata, DUP: total=1.00GiB, used=153.64MiB
>> GlobalReserve, single: total=68.45MiB, used=0.00B
>>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: support request: btrfs df reports drive is out of space, cannot find what occupies it
2024-04-19 13:18 ` Skirnir Torvaldsson
@ 2024-04-19 13:41 ` Andrei Borzenkov
2024-04-19 15:27 ` Skirnir Torvaldsson
0 siblings, 1 reply; 5+ messages in thread
From: Andrei Borzenkov @ 2024-04-19 13:41 UTC (permalink / raw)
To: Skirnir Torvaldsson; +Cc: linux-btrfs
On Fri, Apr 19, 2024 at 4:18 PM Skirnir Torvaldsson
<skirnir.torvaldsson@gmail.com> wrote:
>
> With all due respect, 38G is a grand total, so it is 38G out of 78G
> reported data.
> There is one thing I've noticed that troubles me:
>
> root@next:/home/support/btrfs-list-2.3# ./btrfs-list
> NAME TYPE REFER EXCL MOUNTPOINT
> NEXT_ROOTFS fs - 78.37G (single/dup,
> 15.03G/95.46G free, 15.74%)
> [main] mainvol 16.00k 16.00k /
> @System subvol 16.00k 16.00k /.snapshots
> @Logs subvol 16.00k 16.00k /next/logs/.snapshots
> @Logs/current subvol 4.70G 4.70G /next/logs
> @AppData subvol 16.00k 16.00k /next/appdata/.snapshots
> @AppData/current subvol 370.14M 370.14M /next/appdata
> @AppData/var subvol 16.00k 16.00k
> @Databases subvol 16.00k 16.00k /next/databases/.snapshots
> @Databases/current subvol 754.95M 754.95M /next/databases
> @MessageBus subvol 16.00k 16.00k /next/mbus/.snapshots
> @MessageBus/current subvol 67.70G 67.70G /next/mbus
> @Updates subvol 16.00k 16.00k /next/updates/.snapshots
> @Updates/current subvol 1.81G 1.81G /next/updates
> @SystemData subvol 16.00k 16.00k
> /next/systemdata/.snapshots
> @SystemData/current subvol 1.21G 1.21G /next/systemdata
> @System/prev subvol 1.48G 1.48G
> @System/current subvol 443.27M 443.27M /
> root@next:/home/support/btrfs-list-2.3# du -hd1 /next/mbus
> 0 /next/mbus/.snapshots
> 1.4G /next/mbus/redpanda
> 1.4G /next/mbus
>
> So, the MessageBus subvolume is occupying 67Gb (?), however I fail to
> understand how come this space is not accounted for by du and how I can
> clean it and limit it in future.
>
Could be variation of
https://lore.kernel.org/linux-btrfs/0f4a5a08fe9c4a6fe1bfcb0785691a7532abb958.camel@scientia.org/
> > On Fri, Apr 19, 2024 at 11:40 AM Skirnir Torvaldsson
> > <skirnir.torvaldsson@gmail.com> wrote:
> >> Dear btrfs experts,
> >>
> >> Could you please help me sort out the following situation:
> >>
> >> btrfs df reports my 100Gb device is almost out of space (which agrees with the results produced by the standard "df"):
> >>
> >> root@next:/home/support# btrfs fi df /
> >> Data, single: total=82.00GiB, used=78.23GiB
> >> System, DUP: total=32.00MiB, used=16.00KiB
> >> Metadata, DUP: total=1.00GiB, used=153.70MiB
> >> GlobalReserve, single: total=68.45MiB, used=0.00B
> >> root@next:/home/support# df -h /
> >> Filesystem Size Used Avail Use% Mounted on
> >> /dev/sda3 96G 79G 16G 84% /
> >>
> >> However when I try to locate files to delete with du that's what I get:
> >>
> >> root@next:/home/support# du -hd1 /
> >> 70M /boot
> >> 0 /dev
> >> 2.2G /.snapshots
> >> 14M /bin
> >> 4.5M /etc
> >> 2.5M /home
> >> 348M /lib
> >> 4.0K /lib64
> >> 0 /media
> >> 0 /mnt
> >> 0 /opt
> >> 0 /proc
> >> 40K /root
> >> 2.7M /run
> >> 12M /sbin
> >> 0 /srv
> >> 0 /sys
> >> 0 /tmp
> >> 566M /usr
> >> 5.0G /var
> >> 29G /next
> >> 38G /
> >>
> >> I.e. almost 40Gb just gone somewhere.
> > Huh?
> >
> > 2.2G + 5.0G + 29G + 38G == 75.2G out of 78G reported for DATA. What
> > 40G are you talking about?
> >
> > If you have some other mount points, you could start with explaining
> > your storage layout first.
> >
> >> Am I doing something wrong? Is there a problem or a piece of theory I'm missing? Kindly advice.
> >>
> >> +++++++++++++++++++++++++++++++++++++
> >> root@next:~# uname -a
> >> Linux next 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux
> >> root@next:~# btrfs --version
> >> btrfs-progs v5.10.1
> >> root@next:~# btrfs fi show
> >> Label: 'NEXT_ROOTFS' uuid: abc71bdb-c570-461d-a28a-54294a646089
> >> Total devices 1 FS bytes used 78.37GiB
> >> devid 1 size 95.46GiB used 84.06GiB path /dev/sda3
> >>
> >> root@next:~# btrfs fi df /
> >> Data, single: total=82.00GiB, used=78.22GiB
> >> System, DUP: total=32.00MiB, used=16.00KiB
> >> Metadata, DUP: total=1.00GiB, used=153.64MiB
> >> GlobalReserve, single: total=68.45MiB, used=0.00B
> >>
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: support request: btrfs df reports drive is out of space, cannot find what occupies it
2024-04-19 13:41 ` Andrei Borzenkov
@ 2024-04-19 15:27 ` Skirnir Torvaldsson
0 siblings, 0 replies; 5+ messages in thread
From: Skirnir Torvaldsson @ 2024-04-19 15:27 UTC (permalink / raw)
To: Andrei Borzenkov; +Cc: linux-btrfs
> On Fri, Apr 19, 2024 at 4:18 PM Skirnir Torvaldsson
> <skirnir.torvaldsson@gmail.com> wrote:
>> With all due respect, 38G is a grand total, so it is 38G out of 78G
>> reported data.
>> There is one thing I've noticed that troubles me:
>>
>> root@next:/home/support/btrfs-list-2.3# ./btrfs-list
>> NAME TYPE REFER EXCL MOUNTPOINT
>> NEXT_ROOTFS fs - 78.37G (single/dup,
>> 15.03G/95.46G free, 15.74%)
>> [main] mainvol 16.00k 16.00k /
>> @System subvol 16.00k 16.00k /.snapshots
>> @Logs subvol 16.00k 16.00k /next/logs/.snapshots
>> @Logs/current subvol 4.70G 4.70G /next/logs
>> @AppData subvol 16.00k 16.00k /next/appdata/.snapshots
>> @AppData/current subvol 370.14M 370.14M /next/appdata
>> @AppData/var subvol 16.00k 16.00k
>> @Databases subvol 16.00k 16.00k /next/databases/.snapshots
>> @Databases/current subvol 754.95M 754.95M /next/databases
>> @MessageBus subvol 16.00k 16.00k /next/mbus/.snapshots
>> @MessageBus/current subvol 67.70G 67.70G /next/mbus
>> @Updates subvol 16.00k 16.00k /next/updates/.snapshots
>> @Updates/current subvol 1.81G 1.81G /next/updates
>> @SystemData subvol 16.00k 16.00k
>> /next/systemdata/.snapshots
>> @SystemData/current subvol 1.21G 1.21G /next/systemdata
>> @System/prev subvol 1.48G 1.48G
>> @System/current subvol 443.27M 443.27M /
>> root@next:/home/support/btrfs-list-2.3# du -hd1 /next/mbus
>> 0 /next/mbus/.snapshots
>> 1.4G /next/mbus/redpanda
>> 1.4G /next/mbus
>>
>> So, the MessageBus subvolume is occupying 67Gb (?), however I fail to
>> understand how come this space is not accounted for by du and how I can
>> clean it and limit it in future.
>>
> Could be variation of
>
> https://lore.kernel.org/linux-btrfs/0f4a5a08fe9c4a6fe1bfcb0785691a7532abb958.camel@scientia.org/
Thank you so much, this seems to be my case:
root@next:/next/mbus/redpanda/data# compsize .
Processed 3490 files, 3489 regular extents (3489 refs), 1 inline.
Type Perc Disk Usage Uncompressed Referenced
TOTAL 99% 67G 67G 209M
none 100% 67G 67G 177M
zstd 8% 4.0K 48K 48K
prealloc 100% 32M 32M 31M
And these 67Gb of data are reported as "unreachable" by btdu. However,
manual defragmentation has no effect. Is there anything else I could
try short of deleting these files completely?
>
>
>>> On Fri, Apr 19, 2024 at 11:40 AM Skirnir Torvaldsson
>>> <skirnir.torvaldsson@gmail.com> wrote:
>>>> Dear btrfs experts,
>>>>
>>>> Could you please help me sort out the following situation:
>>>>
>>>> btrfs df reports my 100Gb device is almost out of space (which agrees with the results produced by the standard "df"):
>>>>
>>>> root@next:/home/support# btrfs fi df /
>>>> Data, single: total=82.00GiB, used=78.23GiB
>>>> System, DUP: total=32.00MiB, used=16.00KiB
>>>> Metadata, DUP: total=1.00GiB, used=153.70MiB
>>>> GlobalReserve, single: total=68.45MiB, used=0.00B
>>>> root@next:/home/support# df -h /
>>>> Filesystem Size Used Avail Use% Mounted on
>>>> /dev/sda3 96G 79G 16G 84% /
>>>>
>>>> However when I try to locate files to delete with du that's what I get:
>>>>
>>>> root@next:/home/support# du -hd1 /
>>>> 70M /boot
>>>> 0 /dev
>>>> 2.2G /.snapshots
>>>> 14M /bin
>>>> 4.5M /etc
>>>> 2.5M /home
>>>> 348M /lib
>>>> 4.0K /lib64
>>>> 0 /media
>>>> 0 /mnt
>>>> 0 /opt
>>>> 0 /proc
>>>> 40K /root
>>>> 2.7M /run
>>>> 12M /sbin
>>>> 0 /srv
>>>> 0 /sys
>>>> 0 /tmp
>>>> 566M /usr
>>>> 5.0G /var
>>>> 29G /next
>>>> 38G /
>>>>
>>>> I.e. almost 40Gb just gone somewhere.
>>> Huh?
>>>
>>> 2.2G + 5.0G + 29G + 38G == 75.2G out of 78G reported for DATA. What
>>> 40G are you talking about?
>>>
>>> If you have some other mount points, you could start with explaining
>>> your storage layout first.
>>>
>>>> Am I doing something wrong? Is there a problem or a piece of theory I'm missing? Kindly advice.
>>>>
>>>> +++++++++++++++++++++++++++++++++++++
>>>> root@next:~# uname -a
>>>> Linux next 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux
>>>> root@next:~# btrfs --version
>>>> btrfs-progs v5.10.1
>>>> root@next:~# btrfs fi show
>>>> Label: 'NEXT_ROOTFS' uuid: abc71bdb-c570-461d-a28a-54294a646089
>>>> Total devices 1 FS bytes used 78.37GiB
>>>> devid 1 size 95.46GiB used 84.06GiB path /dev/sda3
>>>>
>>>> root@next:~# btrfs fi df /
>>>> Data, single: total=82.00GiB, used=78.22GiB
>>>> System, DUP: total=32.00MiB, used=16.00KiB
>>>> Metadata, DUP: total=1.00GiB, used=153.64MiB
>>>> GlobalReserve, single: total=68.45MiB, used=0.00B
>>>>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2024-04-19 15:27 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-04-19 8:40 support request: btrfs df reports drive is out of space, cannot find what occupies it Skirnir Torvaldsson
2024-04-19 11:19 ` Andrei Borzenkov
2024-04-19 13:18 ` Skirnir Torvaldsson
2024-04-19 13:41 ` Andrei Borzenkov
2024-04-19 15:27 ` Skirnir Torvaldsson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox