From: Christoph Anton Mitterer <calestyo@scientia.org>
To: kreijack@inwind.it, Qu Wenruo <quwenruo.btrfs@gmx.com>,
linux-btrfs@vger.kernel.org
Subject: Re: btrfs thinks fs is full, though 11GB should be still free
Date: Mon, 18 Dec 2023 23:38:38 +0100 [thread overview]
Message-ID: <fecad7ce2cea1ff125a842d8c53f1fbfe4f1d231.camel@scientia.org> (raw)
In-Reply-To: <7acc8ea1-079d-42bb-8880-dbd9bbfa100b@libero.it>
[-- Attachment #1: Type: text/plain, Size: 11465 bytes --]
Hey.
On Mon, 2023-12-18 at 20:18 +0100, Goffredo Baroncelli wrote:
> Being only 309 files, I suggest to find one file as test case and
> start to inspect what is happening.
I made a small wrapper around compsize:
#!/bin/sh
while IFS='' read -r file; do
tmp="$(compsize -b "$file" | grep '^none' | sed -E 's/ +/ /g')"
du="$(printf '%s\n' "$tmp" | cut -d ' ' -f 3)"
ref="$(printf '%s\n' "$tmp" | cut -d ' ' -f 5)"
delta="$(( $du - $ref ))"
if [ "$delta" -ge 524288 ]; then
printf '%s\t%s\n' "$delta" "$file"
fi
done
called like:
# find /data/main -type f | ~/compsize-helper
252653568 /data/main/prometheus/metrics2/01HHFEZPJ8TPFVYTXV11R7ZH4X/chunks/000001
217198592 /data/main/prometheus/metrics2/01HHFFHGP4SDPEGGY4ME3GT0Q7/chunks/000001
107094016 /data/main/prometheus/metrics2/01HHFPD50EKCPVC2WZCP006WDV/chunks/000001
121311232 /data/main/prometheus/metrics2/01HHFX8YPK0D55GRM4V5TN46J3/chunks/000001
106512384 /data/main/prometheus/metrics2/01HHG44J1F5R8K3527V2VVT5VA/chunks/000001
102907904 /data/main/prometheus/metrics2/01HHGB0A32RWYKWZBDVMH7NG70/chunks/000001
105345024 /data/main/prometheus/metrics2/01HHGHW20CFYANMMNF1DGSMQE2/chunks/000001
105590784 /data/main/prometheus/metrics2/01HHGRQSR0M0F53JJBKZH52S7H/chunks/000001
106688512 /data/main/prometheus/metrics2/01HHGZKJEXEKW3594B74DR9NTH/chunks/000001
106213376 /data/main/prometheus/metrics2/01HHH6FA6YRWC6HAJGQJX9QATF/chunks/000001
106409984 /data/main/prometheus/metrics2/01HHHDB1EHS6V9NYKYAMJTSXBB/chunks/000001
225648640 /data/main/prometheus/metrics2/01HHHM6S6JYGAXCQV9XVMGMW83/chunks/000001
107253760 /data/main/prometheus/metrics2/01HHHV2DZYXNFZ5PRHB2M12E7Z/chunks/000001
106340352 /data/main/prometheus/metrics2/01HHJ1Y5QS4DS408MMCH97WC02/chunks/000001
105254912 /data/main/prometheus/metrics2/01HHJ8SXF3W7DKW5N0YACG3ZWT/chunks/000001
104161280 /data/main/prometheus/metrics2/01HHJFNKQZ3AEZYFXWEQ5RQ3HB/chunks/000001
104771584 /data/main/prometheus/metrics2/01HHJPHBFVMBRB27ECMNB0J3GG/chunks/000001
101986304 /data/main/prometheus/metrics2/01HHJXD37EPQXB5599MHWEESMZ/chunks/000001
106614784 /data/main/prometheus/metrics2/01HHK48TZ2KNFNWT1VTH5HE0HF/chunks/000001
231501824 /data/main/prometheus/metrics2/01HHKB4K6M286JNX0QAX6M5P7C/chunks/000001
107102208 /data/main/prometheus/metrics2/01HHKJ07GSVRGR3B3RW7EY2M03/chunks/000001
106823680 /data/main/prometheus/metrics2/01HHKRVZ85RXB1TZ0M1Q3453D2/chunks/000001
105611264 /data/main/prometheus/metrics2/01HHKZQPZTQV3DCVQCCVH7DXGE/chunks/000001
104497152 /data/main/prometheus/metrics2/01HHM6KB9WNF18ZNC7JE81Z3QW/chunks/000001
107216896 /data/main/prometheus/metrics2/01HHMDF5FRRAX8DCNGCGTNDR98/chunks/000001
105299968 /data/main/prometheus/metrics2/01HHMMAVRD2BTWKCZPYZ1WD6DT/chunks/000001
105328640 /data/main/prometheus/metrics2/01HHMV6KZS6F476HG6DT4ZM180/chunks/000001
223375360 /data/main/prometheus/metrics2/01HHN22CPP9FVDZG1MGET13J4X/chunks/000001
224215040 /data/main/prometheus/metrics2/01HHN8Y10XMFF9M2CRPQJWG94Y/chunks/000001
99090432 /data/main/prometheus/metrics2/01HHNFSS84Y6C6BBVY7E3QMAZS/chunks/000001
104562688 /data/main/prometheus/metrics2/01HHNPNH00AZHAV1RPB6SW8443/chunks/000001
107819008 /data/main/prometheus/metrics2/01HHNXH8QE32FQEMA7DDVZK8M8/chunks/000001
103890944 /data/main/prometheus/metrics2/01HHP4D1EBSX48Y3CY66VV703Y/chunks/000001
105041920 /data/main/prometheus/metrics2/01HHPB8P8AN1W5SH6ZTC90R48Q/chunks/000001
107278336 /data/main/prometheus/metrics2/01HHPJ4EZG9TB32P8W4MDMQSZG/chunks/000001
106553344 /data/main/prometheus/metrics2/01HHPS02SFZEC187ZJJDVSD1Z0/chunks/000001
156815360 /data/main/prometheus/metrics2/01HHPZVV0P5QCGG8QA89C36RVW/chunks/000001
153927680 /data/main/prometheus/metrics2/01HHQ6QJRFY0WF1K4QHM74A12S/chunks/000001
125075456 /data/main/prometheus/metrics2/01HHQDKBFRFFQ7J2XJ836CRHW3/chunks/000001
209260544 /data/main/prometheus/metrics2/01HHQMF375KTC4BWMF9RBCM6RS/chunks/000001
106807296 /data/main/prometheus/metrics2/01HHQVAVESRRXJ5KQ9FN8AQ0XA/chunks/000001
105955328 /data/main/prometheus/metrics2/01HHR26E9C5B2RYFCQPG7G7M4H/chunks/000001
105402368 /data/main/prometheus/metrics2/01HHR92618K1C6QCR1JP3NKRR5/chunks/000001
107376640 /data/main/prometheus/metrics2/01HHRFXY8DJPBQ924RJJWZB3BY/chunks/000001
107413504 /data/main/prometheus/metrics2/01HHRPSP069YJXRDHF3P8CPM5S/chunks/000001
105881600 /data/main/prometheus/metrics2/01HHRXNE7JFSYFQ3QF63WDZ0AB/chunks/000001
217497600 /data/main/prometheus/metrics2/01HHS4H21ZSZVA10JVDNHNQ7Q8/chunks/000001
108908544 /data/main/prometheus/metrics2/01HHSBCT94CC9AW1N9JQA6VF0K/chunks/000001
109170688 /data/main/prometheus/metrics2/01HHSJ8J18HQTQ4V45AZTSN336/chunks/000001
109060096 /data/main/prometheus/metrics2/01HHSS4AQMF186EXPQKT77KG8F/chunks/000001
108732416 /data/main/prometheus/metrics2/01HHSZZWKYJG4AQ8S6X3A0V1RF/chunks/000001
159817728 /data/main/prometheus/metrics2/01HHT6VMTVPDT2RMR76P642HZ9/chunks/000001
109006848 /data/main/prometheus/metrics2/01HHTDQD2H7VJRXBC5BA7K4018/chunks/000001
108662784 /data/main/prometheus/metrics2/01HHTMK68T7QSGE4MHFWBJHM1W/chunks/000001
201793536 /data/main/prometheus/metrics2/01HHTVEYZYPDXDZSGDM91S9QFW/chunks/000001
109015040 /data/main/prometheus/metrics2/01HHV2AQ7DEHX5BNSXX2VQC2TW/chunks/000001
109756416 /data/main/prometheus/metrics2/01HHV96BHBAP1TZY54AQ8BA0YB/chunks/000001
103170048 /data/main/prometheus/metrics2/01HHVG238JVD74GD07F2W87Y9Y/chunks/000001
210407424 /data/main/prometheus/metrics2/01HHVPXVG00GAK4E3YY6ADGDG9/chunks/000001
108261376 /data/main/prometheus/metrics2/01HHVXSMQ5XWWKXAK7SRR61AK8/chunks/000001
108457984 /data/main/prometheus/metrics2/01HHW4N72CMZ90VTVACG7QD87M/chunks/000001
109596672 /data/main/prometheus/metrics2/01HHWBH08ZYZRNEC38VXK9Y946/chunks/000001
194060288 /data/main/prometheus/metrics2/01HHWJCRG7WC5NHB684RXCNKMV/chunks/000001
110596096 /data/main/prometheus/metrics2/01HHWS8GQGHNWH4Z4WETJN454Q/chunks/000001
110592000 /data/main/prometheus/metrics2/01HHX048F9110B1CMK300XCE99/chunks/000001
107954176 /data/main/prometheus/metrics2/01HHX6ZTAZEQ69MGKQDSZ7TG96/chunks/000001
157130752 /data/main/prometheus/metrics2/01HHXDVJJG0YMR3T2ZSS7TMGPA/chunks/000001
122175488 /data/main/prometheus/metrics2/01HHXMQE7152WG1HARM51JPKFT/chunks/000001
208908288 /data/main/prometheus/metrics2/01HHXVK5YTHRVQXEHXWR1K270F/chunks/000001
140976128 /data/main/prometheus/metrics2/01HHY2EXPC19MBPRGM05X3ZB75/chunks/000001
140902400 /data/main/prometheus/metrics2/01HHY9AKFSADNWJVW4XN06QTZY/chunks/000001
250085376 /data/main/prometheus/metrics2/01HHYG6AQEAYV7WBSWJSSMHR02/chunks/000001
114298880 /data/main/prometheus/metrics2/01HHYQ22YNG446ARA9KDTJH9RP/chunks/000001
108421120 /data/main/prometheus/metrics2/01HHYXXT096QKW02KAAXH791SG/chunks/000001
109977600 /data/main/prometheus/metrics2/01HHZ4SEH5S13B9EYV7S4NV4EZ/chunks/000001
99586048 /data/main/prometheus/metrics2/01HHZBN68RG5YTA2EP50TRC6T4/chunks/000001
To answer Qu's question from his reply:
No snapshots are involved, and unless Prometheus itself makes recopies,
no such should be involved either.
I do run Thanos sidecar on top of Prometheus, which indeed makes copies
of the files before uploading them to some remote storage, but it also
deletes them afterwards. And looking at the /proc/<thanos>/fd, it
doesn't keep them open as deleted.
No compression should be used (again, unless Prometheus would manually
set chattr +c or so), btrfs RAID. Nothing except a plain btrfs.
I used to have quota's enabled when Qu asked me to last week, but
disabled them afterwards.
I've also attached the output of:
# find /data/main -type f -exec sh -c 'echo "$1"; compsize "$1"' '' {} \; > compsize.log
in case it helps anyone.
The above shows IMO that most data is lost in the chunk files (these
000001 files are the beef of the data).
Looking at the reverse (files where the delta is less than 0.5 MiB):
233472 /data/main/prometheus/metrics2/wal/00000522
155648 /data/main/prometheus/metrics2/wal/00000523
184320 /data/main/prometheus/metrics2/wal/00000524
200704 /data/main/prometheus/metrics2/wal/00000525
126976 /data/main/prometheus/metrics2/wal/00000526
192512 /data/main/prometheus/metrics2/wal/00000527
200704 /data/main/prometheus/metrics2/wal/00000528
139264 /data/main/prometheus/metrics2/wal/00000529
200704 /data/main/prometheus/metrics2/wal/00000530
172032 /data/main/prometheus/metrics2/wal/00000531
57344 /data/main/prometheus/metrics2/wal/00000532
20480 /data/main/prometheus/metrics2/chunks_head/000151
12288 /data/main/prometheus/metrics2/chunks_head/000152
28672 /data/main/prometheus/metrics2/chunks_head/000153
45056 /data/main/prometheus/metrics2/01HHFFHGP4SDPEGGY4ME3GT0Q7/inde
x
229376 /data/main/prometheus/metrics2/01HHFPD50EKCPVC2WZCP006WDV/inde
x
40960 /data/main/prometheus/metrics2/01HHHM6S6JYGAXCQV9XVMGMW83/inde
x
40960 /data/main/prometheus/metrics2/01HHHV2DZYXNFZ5PRHB2M12E7Z/inde
x
40960 /data/main/prometheus/metrics2/01HHJ1Y5QS4DS408MMCH97WC02/inde
x
40960 /data/main/prometheus/metrics2/01HHJ8SXF3W7DKW5N0YACG3ZWT/inde
x
40960 /data/main/prometheus/metrics2/01HHJPHBFVMBRB27ECMNB0J3GG/inde
x
40960 /data/main/prometheus/metrics2/01HHJXD37EPQXB5599MHWEESMZ/inde
x
40960 /data/main/prometheus/metrics2/01HHKB4K6M286JNX0QAX6M5P7C/inde
x
40960 /data/main/prometheus/metrics2/01HHMV6KZS6F476HG6DT4ZM180/inde
x
40960 /data/main/prometheus/metrics2/01HHN22CPP9FVDZG1MGET13J4X/inde
x
40960 /data/main/prometheus/metrics2/01HHN8Y10XMFF9M2CRPQJWG94Y/inde
x
40960 /data/main/prometheus/metrics2/01HHNPNH00AZHAV1RPB6SW8443/inde
x
40960 /data/main/prometheus/metrics2/01HHPZVV0P5QCGG8QA89C36RVW/inde
x
40960 /data/main/prometheus/metrics2/01HHQ6QJRFY0WF1K4QHM74A12S/inde
x
40960 /data/main/prometheus/metrics2/01HHQDKBFRFFQ7J2XJ836CRHW3/inde
x
40960 /data/main/prometheus/metrics2/01HHQMF375KTC4BWMF9RBCM6RS/inde
x
40960 /data/main/prometheus/metrics2/01HHR92618K1C6QCR1JP3NKRR5/inde
x
40960 /data/main/prometheus/metrics2/01HHRFXY8DJPBQ924RJJWZB3BY/inde
x
40960 /data/main/prometheus/metrics2/01HHS4H21ZSZVA10JVDNHNQ7Q8/inde
x
40960 /data/main/prometheus/metrics2/01HHSBCT94CC9AW1N9JQA6VF0K/inde
x
40960 /data/main/prometheus/metrics2/01HHT6VMTVPDT2RMR76P642HZ9/inde
x
40960 /data/main/prometheus/metrics2/01HHTVEYZYPDXDZSGDM91S9QFW/inde
x
40960 /data/main/prometheus/metrics2/01HHVPXVG00GAK4E3YY6ADGDG9/inde
x
40960 /data/main/prometheus/metrics2/01HHWJCRG7WC5NHB684RXCNKMV/inde
x
40960 /data/main/prometheus/metrics2/01HHX048F9110B1CMK300XCE99/inde
x
40960 /data/main/prometheus/metrics2/01HHXDVJJG0YMR3T2ZSS7TMGPA/inde
x
86016 /data/main/prometheus/metrics2/01HHXVK5YTHRVQXEHXWR1K270F/inde
x
40960 /data/main/prometheus/metrics2/01HHYG6AQEAYV7WBSWJSSMHR02/inde
x
4096 /data/main/prometheus/metrics2/01HHZBN68RG5YTA2EP50TRC6T4/inde
x
So the WAL is not that much of a problem.
As for Goffredo's idea about syncing, that doesn't seem to change
things either:
# btrfs filesystem sync /data/main
# compsize btrfs filesystem sync /data/main
Processed 321 files, 793 regular extents (794 refs), 152 inline.
Type Perc Disk Usage Uncompressed Referenced
TOTAL 100% 23G 23G 14G
none 100% 23G 23G 14G
And this is anyway running long-term... so I'd assume that sooner or
later btrfs syncs it's stuff?
Cheers,
Chris.
[-- Attachment #2: compsize.log.xz --]
[-- Type: application/x-xz, Size: 2984 bytes --]
next prev parent reply other threads:[~2023-12-19 0:59 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-12-11 20:26 btrfs thinks fs is full, though 11GB should be still free Christoph Anton Mitterer
2023-12-11 20:57 ` Qu Wenruo
2023-12-11 22:23 ` Christoph Anton Mitterer
2023-12-11 22:26 ` Christoph Anton Mitterer
2023-12-11 23:20 ` Qu Wenruo
2023-12-11 23:38 ` Christoph Anton Mitterer
2023-12-11 23:54 ` Qu Wenruo
2023-12-12 0:12 ` Christoph Anton Mitterer
2023-12-12 0:58 ` Qu Wenruo
2023-12-12 2:30 ` Qu Wenruo
2023-12-12 3:27 ` Christoph Anton Mitterer
2023-12-12 3:40 ` Christoph Anton Mitterer
2023-12-12 4:13 ` Qu Wenruo
2023-12-15 2:33 ` Chris Murphy
2023-12-15 3:12 ` Qu Wenruo
2023-12-18 16:24 ` Christoph Anton Mitterer
2023-12-18 19:18 ` Goffredo Baroncelli
2023-12-18 20:04 ` Goffredo Baroncelli
2023-12-18 22:38 ` Christoph Anton Mitterer [this message]
2023-12-19 8:22 ` Andrei Borzenkov
2023-12-19 19:09 ` Goffredo Baroncelli
2023-12-21 13:53 ` Christoph Anton Mitterer
2023-12-21 18:03 ` Goffredo Baroncelli
2023-12-21 22:06 ` Christoph Anton Mitterer
2023-12-21 13:46 ` Christoph Anton Mitterer
2023-12-21 20:41 ` Qu Wenruo
2023-12-21 22:15 ` Christoph Anton Mitterer
2023-12-21 22:41 ` Qu Wenruo
2023-12-21 22:54 ` Christoph Anton Mitterer
2023-12-22 0:53 ` Qu Wenruo
2023-12-22 0:56 ` Christoph Anton Mitterer
2023-12-22 1:13 ` Qu Wenruo
2023-12-22 1:23 ` Christoph Anton Mitterer
2024-01-05 3:30 ` Christoph Anton Mitterer
2024-01-05 7:07 ` Qu Wenruo
2024-01-06 0:42 ` Christoph Anton Mitterer
2024-01-06 5:40 ` Qu Wenruo
2024-01-06 8:12 ` Andrei Borzenkov
2024-12-14 19:09 ` Christoph Anton Mitterer
2023-12-18 19:54 ` Qu Wenruo
2023-12-18 22:30 ` Christoph Anton Mitterer
2023-12-13 1:49 ` Remi Gauvin
2023-12-13 8:29 ` Andrea Gelmini
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=fecad7ce2cea1ff125a842d8c53f1fbfe4f1d231.camel@scientia.org \
--to=calestyo@scientia.org \
--cc=kreijack@inwind.it \
--cc=linux-btrfs@vger.kernel.org \
--cc=quwenruo.btrfs@gmx.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox