* Filesystem Read Only due to errno=-28 during metadata allocation
@ 2021-10-13 12:35 mailing
2021-10-18 11:13 ` mailing
2021-10-18 14:09 ` Zygo Blaxell
0 siblings, 2 replies; 7+ messages in thread
From: mailing @ 2021-10-13 12:35 UTC (permalink / raw)
To: linux-btrfs
Hello,
I faced issue with btrfs FS /var forced to RO due to errno=-28 (no space
left).
The server was restarted to bring back FS in RW.
Before reboot:
$ btrfs fi usage /var -m
Overall:
Device size: 2560.00MiB
Device allocated: 2559.00MiB
Device unallocated: 1.00MiB
Device missing: 0.00MiB
Used: 1116.00MiB
Free (estimated): 451.25MiB (min: 451.25MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)
Data,single: Size:1559.25MiB, Used:1108.00MiB
/dev/mapper/rootvg-varvol 1559.25MiB
Metadata,DUP: Size:467.88MiB, Used:3.94MiB
/dev/mapper/rootvg-varvol 935.75MiB
System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB
Unallocated:
/dev/mapper/rootvg-varvol 1.00MiB
The FS went RO on Sunday, with this trace:
2021-10-10T00:13:12.790042+02:00 SERVERNAME kernel: BTRFS: Transaction
aborted (error -28)
2021-10-10T00:13:12.790053+02:00 SERVERNAME kernel: ------------[ cut
here ]------------
2021-10-10T00:13:12.790055+02:00 SERVERNAME kernel: WARNING: CPU: 8 PID:
8532 at ../fs/btrfs/extent-tree.c:2353
btrfs_run_delayed_refs+0x2b4/0x2c0 [btrfs]
2021-10-10T00:13:12.790057+02:00 SERVERNAME kernel: Modules linked in:
lin_tape(OEX) pfo(OEX) nfsv3 nfs_acl nfs lockd grace sunrpc fscache
rpadlpar_io(X) rpaphp(X) tcp_diag udp_diag raw_diag inet_diag unix_diag
af_packet_diag netlink_diag binfmt_misc af_packet xfs libcrc32c st ch
ibmveth(X) vmx_crypto gf128mul crct10dif_vpmsum uio_pdrv_genirq uio
rtc_generic btrfs xor zstd_decompress zstd_compress xxhash raid6_pq
dm_service_time sd_mod ibmvfc(X) scsi_transport_fc crc32c_vpmsum
dm_mirror dm_region_hash dm_log sg dm_multipath dm_mod scsi_dh_rdac
scsi_dh_emc scsi_dh_alua scsi_mod autofs4
2021-10-10T00:13:12.790059+02:00 SERVERNAME kernel: Supported: Yes,
External
2021-10-10T00:13:12.790053+02:00 SERVERNAME kernel: ------------[ cut
here ]------------
2021-10-10T00:13:12.790055+02:00 SERVERNAME kernel: WARNING: CPU: 8 PID:
8532 at ../fs/btrfs/extent-tree.c:2353
btrfs_run_delayed_refs+0x2b4/0x2c0 [btrfs]
2021-10-10T00:13:12.790057+02:00 SERVERNAME kernel: Modules linked in:
lin_tape(OEX) pfo(OEX) nfsv3 nfs_acl nfs lockd grace sunrpc fscache
rpadlpar_io(X) rpaphp(X) tcp_diag udp_diag raw_diag inet_diag unix_diag
af_packet_diag netlink_diag binfmt_misc af_packet xfs libcrc32c st ch
ibmveth(X) vmx_crypto gf128mul crct10dif_vpmsum uio_pdrv_genirq uio
rtc_generic btrfs xor zstd_decompress zstd_compress xxhash raid6_pq
dm_service_time sd_mod ibmvfc(X) scsi_transport_fc crc32c_vpmsum
dm_mirror dm_region_hash dm_log sg dm_multipath dm_mod scsi_dh_rdac
scsi_dh_emc scsi_dh_alua scsi_mod autofs4
2021-10-10T00:13:12.790059+02:00 SERVERNAME kernel: Supported: Yes,
External
2021-10-10T00:13:12.790060+02:00 SERVERNAME kernel: CPU: 8 PID: 8532
Comm: vpdupdate Tainted: G W OE 4.12.14-122.83-default #1 SLE12-SP5
2021-10-10T00:13:12.790076+02:00 SERVERNAME kernel: task:
c0000068d70c1600 task.stack: c000002aba1a0000
2021-10-10T00:13:12.790078+02:00 SERVERNAME kernel: NIP:
d000000035876474 LR: d000000035876470 CTR: 0000000000000000
2021-10-10T00:13:12.790080+02:00 SERVERNAME kernel: REGS:
c000002aba1a3870 TRAP: 0700 Tainted: G W OE (4.12.14-122.83-default)
2021-10-10T00:13:12.790081+02:00 SERVERNAME kernel: MSR:
800000010282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE,TM[E]>
2021-10-10T00:13:12.790083+02:00 SERVERNAME kernel: CR: 24444422 XER:
20040000
2021-10-10T00:13:12.790085+02:00 SERVERNAME kernel: CFAR:
c0000000009c817c SOFTE: 1
2021-10-10T00:13:12.790086+02:00 SERVERNAME kernel: GPR00:
d000000035876470 c000002aba1a3af0 d000000035993288 0000000000000026
2021-10-10T00:13:12.790088+02:00 SERVERNAME kernel: GPR04:
c00000e6bf30ade8 c00000e6bf322a00 0000000000000007 c0000000013ed474
2021-10-10T00:13:12.790090+02:00 SERVERNAME kernel: GPR08:
0000000000000000 c000000000dc16fc 000000e6be550000 000000000000134f
2021-10-10T00:13:12.790091+02:00 SERVERNAME kernel: GPR12:
0000000000004000 c00000000f6c9400 00000000007fffff 0000000000000014
2021-10-10T00:13:12.790093+02:00 SERVERNAME kernel: GPR16:
00000100231aa080 0000010022868988 aaaaaaaaaaaaaaab c000000024ebf778
2021-10-10T00:13:12.790095+02:00 SERVERNAME kernel: GPR20:
0000000000000000 0000000000000000 0000000000000000 c00000e5a1f6eb30
2021-10-10T00:13:12.790097+02:00 SERVERNAME kernel: GPR24:
c00000e5a1f6eb20 00000000000016ba c00000e5a1f6e9c0 c00000e6828cc000
2021-10-10T00:13:12.790098+02:00 SERVERNAME kernel: GPR28:
c0000014b1bc02d0 0000000000000000 c00000e6828cc000 ffffffffffffffe4
2021-10-10T00:13:12.790100+02:00 SERVERNAME kernel: NIP
[d000000035876474] btrfs_run_delayed_refs+0x2b4/0x2c0 [btrfs]
2021-10-10T00:13:12.790101+02:00 SERVERNAME kernel: LR
[d000000035876470] btrfs_run_delayed_refs+0x2b0/0x2c0 [btrfs]
2021-10-10T00:13:12.790103+02:00 SERVERNAME kernel: Call Trace:
2021-10-10T00:13:12.790104+02:00 SERVERNAME kernel: [c000002aba1a3af0]
[d000000035876470] btrfs_run_delayed_refs+0x2b0/0x2c0 [btrfs]
(unreliable)
2021-10-10T00:13:12.790106+02:00 SERVERNAME kernel: [c000002aba1a3bb0]
[d000000035891554] btrfs_commit_transaction+0x74/0xc10 [btrfs]
2021-10-10T00:13:12.790108+02:00 SERVERNAME kernel: [c000002aba1a3c80]
[d0000000358b3328] btrfs_sync_file+0x3a8/0x510 [btrfs]
2021-10-10T00:13:12.790110+02:00 SERVERNAME kernel: [c000002aba1a3d80]
[c000000000408720] vfs_fsync_range+0x70/0x120
2021-10-10T00:13:12.790111+02:00 SERVERNAME kernel: [c000002aba1a3dd0]
[c00000000040886c] do_fsync+0x5c/0xb0
2021-10-10T00:13:12.790113+02:00 SERVERNAME kernel: [c000002aba1a3e10]
[c000000000408cfc] SyS_fdatasync+0x2c/0x40
2021-10-10T00:13:12.790115+02:00 SERVERNAME kernel: [c000002aba1a3e30]
[c00000000000b308] system_call+0x3c/0x130
2021-10-10T00:13:12.790116+02:00 SERVERNAME kernel: Instruction dump:
2021-10-10T00:13:12.790118+02:00 SERVERNAME kernel: e87c0060 3c820000
e8848b48 38a0fffb 4bfe6905 60000000 4bffffb4 3c620000
2021-10-10T00:13:12.790120+02:00 SERVERNAME kernel: e8638b38 7fe4fb78
480e3f55 e8410018 <0fe00000> 4bffff98 60000000 3c4c0012
2021-10-10T00:13:12.790122+02:00 SERVERNAME kernel: ---[ end trace
c9aa23777165dfdc ]---
2021-10-10T00:13:12.790124+02:00 SERVERNAME kernel: BTRFS: error (device
dm-22) in btrfs_run_delayed_refs:2353: errno=-28 No space left
2021-10-10T00:13:12.790125+02:00 SERVERNAME kernel: btrfs_printk: 12
callbacks suppressed
2021-10-10T00:13:12.790127+02:00 SERVERNAME kernel: BTRFS info (device
dm-22): forced readonly
2021-10-10T00:13:12.790060+02:00 SERVERNAME kernel: CPU: 8 PID: 8532
Comm: vpdupdate Tainted: G W OE 4.12.14-122.83-default #1 SLE12-SP5
2021-10-10T00:13:12.790076+02:00 SERVERNAME kernel: task:
c0000068d70c1600 task.stack: c000002aba1a0000
2021-10-10T00:13:12.790078+02:00 SERVERNAME kernel: NIP:
d000000035876474 LR: d000000035876470 CTR: 0000000000000000
2021-10-10T00:13:12.790080+02:00 SERVERNAME kernel: REGS:
c000002aba1a3870 TRAP: 0700 Tainted: G W OE (4.12.14-122.83-default)
2021-10-10T00:13:12.790081+02:00 SERVERNAME kernel: MSR:
800000010282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE,TM[E]>
2021-10-10T00:13:12.790083+02:00 SERVERNAME kernel: CR: 24444422 XER:
20040000
2021-10-10T00:13:12.790085+02:00 SERVERNAME kernel: CFAR:
c0000000009c817c SOFTE: 1
2021-10-10T00:13:12.790086+02:00 SERVERNAME kernel: GPR00:
d000000035876470 c000002aba1a3af0 d000000035993288 0000000000000026
2021-10-10T00:13:12.790088+02:00 SERVERNAME kernel: GPR04:
c00000e6bf30ade8 c00000e6bf322a00 0000000000000007 c0000000013ed474
2021-10-10T00:13:12.790090+02:00 SERVERNAME kernel: GPR08:
0000000000000000 c000000000dc16fc 000000e6be550000 000000000000134f
2021-10-10T00:13:12.790091+02:00 SERVERNAME kernel: GPR12:
0000000000004000 c00000000f6c9400 00000000007fffff 0000000000000014
2021-10-10T00:13:12.790093+02:00 SERVERNAME kernel: GPR16:
00000100231aa080 0000010022868988 aaaaaaaaaaaaaaab c000000024ebf778
2021-10-10T00:13:12.790095+02:00 SERVERNAME kernel: GPR20:
0000000000000000 0000000000000000 0000000000000000 c00000e5a1f6eb30
2021-10-10T00:13:12.790097+02:00 SERVERNAME kernel: GPR24:
c00000e5a1f6eb20 00000000000016ba c00000e5a1f6e9c0 c00000e6828cc000
2021-10-10T00:13:12.790098+02:00 SERVERNAME kernel: GPR28:
c0000014b1bc02d0 0000000000000000 c00000e6828cc000 ffffffffffffffe4
2021-10-10T00:13:12.790100+02:00 SERVERNAME kernel: NIP
[d000000035876474] btrfs_run_delayed_refs+0x2b4/0x2c0 [btrfs]
2021-10-10T00:13:12.790101+02:00 SERVERNAME kernel: LR
[d000000035876470] btrfs_run_delayed_refs+0x2b0/0x2c0 [btrfs]
2021-10-10T00:13:12.790103+02:00 SERVERNAME kernel: Call Trace:
2021-10-10T00:13:12.790104+02:00 SERVERNAME kernel: [c000002aba1a3af0]
[d000000035876470] btrfs_run_delayed_refs+0x2b0/0x2c0 [btrfs]
(unreliable)
2021-10-10T00:13:12.790106+02:00 SERVERNAME kernel: [c000002aba1a3bb0]
[d000000035891554] btrfs_commit_transaction+0x74/0xc10 [btrfs]
2021-10-10T00:13:12.790108+02:00 SERVERNAME kernel: [c000002aba1a3c80]
[d0000000358b3328] btrfs_sync_file+0x3a8/0x510 [btrfs]
2021-10-10T00:13:12.790110+02:00 SERVERNAME kernel: [c000002aba1a3d80]
[c000000000408720] vfs_fsync_range+0x70/0x120
2021-10-10T00:13:12.790111+02:00 SERVERNAME kernel: [c000002aba1a3dd0]
[c00000000040886c] do_fsync+0x5c/0xb0
2021-10-10T00:13:12.790113+02:00 SERVERNAME kernel: [c000002aba1a3e10]
[c000000000408cfc] SyS_fdatasync+0x2c/0x40
2021-10-10T00:13:12.790115+02:00 SERVERNAME kernel: [c000002aba1a3e30]
[c00000000000b308] system_call+0x3c/0x130
2021-10-10T00:13:12.790116+02:00 SERVERNAME kernel: Instruction dump:
2021-10-10T00:13:12.790118+02:00 SERVERNAME kernel: e87c0060 3c820000
e8848b48 38a0fffb 4bfe6905 60000000 4bffffb4 3c620000
2021-10-10T00:13:12.790120+02:00 SERVERNAME kernel: e8638b38 7fe4fb78
480e3f55 e8410018 <0fe00000> 4bffff98 60000000 3c4c0012
2021-10-10T00:13:12.790122+02:00 SERVERNAME kernel: ---[ end trace
c9aa23777165dfdc ]---
2021-10-10T00:13:12.790124+02:00 SERVERNAME kernel: BTRFS: error (device
dm-22) in btrfs_run_delayed_refs:2353: errno=-28 No space left
2021-10-10T00:13:12.790125+02:00 SERVERNAME kernel: btrfs_printk: 12
callbacks suppressed
2021-10-10T00:13:12.790127+02:00 SERVERNAME kernel: BTRFS info (device
dm-22): forced readonly
$ btrfs --version
btrfs-progs v4.5.3+20160729
$ btrfs fi show /var
Label: none uuid: f96f4980-4682-4d2d-8d7a-3c0e2c1c6680
Total devices 1 FS bytes used 1.06GiB
devid 1 size 2.50GiB used 2.50GiB path
/dev/mapper/rootvg-varvol
uname -a
Linux SERVERNAME 4.12.14-122.83-default #1 SMP Tue Aug 3 08:37:22 UTC
2021 (c86c48c) ppc64le ppc64le ppc64le GNU/Linux
On the previous Friday after weekly balance:
btrfs fi usage /var
Overall:
Device size: 2.50GiB
Device allocated: 1.73GiB
Device unallocated: 792.75MiB
Device missing: 0.00B
Used: 1.09GiB
Free (estimated): 1.11GiB (min: 739.62MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00B)
Data,single: Size:1.41GiB, Used:1.08GiB
/dev/mapper/rootvg-varvol 1.41GiB
Metadata,DUP: Size:128.00MiB, Used:3.94MiB
/dev/mapper/rootvg-varvol 256.00MiB
System,DUP: Size:32.00MiB, Used:64.00KiB
/dev/mapper/rootvg-varvol 64.00MiB
Unallocated:
/dev/mapper/rootvg-varvol 792.75MiB
I don't have extract of btrfs fi usage /var command during the weekend,
but a script is extracting the Space allocated ("Size") and Used in Data
and Metadata. I observed twice during the weekend space allocated to
metadata is suddenly growing while the metadata used remains the same.
The first time I had enough "Device unallocated" and no problem was
observed, the second (on Sunday after midnight), it leads to FS RO (no
space left).
Is there any situation that can lead to metadata allocation but without
actual usage of metadata?
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Filesystem Read Only due to errno=-28 during metadata allocation
2021-10-13 12:35 Filesystem Read Only due to errno=-28 during metadata allocation mailing
@ 2021-10-18 11:13 ` mailing
2021-10-18 14:09 ` Zygo Blaxell
1 sibling, 0 replies; 7+ messages in thread
From: mailing @ 2021-10-18 11:13 UTC (permalink / raw)
To: linux-btrfs
On 13.10.2021 14:35, mailing@dmilz.net wrote:
> Hello,
>
> I faced issue with btrfs FS /var forced to RO due to errno=-28 (no
> space left).
>
> The server was restarted to bring back FS in RW.
>
> Before reboot:
> $ btrfs fi usage /var -m
> Overall:
> Device size: 2560.00MiB
> Device allocated: 2559.00MiB
> Device unallocated: 1.00MiB
> Device missing: 0.00MiB
> Used: 1116.00MiB
> Free (estimated): 451.25MiB (min: 451.25MiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 13.00MiB (used: 0.00MiB)
>
> Data,single: Size:1559.25MiB, Used:1108.00MiB
> /dev/mapper/rootvg-varvol 1559.25MiB
>
> Metadata,DUP: Size:467.88MiB, Used:3.94MiB
> /dev/mapper/rootvg-varvol 935.75MiB
>
> System,DUP: Size:32.00MiB, Used:0.06MiB
> /dev/mapper/rootvg-varvol 64.00MiB
>
> Unallocated:
> /dev/mapper/rootvg-varvol 1.00MiB
>
> The FS went RO on Sunday, with this trace:
> 2021-10-10T00:13:12.790042+02:00 SERVERNAME kernel: BTRFS: Transaction
> aborted (error -28)
> 2021-10-10T00:13:12.790053+02:00 SERVERNAME kernel: ------------[ cut
> here ]------------
> 2021-10-10T00:13:12.790055+02:00 SERVERNAME kernel: WARNING: CPU: 8
> PID: 8532 at ../fs/btrfs/extent-tree.c:2353
> btrfs_run_delayed_refs+0x2b4/0x2c0 [btrfs]
> 2021-10-10T00:13:12.790057+02:00 SERVERNAME kernel: Modules linked in:
> lin_tape(OEX) pfo(OEX) nfsv3 nfs_acl nfs lockd grace sunrpc fscache
> rpadlpar_io(X) rpaphp(X) tcp_diag udp_diag raw_diag inet_diag
> unix_diag af_packet_diag netlink_diag binfmt_misc af_packet xfs
> libcrc32c st ch ibmveth(X) vmx_crypto gf128mul crct10dif_vpmsum
> uio_pdrv_genirq uio rtc_generic btrfs xor zstd_decompress
> zstd_compress xxhash raid6_pq dm_service_time sd_mod ibmvfc(X)
> scsi_transport_fc crc32c_vpmsum dm_mirror dm_region_hash dm_log sg
> dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_mod
> autofs4
> 2021-10-10T00:13:12.790059+02:00 SERVERNAME kernel: Supported: Yes,
> External
> 2021-10-10T00:13:12.790053+02:00 SERVERNAME kernel: ------------[ cut
> here ]------------
> 2021-10-10T00:13:12.790055+02:00 SERVERNAME kernel: WARNING: CPU: 8
> PID: 8532 at ../fs/btrfs/extent-tree.c:2353
> btrfs_run_delayed_refs+0x2b4/0x2c0 [btrfs]
> 2021-10-10T00:13:12.790057+02:00 SERVERNAME kernel: Modules linked in:
> lin_tape(OEX) pfo(OEX) nfsv3 nfs_acl nfs lockd grace sunrpc fscache
> rpadlpar_io(X) rpaphp(X) tcp_diag udp_diag raw_diag inet_diag
> unix_diag af_packet_diag netlink_diag binfmt_misc af_packet xfs
> libcrc32c st ch ibmveth(X) vmx_crypto gf128mul crct10dif_vpmsum
> uio_pdrv_genirq uio rtc_generic btrfs xor zstd_decompress
> zstd_compress xxhash raid6_pq dm_service_time sd_mod ibmvfc(X)
> scsi_transport_fc crc32c_vpmsum dm_mirror dm_region_hash dm_log sg
> dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_mod
> autofs4
> 2021-10-10T00:13:12.790059+02:00 SERVERNAME kernel: Supported: Yes,
> External
> 2021-10-10T00:13:12.790060+02:00 SERVERNAME kernel: CPU: 8 PID: 8532
> Comm: vpdupdate Tainted: G W OE 4.12.14-122.83-default #1 SLE12-SP5
> 2021-10-10T00:13:12.790076+02:00 SERVERNAME kernel: task:
> c0000068d70c1600 task.stack: c000002aba1a0000
> 2021-10-10T00:13:12.790078+02:00 SERVERNAME kernel: NIP:
> d000000035876474 LR: d000000035876470 CTR: 0000000000000000
> 2021-10-10T00:13:12.790080+02:00 SERVERNAME kernel: REGS:
> c000002aba1a3870 TRAP: 0700 Tainted: G W OE (4.12.14-122.83-default)
> 2021-10-10T00:13:12.790081+02:00 SERVERNAME kernel: MSR:
> 800000010282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE,TM[E]>
> 2021-10-10T00:13:12.790083+02:00 SERVERNAME kernel: CR: 24444422 XER:
> 20040000
> 2021-10-10T00:13:12.790085+02:00 SERVERNAME kernel: CFAR:
> c0000000009c817c SOFTE: 1
> 2021-10-10T00:13:12.790086+02:00 SERVERNAME kernel: GPR00:
> d000000035876470 c000002aba1a3af0 d000000035993288 0000000000000026
> 2021-10-10T00:13:12.790088+02:00 SERVERNAME kernel: GPR04:
> c00000e6bf30ade8 c00000e6bf322a00 0000000000000007 c0000000013ed474
> 2021-10-10T00:13:12.790090+02:00 SERVERNAME kernel: GPR08:
> 0000000000000000 c000000000dc16fc 000000e6be550000 000000000000134f
> 2021-10-10T00:13:12.790091+02:00 SERVERNAME kernel: GPR12:
> 0000000000004000 c00000000f6c9400 00000000007fffff 0000000000000014
> 2021-10-10T00:13:12.790093+02:00 SERVERNAME kernel: GPR16:
> 00000100231aa080 0000010022868988 aaaaaaaaaaaaaaab c000000024ebf778
> 2021-10-10T00:13:12.790095+02:00 SERVERNAME kernel: GPR20:
> 0000000000000000 0000000000000000 0000000000000000 c00000e5a1f6eb30
> 2021-10-10T00:13:12.790097+02:00 SERVERNAME kernel: GPR24:
> c00000e5a1f6eb20 00000000000016ba c00000e5a1f6e9c0 c00000e6828cc000
> 2021-10-10T00:13:12.790098+02:00 SERVERNAME kernel: GPR28:
> c0000014b1bc02d0 0000000000000000 c00000e6828cc000 ffffffffffffffe4
> 2021-10-10T00:13:12.790100+02:00 SERVERNAME kernel: NIP
> [d000000035876474] btrfs_run_delayed_refs+0x2b4/0x2c0 [btrfs]
> 2021-10-10T00:13:12.790101+02:00 SERVERNAME kernel: LR
> [d000000035876470] btrfs_run_delayed_refs+0x2b0/0x2c0 [btrfs]
> 2021-10-10T00:13:12.790103+02:00 SERVERNAME kernel: Call Trace:
> 2021-10-10T00:13:12.790104+02:00 SERVERNAME kernel: [c000002aba1a3af0]
> [d000000035876470] btrfs_run_delayed_refs+0x2b0/0x2c0 [btrfs]
> (unreliable)
> 2021-10-10T00:13:12.790106+02:00 SERVERNAME kernel: [c000002aba1a3bb0]
> [d000000035891554] btrfs_commit_transaction+0x74/0xc10 [btrfs]
> 2021-10-10T00:13:12.790108+02:00 SERVERNAME kernel: [c000002aba1a3c80]
> [d0000000358b3328] btrfs_sync_file+0x3a8/0x510 [btrfs]
> 2021-10-10T00:13:12.790110+02:00 SERVERNAME kernel: [c000002aba1a3d80]
> [c000000000408720] vfs_fsync_range+0x70/0x120
> 2021-10-10T00:13:12.790111+02:00 SERVERNAME kernel: [c000002aba1a3dd0]
> [c00000000040886c] do_fsync+0x5c/0xb0
> 2021-10-10T00:13:12.790113+02:00 SERVERNAME kernel: [c000002aba1a3e10]
> [c000000000408cfc] SyS_fdatasync+0x2c/0x40
> 2021-10-10T00:13:12.790115+02:00 SERVERNAME kernel: [c000002aba1a3e30]
> [c00000000000b308] system_call+0x3c/0x130
> 2021-10-10T00:13:12.790116+02:00 SERVERNAME kernel: Instruction dump:
> 2021-10-10T00:13:12.790118+02:00 SERVERNAME kernel: e87c0060 3c820000
> e8848b48 38a0fffb 4bfe6905 60000000 4bffffb4 3c620000
> 2021-10-10T00:13:12.790120+02:00 SERVERNAME kernel: e8638b38 7fe4fb78
> 480e3f55 e8410018 <0fe00000> 4bffff98 60000000 3c4c0012
> 2021-10-10T00:13:12.790122+02:00 SERVERNAME kernel: ---[ end trace
> c9aa23777165dfdc ]---
> 2021-10-10T00:13:12.790124+02:00 SERVERNAME kernel: BTRFS: error
> (device dm-22) in btrfs_run_delayed_refs:2353: errno=-28 No space left
> 2021-10-10T00:13:12.790125+02:00 SERVERNAME kernel: btrfs_printk: 12
> callbacks suppressed
> 2021-10-10T00:13:12.790127+02:00 SERVERNAME kernel: BTRFS info (device
> dm-22): forced readonly
> 2021-10-10T00:13:12.790060+02:00 SERVERNAME kernel: CPU: 8 PID: 8532
> Comm: vpdupdate Tainted: G W OE 4.12.14-122.83-default #1 SLE12-SP5
> 2021-10-10T00:13:12.790076+02:00 SERVERNAME kernel: task:
> c0000068d70c1600 task.stack: c000002aba1a0000
> 2021-10-10T00:13:12.790078+02:00 SERVERNAME kernel: NIP:
> d000000035876474 LR: d000000035876470 CTR: 0000000000000000
> 2021-10-10T00:13:12.790080+02:00 SERVERNAME kernel: REGS:
> c000002aba1a3870 TRAP: 0700 Tainted: G W OE (4.12.14-122.83-default)
> 2021-10-10T00:13:12.790081+02:00 SERVERNAME kernel: MSR:
> 800000010282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE,TM[E]>
> 2021-10-10T00:13:12.790083+02:00 SERVERNAME kernel: CR: 24444422 XER:
> 20040000
> 2021-10-10T00:13:12.790085+02:00 SERVERNAME kernel: CFAR:
> c0000000009c817c SOFTE: 1
> 2021-10-10T00:13:12.790086+02:00 SERVERNAME kernel: GPR00:
> d000000035876470 c000002aba1a3af0 d000000035993288 0000000000000026
> 2021-10-10T00:13:12.790088+02:00 SERVERNAME kernel: GPR04:
> c00000e6bf30ade8 c00000e6bf322a00 0000000000000007 c0000000013ed474
> 2021-10-10T00:13:12.790090+02:00 SERVERNAME kernel: GPR08:
> 0000000000000000 c000000000dc16fc 000000e6be550000 000000000000134f
> 2021-10-10T00:13:12.790091+02:00 SERVERNAME kernel: GPR12:
> 0000000000004000 c00000000f6c9400 00000000007fffff 0000000000000014
> 2021-10-10T00:13:12.790093+02:00 SERVERNAME kernel: GPR16:
> 00000100231aa080 0000010022868988 aaaaaaaaaaaaaaab c000000024ebf778
> 2021-10-10T00:13:12.790095+02:00 SERVERNAME kernel: GPR20:
> 0000000000000000 0000000000000000 0000000000000000 c00000e5a1f6eb30
> 2021-10-10T00:13:12.790097+02:00 SERVERNAME kernel: GPR24:
> c00000e5a1f6eb20 00000000000016ba c00000e5a1f6e9c0 c00000e6828cc000
> 2021-10-10T00:13:12.790098+02:00 SERVERNAME kernel: GPR28:
> c0000014b1bc02d0 0000000000000000 c00000e6828cc000 ffffffffffffffe4
> 2021-10-10T00:13:12.790100+02:00 SERVERNAME kernel: NIP
> [d000000035876474] btrfs_run_delayed_refs+0x2b4/0x2c0 [btrfs]
> 2021-10-10T00:13:12.790101+02:00 SERVERNAME kernel: LR
> [d000000035876470] btrfs_run_delayed_refs+0x2b0/0x2c0 [btrfs]
> 2021-10-10T00:13:12.790103+02:00 SERVERNAME kernel: Call Trace:
> 2021-10-10T00:13:12.790104+02:00 SERVERNAME kernel: [c000002aba1a3af0]
> [d000000035876470] btrfs_run_delayed_refs+0x2b0/0x2c0 [btrfs]
> (unreliable)
> 2021-10-10T00:13:12.790106+02:00 SERVERNAME kernel: [c000002aba1a3bb0]
> [d000000035891554] btrfs_commit_transaction+0x74/0xc10 [btrfs]
> 2021-10-10T00:13:12.790108+02:00 SERVERNAME kernel: [c000002aba1a3c80]
> [d0000000358b3328] btrfs_sync_file+0x3a8/0x510 [btrfs]
> 2021-10-10T00:13:12.790110+02:00 SERVERNAME kernel: [c000002aba1a3d80]
> [c000000000408720] vfs_fsync_range+0x70/0x120
> 2021-10-10T00:13:12.790111+02:00 SERVERNAME kernel: [c000002aba1a3dd0]
> [c00000000040886c] do_fsync+0x5c/0xb0
> 2021-10-10T00:13:12.790113+02:00 SERVERNAME kernel: [c000002aba1a3e10]
> [c000000000408cfc] SyS_fdatasync+0x2c/0x40
> 2021-10-10T00:13:12.790115+02:00 SERVERNAME kernel: [c000002aba1a3e30]
> [c00000000000b308] system_call+0x3c/0x130
> 2021-10-10T00:13:12.790116+02:00 SERVERNAME kernel: Instruction dump:
> 2021-10-10T00:13:12.790118+02:00 SERVERNAME kernel: e87c0060 3c820000
> e8848b48 38a0fffb 4bfe6905 60000000 4bffffb4 3c620000
> 2021-10-10T00:13:12.790120+02:00 SERVERNAME kernel: e8638b38 7fe4fb78
> 480e3f55 e8410018 <0fe00000> 4bffff98 60000000 3c4c0012
> 2021-10-10T00:13:12.790122+02:00 SERVERNAME kernel: ---[ end trace
> c9aa23777165dfdc ]---
> 2021-10-10T00:13:12.790124+02:00 SERVERNAME kernel: BTRFS: error
> (device dm-22) in btrfs_run_delayed_refs:2353: errno=-28 No space left
> 2021-10-10T00:13:12.790125+02:00 SERVERNAME kernel: btrfs_printk: 12
> callbacks suppressed
> 2021-10-10T00:13:12.790127+02:00 SERVERNAME kernel: BTRFS info (device
> dm-22): forced readonly
>
> $ btrfs --version
> btrfs-progs v4.5.3+20160729
>
> $ btrfs fi show /var
> Label: none uuid: f96f4980-4682-4d2d-8d7a-3c0e2c1c6680
> Total devices 1 FS bytes used 1.06GiB
> devid 1 size 2.50GiB used 2.50GiB path
> /dev/mapper/rootvg-varvol
>
> uname -a
> Linux SERVERNAME 4.12.14-122.83-default #1 SMP Tue Aug 3 08:37:22 UTC
> 2021 (c86c48c) ppc64le ppc64le ppc64le GNU/Linux
>
> On the previous Friday after weekly balance:
> btrfs fi usage /var
> Overall:
> Device size: 2.50GiB
> Device allocated: 1.73GiB
> Device unallocated: 792.75MiB
> Device missing: 0.00B
> Used: 1.09GiB
> Free (estimated): 1.11GiB (min: 739.62MiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 13.00MiB (used: 0.00B)
>
> Data,single: Size:1.41GiB, Used:1.08GiB
> /dev/mapper/rootvg-varvol 1.41GiB
>
> Metadata,DUP: Size:128.00MiB, Used:3.94MiB
> /dev/mapper/rootvg-varvol 256.00MiB
>
> System,DUP: Size:32.00MiB, Used:64.00KiB
> /dev/mapper/rootvg-varvol 64.00MiB
>
> Unallocated:
> /dev/mapper/rootvg-varvol 792.75MiB
>
>
> I don't have extract of btrfs fi usage /var command during the
> weekend, but a script is extracting the Space allocated ("Size") and
> Used in Data and Metadata. I observed twice during the weekend space
> allocated to metadata is suddenly growing while the metadata used
> remains the same. The first time I had enough "Device unallocated" and
> no problem was observed, the second (on Sunday after midnight), it
> leads to FS RO (no space left).
>
> Is there any situation that can lead to metadata allocation but
> without actual usage of metadata?
Same behavior this weekend (but no RO due to enough Device unallocated),
something is allocating space to metadata but is not using it:
### Sat Oct 16 23:59:57 CEST 2021
Overall:
Device size: 5120.00MiB
Device allocated: 2647.25MiB
Device unallocated: 2472.75MiB
Device missing: 0.00MiB
Used: 1097.75MiB
Free (estimated): 2942.38MiB (min:
1706.00MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)
Data,single: Size:1559.25MiB, Used:1089.62MiB
/dev/mapper/rootvg-varvol 1559.25MiB
Metadata,DUP: Size:512.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1024.00MiB
System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB
Unallocated:
/dev/mapper/rootvg-varvol 2472.75MiB
### Sun Oct 17 00:00:32 CEST 2021
Overall:
Device size: 5120.00MiB
Device allocated: 2903.25MiB
Device unallocated: 2216.75MiB
Device missing: 0.00MiB
Used: 1097.44MiB
Free (estimated): 2686.69MiB (min:
1578.31MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)
Data,single: Size:1559.25MiB, Used:1089.31MiB
/dev/mapper/rootvg-varvol 1559.25MiB
Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB
System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB
Unallocated:
/dev/mapper/rootvg-varvol 2216.75MiB
### Sun Oct 17 00:05:27 CEST 2021
Overall:
Device size: 5120.00MiB
Device allocated: 2903.25MiB
Device unallocated: 2216.75MiB
Device missing: 0.00MiB
Used: 1099.69MiB
Free (estimated): 2684.44MiB (min:
1576.06MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)
Data,single: Size:1559.25MiB, Used:1091.56MiB
/dev/mapper/rootvg-varvol 1559.25MiB
Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB
System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB
Unallocated:
/dev/mapper/rootvg-varvol 2216.75MiB
### Sun Oct 17 00:05:32 CEST 2021
Overall:
Device size: 5120.00MiB
Device allocated: 3159.25MiB
Device unallocated: 1960.75MiB
Device missing: 0.00MiB
Used: 1100.25MiB
Free (estimated): 2427.88MiB (min:
1447.50MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)
Data,single: Size:1559.25MiB, Used:1092.12MiB
/dev/mapper/rootvg-varvol 1559.25MiB
Metadata,DUP: Size:768.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1536.00MiB
System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB
Unallocated:
/dev/mapper/rootvg-varvol 1960.75MiB
### Sun Oct 17 00:12:53 CEST 2021
Overall:
Device size: 5120.00MiB
Device allocated: 3159.25MiB
Device unallocated: 1960.75MiB
Device missing: 0.00MiB
Used: 1100.62MiB
Free (estimated): 2427.50MiB (min:
1447.12MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)
Data,single: Size:1559.25MiB, Used:1092.50MiB
/dev/mapper/rootvg-varvol 1559.25MiB
Metadata,DUP: Size:768.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1536.00MiB
System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB
Unallocated:
/dev/mapper/rootvg-varvol 1960.75MiB
### Sun Oct 17 00:12:58 CEST 2021
Overall:
Device size: 5120.00MiB
Device allocated: 2903.25MiB
Device unallocated: 2216.75MiB
Device missing: 0.00MiB
Used: 1100.69MiB
Free (estimated): 2683.44MiB (min:
1575.06MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.12MiB)
Data,single: Size:1559.25MiB, Used:1092.56MiB
/dev/mapper/rootvg-varvol 1559.25MiB
Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB
System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB
Unallocated:
/dev/mapper/rootvg-varvol 2216.75MiB
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Filesystem Read Only due to errno=-28 during metadata allocation
2021-10-13 12:35 Filesystem Read Only due to errno=-28 during metadata allocation mailing
2021-10-18 11:13 ` mailing
@ 2021-10-18 14:09 ` Zygo Blaxell
2021-10-19 10:57 ` mailing
1 sibling, 1 reply; 7+ messages in thread
From: Zygo Blaxell @ 2021-10-18 14:09 UTC (permalink / raw)
To: mailing; +Cc: linux-btrfs
On Wed, Oct 13, 2021 at 02:35:39PM +0200, mailing@dmilz.net wrote:
> Hello,
>
> I faced issue with btrfs FS /var forced to RO due to errno=-28 (no space
> left).
>
> The server was restarted to bring back FS in RW.
>
> Before reboot:
> $ btrfs fi usage /var -m
> Overall:
> Device size: 2560.00MiB
> Device allocated: 2559.00MiB
> Device unallocated: 1.00MiB
> Device missing: 0.00MiB
> Used: 1116.00MiB
> Free (estimated): 451.25MiB (min: 451.25MiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 13.00MiB (used: 0.00MiB)
>
> Data,single: Size:1559.25MiB, Used:1108.00MiB
> /dev/mapper/rootvg-varvol 1559.25MiB
>
> Metadata,DUP: Size:467.88MiB, Used:3.94MiB
> /dev/mapper/rootvg-varvol 935.75MiB
>
> System,DUP: Size:32.00MiB, Used:0.06MiB
> /dev/mapper/rootvg-varvol 64.00MiB
>
> Unallocated:
> /dev/mapper/rootvg-varvol 1.00MiB
This is a tiny filesystem, below the minimum practical size for separate
data and metadata block groups, but above the size for mixed block
groups to be the mkfs default. It might be better to format it with
the mixed-bg option.
> The FS went RO on Sunday, with this trace:
> 2021-10-10T00:13:12.790042+02:00 SERVERNAME kernel: BTRFS: Transaction
> aborted (error -28)
[...]
> 2021-10-10T00:13:12.790124+02:00 SERVERNAME kernel: BTRFS: error (device
> dm-22) in btrfs_run_delayed_refs:2353: errno=-28 No space left
> 2021-10-10T00:13:12.790125+02:00 SERVERNAME kernel: btrfs_printk: 12
> callbacks suppressed
> 2021-10-10T00:13:12.790127+02:00 SERVERNAME kernel: BTRFS info (device
> dm-22): forced readonly
>
> $ btrfs --version
> btrfs-progs v4.5.3+20160729
>
> $ btrfs fi show /var
> Label: none uuid: f96f4980-4682-4d2d-8d7a-3c0e2c1c6680
> Total devices 1 FS bytes used 1.06GiB
> devid 1 size 2.50GiB used 2.50GiB path /dev/mapper/rootvg-varvol
>
> uname -a
> Linux SERVERNAME 4.12.14-122.83-default #1 SMP Tue Aug 3 08:37:22 UTC 2021
> (c86c48c) ppc64le ppc64le ppc64le GNU/Linux
This kernel is 4 years old. It may have additional practical problems in
the implementation beyond the design-level issues I've described here.
> On the previous Friday after weekly balance:
> btrfs fi usage /var
> Overall:
> Device size: 2.50GiB
> Device allocated: 1.73GiB
> Device unallocated: 792.75MiB
> Device missing: 0.00B
> Used: 1.09GiB
> Free (estimated): 1.11GiB (min: 739.62MiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 13.00MiB (used: 0.00B)
>
> Data,single: Size:1.41GiB, Used:1.08GiB
> /dev/mapper/rootvg-varvol 1.41GiB
>
> Metadata,DUP: Size:128.00MiB, Used:3.94MiB
> /dev/mapper/rootvg-varvol 256.00MiB
>
> System,DUP: Size:32.00MiB, Used:64.00KiB
> /dev/mapper/rootvg-varvol 64.00MiB
>
> Unallocated:
> /dev/mapper/rootvg-varvol 792.75MiB
>
>
> I don't have extract of btrfs fi usage /var command during the weekend, but
> a script is extracting the Space allocated ("Size") and Used in Data and
> Metadata. I observed twice during the weekend space allocated to metadata is
> suddenly growing while the metadata used remains the same. The first time I
> had enough "Device unallocated" and no problem was observed, the second (on
> Sunday after midnight), it leads to FS RO (no space left).
>
> Is there any situation that can lead to metadata allocation but without
> actual usage of metadata?
Some btrfs maintenance operations lock block groups to prevent concurrent
modification, such as balance, scrub, and discard. When this occurs,
any free space that has been allocated in the metadata block groups
is temporarily unavailable, and if there's no free space in any other
block group, then btrfs will need to allocate another metadata block
group to continue. (The same happens with data block groups but the
impact is smaller due to the much larger data sizes.) The worst case
for metadata on a normal filesystem is:
raid_profile_redundancy (2 for dup metadata) * (
chunk_size (usually 1GB) * (
number_of_disks (for scrub) +
1 (for discard) +
1 (for balance)
) + global_reserve (up to 512MB)
)
i.e. on a single-disk filesystem with dup metadata, there should be at
least 7GB allocated or available for metadata (3.5GB dup metadata takes
7GB of raw space, you can have either the 3.5GB allocated-but-free
metadata or 7GB unallocated, or any combination totalling at least 7GB).
Obviously, keeping 7GB reserved for metadata doesn't work on a device
that is only 2.5GB to start with. Even if you elect not to use discard
or scrub, you'd still need 2.5GB for dup metadata.
btrfs has two ways to deal with this: mixed block groups (mixed-bg mkfs
option), which puts all the space into a single pool with no distinction
between data and metadata, and reducing block group chunk size for
non-mixed-bg filesystems.
On a filesystem this size, the block groups use 128MB chunks instead
of 1GB. Global reserve is also reduced (it is computed based on device
size and capped at 512MB). In this case, the worst case metadata free
space requirement is 128MB chunk size * (1 disk + 2) + global_reserve
which works out to 397MB of free metadata space or 794MB of unallocated
space with dup metadata.
You have 467MB allocated to metadata and most of it is free space, which
is theoretically enough, but if you count space in terms of block groups,
it's exactly full--if the first 3 block groups are locked, the 4th block
group is the last one where space might be allocated, and it's an odd
size so it might not be large enough.
Mixed block groups (mixed-bg mkfs option) put all the space is in a
single pot that can be allocated to metadata or data blocks as needed.
This is the default for filesystems below 1GB, but it's a good idea to
use mixed-bg for larger filesystems as well because the metadata block
group locking cost is so high relative to the filesystem size.
While it's possible to use non-mixed block groups on a filesystem that
has only a few GB, it's not possible to use a significant number of
btrfs features, including scrub, balance, RAID disk replacement, online
conversion to other RAID profiles, device shrink, or discard, due to
the requirement to have an extra unlocked block group with available
free space during these operations.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Filesystem Read Only due to errno=-28 during metadata allocation
2021-10-18 14:09 ` Zygo Blaxell
@ 2021-10-19 10:57 ` mailing
2021-10-19 13:26 ` Zygo Blaxell
0 siblings, 1 reply; 7+ messages in thread
From: mailing @ 2021-10-19 10:57 UTC (permalink / raw)
To: Zygo Blaxell; +Cc: linux-btrfs
On 18.10.2021 16:09, Zygo Blaxell wrote:
> On Wed, Oct 13, 2021 at 02:35:39PM +0200, mailing@dmilz.net wrote:
>> Hello,
>>
>> I faced issue with btrfs FS /var forced to RO due to errno=-28 (no
>> space
>> left).
>>
>> The server was restarted to bring back FS in RW.
>>
>> Before reboot:
>> $ btrfs fi usage /var -m
>> Overall:
>> Device size: 2560.00MiB
>> Device allocated: 2559.00MiB
>> Device unallocated: 1.00MiB
>> Device missing: 0.00MiB
>> Used: 1116.00MiB
>> Free (estimated): 451.25MiB (min: 451.25MiB)
>> Data ratio: 1.00
>> Metadata ratio: 2.00
>> Global reserve: 13.00MiB (used: 0.00MiB)
>>
>> Data,single: Size:1559.25MiB, Used:1108.00MiB
>> /dev/mapper/rootvg-varvol 1559.25MiB
>>
>> Metadata,DUP: Size:467.88MiB, Used:3.94MiB
>> /dev/mapper/rootvg-varvol 935.75MiB
>>
>> System,DUP: Size:32.00MiB, Used:0.06MiB
>> /dev/mapper/rootvg-varvol 64.00MiB
>>
>> Unallocated:
>> /dev/mapper/rootvg-varvol 1.00MiB
>
> This is a tiny filesystem, below the minimum practical size for
> separate
> data and metadata block groups, but above the size for mixed block
> groups to be the mkfs default. It might be better to format it with
> the mixed-bg option.
>
>> The FS went RO on Sunday, with this trace:
>> 2021-10-10T00:13:12.790042+02:00 SERVERNAME kernel: BTRFS: Transaction
>> aborted (error -28)
> [...]
>> 2021-10-10T00:13:12.790124+02:00 SERVERNAME kernel: BTRFS: error
>> (device
>> dm-22) in btrfs_run_delayed_refs:2353: errno=-28 No space left
>> 2021-10-10T00:13:12.790125+02:00 SERVERNAME kernel: btrfs_printk: 12
>> callbacks suppressed
>> 2021-10-10T00:13:12.790127+02:00 SERVERNAME kernel: BTRFS info (device
>> dm-22): forced readonly
>>
>> $ btrfs --version
>> btrfs-progs v4.5.3+20160729
>>
>> $ btrfs fi show /var
>> Label: none uuid: f96f4980-4682-4d2d-8d7a-3c0e2c1c6680
>> Total devices 1 FS bytes used 1.06GiB
>> devid 1 size 2.50GiB used 2.50GiB path
>> /dev/mapper/rootvg-varvol
>>
>> uname -a
>> Linux SERVERNAME 4.12.14-122.83-default #1 SMP Tue Aug 3 08:37:22 UTC
>> 2021
>> (c86c48c) ppc64le ppc64le ppc64le GNU/Linux
>
> This kernel is 4 years old. It may have additional practical problems
> in
> the implementation beyond the design-level issues I've described here.
>
>> On the previous Friday after weekly balance:
>> btrfs fi usage /var
>> Overall:
>> Device size: 2.50GiB
>> Device allocated: 1.73GiB
>> Device unallocated: 792.75MiB
>> Device missing: 0.00B
>> Used: 1.09GiB
>> Free (estimated): 1.11GiB (min: 739.62MiB)
>> Data ratio: 1.00
>> Metadata ratio: 2.00
>> Global reserve: 13.00MiB (used: 0.00B)
>>
>> Data,single: Size:1.41GiB, Used:1.08GiB
>> /dev/mapper/rootvg-varvol 1.41GiB
>>
>> Metadata,DUP: Size:128.00MiB, Used:3.94MiB
>> /dev/mapper/rootvg-varvol 256.00MiB
>>
>> System,DUP: Size:32.00MiB, Used:64.00KiB
>> /dev/mapper/rootvg-varvol 64.00MiB
>>
>> Unallocated:
>> /dev/mapper/rootvg-varvol 792.75MiB
>>
>>
>> I don't have extract of btrfs fi usage /var command during the
>> weekend, but
>> a script is extracting the Space allocated ("Size") and Used in Data
>> and
>> Metadata. I observed twice during the weekend space allocated to
>> metadata is
>> suddenly growing while the metadata used remains the same. The first
>> time I
>> had enough "Device unallocated" and no problem was observed, the
>> second (on
>> Sunday after midnight), it leads to FS RO (no space left).
>>
>> Is there any situation that can lead to metadata allocation but
>> without
>> actual usage of metadata?
>
> Some btrfs maintenance operations lock block groups to prevent
> concurrent
> modification, such as balance, scrub, and discard. When this occurs,
> any free space that has been allocated in the metadata block groups
> is temporarily unavailable, and if there's no free space in any other
> block group, then btrfs will need to allocate another metadata block
> group to continue. (The same happens with data block groups but the
> impact is smaller due to the much larger data sizes.) The worst case
> for metadata on a normal filesystem is:
>
> raid_profile_redundancy (2 for dup metadata) * (
> chunk_size (usually 1GB) * (
> number_of_disks (for scrub) +
> 1 (for discard) +
> 1 (for balance)
> ) + global_reserve (up to 512MB)
> )
>
> i.e. on a single-disk filesystem with dup metadata, there should be at
> least 7GB allocated or available for metadata (3.5GB dup metadata takes
> 7GB of raw space, you can have either the 3.5GB allocated-but-free
> metadata or 7GB unallocated, or any combination totalling at least
> 7GB).
>
> Obviously, keeping 7GB reserved for metadata doesn't work on a device
> that is only 2.5GB to start with. Even if you elect not to use discard
> or scrub, you'd still need 2.5GB for dup metadata.
>
Since this incident, the FS has been extended to 5GB.
But in our case the chunk size for metadata is 128MB, so:
2 [dup metadata] * ( 128 * ( 1 [disk] + 1 [discard] + 1 [balance] ) +
512 [Global Reserve] ) = 1792 MB ?
> btrfs has two ways to deal with this: mixed block groups (mixed-bg mkfs
> option), which puts all the space into a single pool with no
> distinction
> between data and metadata, and reducing block group chunk size for
> non-mixed-bg filesystems.
>
> On a filesystem this size, the block groups use 128MB chunks instead
> of 1GB. Global reserve is also reduced (it is computed based on device
> size and capped at 512MB). In this case, the worst case metadata free
> space requirement is 128MB chunk size * (1 disk + 2) + global_reserve
> which works out to 397MB of free metadata space or 794MB of unallocated
> space with dup metadata.
>
> You have 467MB allocated to metadata and most of it is free space,
> which
> is theoretically enough, but if you count space in terms of block
> groups,
> it's exactly full--if the first 3 block groups are locked, the 4th
> block
> group is the last one where space might be allocated, and it's an odd
> size so it might not be large enough.
>
From obervation the last chunk allocated is not reported as totally
used, so I see size for data/metdata as "odd size" (not an integer
number of chunks).
> Mixed block groups (mixed-bg mkfs option) put all the space is in a
> single pot that can be allocated to metadata or data blocks as needed.
> This is the default for filesystems below 1GB, but it's a good idea to
> use mixed-bg for larger filesystems as well because the metadata block
> group locking cost is so high relative to the filesystem size.
>
> While it's possible to use non-mixed block groups on a filesystem that
> has only a few GB, it's not possible to use a significant number of
> btrfs features, including scrub, balance, RAID disk replacement, online
> conversion to other RAID profiles, device shrink, or discard, due to
> the requirement to have an extra unlocked block group with available
> free space during these operations.
Last weekend I had the same behavior, the size allocated to metadata
went from 128MB to up to 768MB, so up to 6 * 12* MB metadata but the
metdata usage didn't grow:
### Sat Oct 16 23:59:57 CEST 2021
Overall:
Device size: 5120.00MiB
Device allocated: 2647.25MiB
Device unallocated: 2472.75MiB
Device missing: 0.00MiB
Used: 1097.75MiB
Free (estimated): 2942.38MiB (min:
1706.00MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)
Data,single: Size:1559.25MiB, Used:1089.62MiB
/dev/mapper/rootvg-varvol 1559.25MiB
Metadata,DUP: Size:512.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1024.00MiB
System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB
Unallocated:
/dev/mapper/rootvg-varvol 2472.75MiB
### Sun Oct 17 00:00:32 CEST 2021
Overall:
Device size: 5120.00MiB
Device allocated: 2903.25MiB
Device unallocated: 2216.75MiB
Device missing: 0.00MiB
Used: 1097.44MiB
Free (estimated): 2686.69MiB (min:
1578.31MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)
Data,single: Size:1559.25MiB, Used:1089.31MiB
/dev/mapper/rootvg-varvol 1559.25MiB
Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB
System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB
Unallocated:
/dev/mapper/rootvg-varvol 2216.75MiB
### Sun Oct 17 00:05:27 CEST 2021
Overall:
Device size: 5120.00MiB
Device allocated: 2903.25MiB
Device unallocated: 2216.75MiB
Device missing: 0.00MiB
Used: 1099.69MiB
Free (estimated): 2684.44MiB (min:
1576.06MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)
Data,single: Size:1559.25MiB, Used:1091.56MiB
/dev/mapper/rootvg-varvol 1559.25MiB
Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB
System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB
Unallocated:
/dev/mapper/rootvg-varvol 2216.75MiB
### Sun Oct 17 00:05:32 CEST 2021
Overall:
Device size: 5120.00MiB
Device allocated: 3159.25MiB
Device unallocated: 1960.75MiB
Device missing: 0.00MiB
Used: 1100.25MiB
Free (estimated): 2427.88MiB (min:
1447.50MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)
Data,single: Size:1559.25MiB, Used:1092.12MiB
/dev/mapper/rootvg-varvol 1559.25MiB
Metadata,DUP: Size:768.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1536.00MiB
System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB
Unallocated:
/dev/mapper/rootvg-varvol 1960.75MiB
### Sun Oct 17 00:12:53 CEST 2021
Overall:
Device size: 5120.00MiB
Device allocated: 3159.25MiB
Device unallocated: 1960.75MiB
Device missing: 0.00MiB
Used: 1100.62MiB
Free (estimated): 2427.50MiB (min:
1447.12MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)
Data,single: Size:1559.25MiB, Used:1092.50MiB
/dev/mapper/rootvg-varvol 1559.25MiB
Metadata,DUP: Size:768.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1536.00MiB
System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB
Unallocated:
/dev/mapper/rootvg-varvol 1960.75MiB
### Sun Oct 17 00:12:58 CEST 2021
Overall:
Device size: 5120.00MiB
Device allocated: 2903.25MiB
Device unallocated: 2216.75MiB
Device missing: 0.00MiB
Used: 1100.69MiB
Free (estimated): 2683.44MiB (min:
1575.06MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.12MiB)
Data,single: Size:1559.25MiB, Used:1092.56MiB
/dev/mapper/rootvg-varvol 1559.25MiB
Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB
System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB
Unallocated:
/dev/mapper/rootvg-varvol 2216.75MiB
So if I understand properly it might be due to chunk being needed for
balance/discard or scrub.
Is there 3rd party tools which are also able to lock metadata block
group as well? It seems to happens at the same time, when backup is
running (spectrum), and/or HANA backup and/or ReaR backup as well.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Filesystem Read Only due to errno=-28 during metadata allocation
2021-10-19 10:57 ` mailing
@ 2021-10-19 13:26 ` Zygo Blaxell
2021-10-25 9:53 ` mailing
0 siblings, 1 reply; 7+ messages in thread
From: Zygo Blaxell @ 2021-10-19 13:26 UTC (permalink / raw)
To: mailing; +Cc: linux-btrfs
On Tue, Oct 19, 2021 at 12:57:39PM +0200, mailing@dmilz.net wrote:
> On 18.10.2021 16:09, Zygo Blaxell wrote:
> > On Wed, Oct 13, 2021 at 02:35:39PM +0200, mailing@dmilz.net wrote:
> > > Hello,
> > >
> > > I faced issue with btrfs FS /var forced to RO due to errno=-28 (no
> > > space
> > > left).
> > Obviously, keeping 7GB reserved for metadata doesn't work on a device
> > that is only 2.5GB to start with. Even if you elect not to use discard
> > or scrub, you'd still need 2.5GB for dup metadata.
> >
>
> Since this incident, the FS has been extended to 5GB.
>
> But in our case the chunk size for metadata is 128MB, so:
> 2 [dup metadata] * ( 128 * ( 1 [disk] + 1 [discard] + 1 [balance] ) + 512
> [Global Reserve] ) = 1792 MB ?
The global reserve is normally much less than 512 MB on smaller disks.
It is only 13MB on the 5GB disks.
> > While it's possible to use non-mixed block groups on a filesystem that
> > has only a few GB, it's not possible to use a significant number of
> > btrfs features, including scrub, balance, RAID disk replacement, online
> > conversion to other RAID profiles, device shrink, or discard, due to
> > the requirement to have an extra unlocked block group with available
> > free space during these operations.
>
> Last weekend I had the same behavior, the size allocated to metadata went
> from 128MB to up to 768MB, so up to 6 * 12* MB metadata but the metdata
> usage didn't grow:
>
> ### Sat Oct 16 23:59:57 CEST 2021
> Overall:
> Device size: 5120.00MiB
> Device allocated: 2647.25MiB
> Device unallocated: 2472.75MiB
> Device missing: 0.00MiB
> Used: 1097.75MiB
> Free (estimated): 2942.38MiB (min: 1706.00MiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 13.00MiB (used: 0.00MiB)
>
> Data,single: Size:1559.25MiB, Used:1089.62MiB
> /dev/mapper/rootvg-varvol 1559.25MiB
>
> Metadata,DUP: Size:512.00MiB, Used:4.00MiB
> /dev/mapper/rootvg-varvol 1024.00MiB
>
> System,DUP: Size:32.00MiB, Used:0.06MiB
> /dev/mapper/rootvg-varvol 64.00MiB
>
> Unallocated:
> /dev/mapper/rootvg-varvol 2472.75MiB
>
>
>
> ### Sun Oct 17 00:00:32 CEST 2021
> Overall:
> Device size: 5120.00MiB
> Device allocated: 2903.25MiB
> Device unallocated: 2216.75MiB
> Device missing: 0.00MiB
> Used: 1097.44MiB
> Free (estimated): 2686.69MiB (min: 1578.31MiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 13.00MiB (used: 0.00MiB)
>
> Data,single: Size:1559.25MiB, Used:1089.31MiB
> /dev/mapper/rootvg-varvol 1559.25MiB
>
> Metadata,DUP: Size:640.00MiB, Used:4.00MiB
> /dev/mapper/rootvg-varvol 1280.00MiB
>
> System,DUP: Size:32.00MiB, Used:0.06MiB
> /dev/mapper/rootvg-varvol 64.00MiB
>
> Unallocated:
> /dev/mapper/rootvg-varvol 2216.75MiB
>
>
> ### Sun Oct 17 00:05:27 CEST 2021
> Overall:
> Device size: 5120.00MiB
> Device allocated: 2903.25MiB
> Device unallocated: 2216.75MiB
> Device missing: 0.00MiB
> Used: 1099.69MiB
> Free (estimated): 2684.44MiB (min: 1576.06MiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 13.00MiB (used: 0.00MiB)
>
> Data,single: Size:1559.25MiB, Used:1091.56MiB
> /dev/mapper/rootvg-varvol 1559.25MiB
>
> Metadata,DUP: Size:640.00MiB, Used:4.00MiB
> /dev/mapper/rootvg-varvol 1280.00MiB
>
> System,DUP: Size:32.00MiB, Used:0.06MiB
> /dev/mapper/rootvg-varvol 64.00MiB
>
> Unallocated:
> /dev/mapper/rootvg-varvol 2216.75MiB
>
>
> ### Sun Oct 17 00:05:32 CEST 2021
> Overall:
> Device size: 5120.00MiB
> Device allocated: 3159.25MiB
> Device unallocated: 1960.75MiB
> Device missing: 0.00MiB
> Used: 1100.25MiB
> Free (estimated): 2427.88MiB (min: 1447.50MiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 13.00MiB (used: 0.00MiB)
>
> Data,single: Size:1559.25MiB, Used:1092.12MiB
> /dev/mapper/rootvg-varvol 1559.25MiB
>
> Metadata,DUP: Size:768.00MiB, Used:4.00MiB
> /dev/mapper/rootvg-varvol 1536.00MiB
>
> System,DUP: Size:32.00MiB, Used:0.06MiB
> /dev/mapper/rootvg-varvol 64.00MiB
>
> Unallocated:
> /dev/mapper/rootvg-varvol 1960.75MiB
>
> ### Sun Oct 17 00:12:53 CEST 2021
> Overall:
> Device size: 5120.00MiB
> Device allocated: 3159.25MiB
> Device unallocated: 1960.75MiB
> Device missing: 0.00MiB
> Used: 1100.62MiB
> Free (estimated): 2427.50MiB (min: 1447.12MiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 13.00MiB (used: 0.00MiB)
>
> Data,single: Size:1559.25MiB, Used:1092.50MiB
> /dev/mapper/rootvg-varvol 1559.25MiB
>
> Metadata,DUP: Size:768.00MiB, Used:4.00MiB
> /dev/mapper/rootvg-varvol 1536.00MiB
>
> System,DUP: Size:32.00MiB, Used:0.06MiB
> /dev/mapper/rootvg-varvol 64.00MiB
>
> Unallocated:
> /dev/mapper/rootvg-varvol 1960.75MiB
> ### Sun Oct 17 00:12:58 CEST 2021
> Overall:
> Device size: 5120.00MiB
> Device allocated: 2903.25MiB
> Device unallocated: 2216.75MiB
> Device missing: 0.00MiB
> Used: 1100.69MiB
> Free (estimated): 2683.44MiB (min: 1575.06MiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 13.00MiB (used: 0.12MiB)
>
> Data,single: Size:1559.25MiB, Used:1092.56MiB
> /dev/mapper/rootvg-varvol 1559.25MiB
>
> Metadata,DUP: Size:640.00MiB, Used:4.00MiB
> /dev/mapper/rootvg-varvol 1280.00MiB
>
> System,DUP: Size:32.00MiB, Used:0.06MiB
> /dev/mapper/rootvg-varvol 64.00MiB
>
> Unallocated:
> /dev/mapper/rootvg-varvol 2216.75MiB
>
> So if I understand properly it might be due to chunk being needed for
> balance/discard or scrub.
> Is there 3rd party tools which are also able to lock metadata block group as
> well? It seems to happens at the same time, when backup is running
> (spectrum), and/or HANA backup and/or ReaR backup as well.
If the 3rd party tools are triggering any of the btrfs maintenance
functions then they'll lock block groups; otherwise, normal filesystem
operations don't generally hold locks at the block group level.
There might be some additional issues with the 4.12 kernel that cause
it to allocate more than the minimum metadata. I recall there were
some problems with old kernels where multiple threads allocating at
the same time will all grab their own chunks, but I'm not sure which
kernel those were fixed in. There are also changes to the allocator's
behavior with the 'ssd' and 'nossd' mount options in 4.14 which might
cause these effects.
Some temporary overallocation is normal. It's likely it would have
worked even without the overallocation, the overallocation has a pretty
large impact on these tiny filesystems. This seems a bit higher than
the amount I've come to expect from modern btrfs.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Filesystem Read Only due to errno=-28 during metadata allocation
2021-10-19 13:26 ` Zygo Blaxell
@ 2021-10-25 9:53 ` mailing
2021-10-25 10:32 ` Nikolay Borisov
0 siblings, 1 reply; 7+ messages in thread
From: mailing @ 2021-10-25 9:53 UTC (permalink / raw)
To: Zygo Blaxell; +Cc: linux-btrfs
On 19.10.2021 15:26, Zygo Blaxell wrote:
> On Tue, Oct 19, 2021 at 12:57:39PM +0200, mailing@dmilz.net wrote:
>> On 18.10.2021 16:09, Zygo Blaxell wrote:
>> > On Wed, Oct 13, 2021 at 02:35:39PM +0200, mailing@dmilz.net wrote:
>> > > Hello,
>> > >
>> > > I faced issue with btrfs FS /var forced to RO due to errno=-28 (no
>> > > space
>> > > left).
>
>> > Obviously, keeping 7GB reserved for metadata doesn't work on a device
>> > that is only 2.5GB to start with. Even if you elect not to use discard
>> > or scrub, you'd still need 2.5GB for dup metadata.
>> >
>>
>> Since this incident, the FS has been extended to 5GB.
>>
>> But in our case the chunk size for metadata is 128MB, so:
>> 2 [dup metadata] * ( 128 * ( 1 [disk] + 1 [discard] + 1 [balance] ) +
>> 512
>> [Global Reserve] ) = 1792 MB ?
>
> The global reserve is normally much less than 512 MB on smaller disks.
> It is only 13MB on the 5GB disks.
>
>> > While it's possible to use non-mixed block groups on a filesystem that
>> > has only a few GB, it's not possible to use a significant number of
>> > btrfs features, including scrub, balance, RAID disk replacement, online
>> > conversion to other RAID profiles, device shrink, or discard, due to
>> > the requirement to have an extra unlocked block group with available
>> > free space during these operations.
>>
>> Last weekend I had the same behavior, the size allocated to metadata
>> went
>> from 128MB to up to 768MB, so up to 6 * 12* MB metadata but the
>> metdata
>> usage didn't grow:
>>
>> ### Sat Oct 16 23:59:57 CEST 2021
>> Overall:
>> Device size: 5120.00MiB
>> Device allocated: 2647.25MiB
>> Device unallocated: 2472.75MiB
>> Device missing: 0.00MiB
>> Used: 1097.75MiB
>> Free (estimated): 2942.38MiB (min:
>> 1706.00MiB)
>> Data ratio: 1.00
>> Metadata ratio: 2.00
>> Global reserve: 13.00MiB (used:
>> 0.00MiB)
>>
>> Data,single: Size:1559.25MiB, Used:1089.62MiB
>> /dev/mapper/rootvg-varvol 1559.25MiB
>>
>> Metadata,DUP: Size:512.00MiB, Used:4.00MiB
>> /dev/mapper/rootvg-varvol 1024.00MiB
>>
>> System,DUP: Size:32.00MiB, Used:0.06MiB
>> /dev/mapper/rootvg-varvol 64.00MiB
>>
>> Unallocated:
>> /dev/mapper/rootvg-varvol 2472.75MiB
>>
>>
>>
>> ### Sun Oct 17 00:00:32 CEST 2021
>> Overall:
>> Device size: 5120.00MiB
>> Device allocated: 2903.25MiB
>> Device unallocated: 2216.75MiB
>> Device missing: 0.00MiB
>> Used: 1097.44MiB
>> Free (estimated): 2686.69MiB (min:
>> 1578.31MiB)
>> Data ratio: 1.00
>> Metadata ratio: 2.00
>> Global reserve: 13.00MiB (used:
>> 0.00MiB)
>>
>> Data,single: Size:1559.25MiB, Used:1089.31MiB
>> /dev/mapper/rootvg-varvol 1559.25MiB
>>
>> Metadata,DUP: Size:640.00MiB, Used:4.00MiB
>> /dev/mapper/rootvg-varvol 1280.00MiB
>>
>> System,DUP: Size:32.00MiB, Used:0.06MiB
>> /dev/mapper/rootvg-varvol 64.00MiB
>>
>> Unallocated:
>> /dev/mapper/rootvg-varvol 2216.75MiB
>>
>>
>> ### Sun Oct 17 00:05:27 CEST 2021
>> Overall:
>> Device size: 5120.00MiB
>> Device allocated: 2903.25MiB
>> Device unallocated: 2216.75MiB
>> Device missing: 0.00MiB
>> Used: 1099.69MiB
>> Free (estimated): 2684.44MiB (min:
>> 1576.06MiB)
>> Data ratio: 1.00
>> Metadata ratio: 2.00
>> Global reserve: 13.00MiB (used:
>> 0.00MiB)
>>
>> Data,single: Size:1559.25MiB, Used:1091.56MiB
>> /dev/mapper/rootvg-varvol 1559.25MiB
>>
>> Metadata,DUP: Size:640.00MiB, Used:4.00MiB
>> /dev/mapper/rootvg-varvol 1280.00MiB
>>
>> System,DUP: Size:32.00MiB, Used:0.06MiB
>> /dev/mapper/rootvg-varvol 64.00MiB
>>
>> Unallocated:
>> /dev/mapper/rootvg-varvol 2216.75MiB
>>
>>
>> ### Sun Oct 17 00:05:32 CEST 2021
>> Overall:
>> Device size: 5120.00MiB
>> Device allocated: 3159.25MiB
>> Device unallocated: 1960.75MiB
>> Device missing: 0.00MiB
>> Used: 1100.25MiB
>> Free (estimated): 2427.88MiB (min:
>> 1447.50MiB)
>> Data ratio: 1.00
>> Metadata ratio: 2.00
>> Global reserve: 13.00MiB (used:
>> 0.00MiB)
>>
>> Data,single: Size:1559.25MiB, Used:1092.12MiB
>> /dev/mapper/rootvg-varvol 1559.25MiB
>>
>> Metadata,DUP: Size:768.00MiB, Used:4.00MiB
>> /dev/mapper/rootvg-varvol 1536.00MiB
>>
>> System,DUP: Size:32.00MiB, Used:0.06MiB
>> /dev/mapper/rootvg-varvol 64.00MiB
>>
>> Unallocated:
>> /dev/mapper/rootvg-varvol 1960.75MiB
>>
>> ### Sun Oct 17 00:12:53 CEST 2021
>> Overall:
>> Device size: 5120.00MiB
>> Device allocated: 3159.25MiB
>> Device unallocated: 1960.75MiB
>> Device missing: 0.00MiB
>> Used: 1100.62MiB
>> Free (estimated): 2427.50MiB (min:
>> 1447.12MiB)
>> Data ratio: 1.00
>> Metadata ratio: 2.00
>> Global reserve: 13.00MiB (used:
>> 0.00MiB)
>>
>> Data,single: Size:1559.25MiB, Used:1092.50MiB
>> /dev/mapper/rootvg-varvol 1559.25MiB
>>
>> Metadata,DUP: Size:768.00MiB, Used:4.00MiB
>> /dev/mapper/rootvg-varvol 1536.00MiB
>>
>> System,DUP: Size:32.00MiB, Used:0.06MiB
>> /dev/mapper/rootvg-varvol 64.00MiB
>>
>> Unallocated:
>> /dev/mapper/rootvg-varvol 1960.75MiB
>> ### Sun Oct 17 00:12:58 CEST 2021
>> Overall:
>> Device size: 5120.00MiB
>> Device allocated: 2903.25MiB
>> Device unallocated: 2216.75MiB
>> Device missing: 0.00MiB
>> Used: 1100.69MiB
>> Free (estimated): 2683.44MiB (min:
>> 1575.06MiB)
>> Data ratio: 1.00
>> Metadata ratio: 2.00
>> Global reserve: 13.00MiB (used:
>> 0.12MiB)
>>
>> Data,single: Size:1559.25MiB, Used:1092.56MiB
>> /dev/mapper/rootvg-varvol 1559.25MiB
>>
>> Metadata,DUP: Size:640.00MiB, Used:4.00MiB
>> /dev/mapper/rootvg-varvol 1280.00MiB
>>
>> System,DUP: Size:32.00MiB, Used:0.06MiB
>> /dev/mapper/rootvg-varvol 64.00MiB
>>
>> Unallocated:
>> /dev/mapper/rootvg-varvol 2216.75MiB
>>
>> So if I understand properly it might be due to chunk being needed for
>> balance/discard or scrub.
>> Is there 3rd party tools which are also able to lock metadata block
>> group as
>> well? It seems to happens at the same time, when backup is running
>> (spectrum), and/or HANA backup and/or ReaR backup as well.
>
> If the 3rd party tools are triggering any of the btrfs maintenance
> functions then they'll lock block groups; otherwise, normal filesystem
> operations don't generally hold locks at the block group level.
>
> There might be some additional issues with the 4.12 kernel that cause
> it to allocate more than the minimum metadata. I recall there were
> some problems with old kernels where multiple threads allocating at
> the same time will all grab their own chunks, but I'm not sure which
> kernel those were fixed in. There are also changes to the allocator's
> behavior with the 'ssd' and 'nossd' mount options in 4.14 which might
> cause these effects.
>
> Some temporary overallocation is normal. It's likely it would have
> worked even without the overallocation, the overallocation has a pretty
> large impact on these tiny filesystems. This seems a bit higher than
> the amount I've come to expect from modern btrfs.
Thanks for all information!
If the chunk size (128MB, 256MB... 1GB) is in relation to the FS size,
is there a command to determine the chunk size for data and metadata?
Should I expect BTRFS to start allocating bigger chunk at some point
after filesystem extension?
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Filesystem Read Only due to errno=-28 during metadata allocation
2021-10-25 9:53 ` mailing
@ 2021-10-25 10:32 ` Nikolay Borisov
0 siblings, 0 replies; 7+ messages in thread
From: Nikolay Borisov @ 2021-10-25 10:32 UTC (permalink / raw)
To: mailing, Zygo Blaxell; +Cc: linux-btrfs
On 25.10.21 г. 12:53, mailing@dmilz.net wrote:
<snip>
> Thanks for all information!
>
> If the chunk size (128MB, 256MB... 1GB) is in relation to the FS size,
> is there a command to determine the chunk size for data and metadata?
You can determine it by dumping the on-disk data structure via btrfs
inspect-internal dump-tree -t2 /dev/foo this will contain the currently
allocated blockgroups and their respective size.
> Should I expect BTRFS to start allocating bigger chunk at some point
> after filesystem extension?
Yes, this is mandated by the code in init_alloc_chunk_ctl_policy_regular
function.
Basically data chunks are at most 1G in size, metadat chunks are either
1G or 256m depending on whether the fs is larger than 50gb. But a chunk
can never be more than 10% of the writable space on disk.
>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-10-25 10:32 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-10-13 12:35 Filesystem Read Only due to errno=-28 during metadata allocation mailing
2021-10-18 11:13 ` mailing
2021-10-18 14:09 ` Zygo Blaxell
2021-10-19 10:57 ` mailing
2021-10-19 13:26 ` Zygo Blaxell
2021-10-25 9:53 ` mailing
2021-10-25 10:32 ` Nikolay Borisov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).