From: Mark Hills <mark@xwax.org>
To: linux-ext4@vger.kernel.org
Subject: Maildir quickly hitting max htree
Date: Fri, 12 Nov 2021 19:52:37 +0000 (GMT) [thread overview]
Message-ID: <2111121900560.16086@stax.localdomain> (raw)
Surprised to hit a limit when handling a modest Maildir case; does this
reflect a bug?
rsync'ing to a new mail server, after fewer than 100,000 files there are
intermittent failures:
rsync: [receiver] open "/home/mark/Maildir/.robot/cur/1633731549.M990187P7732.yello.[redacted],S=69473,W=70413:2," failed: No space left on device (28)
rsync: [receiver] rename "/home/mark/Maildir/.robot/cur/.1624626598.M748388P84607.yello.[redacted],S=17049,W=17352:2,.oBphKA" -> ".robot/cur/1624626598.M748388P84607.yello.[redacted],S=17049,W=17352:2,": No space left on device (28)
The kernel:
EXT4-fs warning (device dm-4): ext4_dx_add_entry:2351: Directory (ino: 225811) index full, reach max htree level :2
EXT4-fs warning (device dm-4): ext4_dx_add_entry:2355: Large directory feature is not enabled on this filesystem
Reaching for 'large_dir' seems premature as this feature is reported as
useful for 10M+ files, but this is much lower.
A 'bad' filename will fail consistently. Assuming the 10M+ absolute limit,
is the tree grossly imbalanced?
Intuitively, 'htree level :2' does not sound particular deep.
The source folder is 195,000 files -- large, but not crazy. rsync
eventually hit a ceiling having written 177,482 of the files. I can still
create new ones on the command line with non-Maildir names.
Ruled out quotas, by disabling them with "tune2fs -O ^quota" and
remounting.
See below for additional info.
--
Mark
$ uname -a
Linux floyd 5.10.78-0-virt #1-Alpine SMP Thu, 11 Nov 2021 14:31:09 +0000 x86_64 GNU/Linux
$ mke2fs -q -t ext4 /dev/vg0/home
$ rsync -va --exclude 'dovecot*' yello:Maildir/. $HOME/Maildir
$ ls | head -15
1605139205.M487508P91922.yello.[redacted],S=7625,W=7775:2,
1605139440.M413280P92363.yello.[redacted],S=7632,W=7782:2,
1605139466.M699663P92402.yello.[redacted],S=7560,W=7710:2,
1605139479.M651510P92421.yello.[redacted],S=7474,W=7623:2,
1605139508.M934351P92514.yello.[redacted],S=7626,W=7776:2,
1605139596.M459228P92713.yello.[redacted],S=7559,W=7709:2,
1605139645.M57446P92736.yello.[redacted],S=7632,W=7782:2,
1605139670.M964535P92758.yello.[redacted],S=7628,W=7778:2,
1605139697.M273694P92807.yello.[redacted],S=7632,W=7782:2,
1605139748.M607989P92853.yello.[redacted],S=7560,W=7710:2,
1605139759.M655635P92868.yello.[redacted],S=5912,W=6018:2,
1605139808.M338286P93071.yello.[redacted],S=7628,W=7778:2,
1605139961.M915501P93235.yello.[redacted],S=7625,W=7775:2,
1605140303.M219848P93591.yello.[redacted],S=6898,W=7023:2,
1605140580.M166212P93921.yello.[redacted],S=6896,W=7021:2,
$ touch abc
[success]
$ touch 1624626598.M748388P84607.yello.[redacted],S=17049,W=17352:2,
touch: cannot touch '1624626598.M748388P84607.yello.[redacted],S=17049,W=17352:2,': No space left on device
$ dumpe2fs /dev/vg0/home
Filesystem volume name: <none>
Last mounted on: /home
Filesystem UUID: ad26c968-d057-4d44-bef9-1e2df347580e
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 5225472
Block count: 21229568
Reserved block count: 851459
Overhead clusters: 22361
Free blocks: 8058180
Free inodes: 4799979
First block: 1
Block size: 1024
Fragment size: 1024
Group descriptor size: 64
Reserved GDT blocks: 96
Blocks per group: 8192
Fragments per group: 8192
Inodes per group: 2016
Inode blocks per group: 504
Flex block group size: 16
Filesystem created: Mon Nov 8 13:14:56 2021
Last mount time: Fri Nov 12 18:43:14 2021
Last write time: Fri Nov 12 18:43:14 2021
Mount count: 27
Maximum mount count: -1
Last checked: Mon Nov 8 13:14:56 2021
Check interval: 0 (<none>)
Lifetime writes: 14 GB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 32
Desired extra isize: 32
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: 839d2871-b97e-456d-9724-096db15931b8
Journal backup: inode blocks
Checksum type: crc32c
Checksum: 0x5974a8b1
Journal features: journal_incompat_revoke journal_64bit
journal_checksum_v3
Total journal size: 4096k
Total journal blocks: 4096
Max transaction length: 4096
Fast commit length: 0
Journal sequence: 0x00000a2a
Journal start: 702
Journal checksum type: crc32c
Journal checksum: 0x4d693e79
next reply other threads:[~2021-11-12 20:37 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-11-12 19:52 Mark Hills [this message]
[not found] ` <36FABD31-B636-4D94-B14D-93F3D2B4C148@dilger.ca>
2021-11-13 12:05 ` Maildir quickly hitting max htree Mark Hills
2021-11-13 17:19 ` Andreas Dilger
2021-11-16 17:52 ` Mark Hills
2021-11-14 17:44 ` Theodore Ts'o
2021-11-16 19:31 ` Mark Hills
2021-11-17 5:20 ` Theodore Ts'o
2021-11-17 13:13 ` Mark Hills
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2111121900560.16086@stax.localdomain \
--to=mark@xwax.org \
--cc=linux-ext4@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox