Re: ext4: first write to large ext3 filesystem takes 96 seconds

[Andreas Dilger <adilger@dilger.ca>]

The main problem here is that reading all of the block bitmaps takes
a huge amount of time for a large filesystem.

7.8TB / 128MB/group ~= 8000 groups
8000 bitmaps / 100 seeks/sec = 80s

So that is what is making things slow. Once the allocator has all the
blocks in memory there are no problems. There are some heuristics
to skip bitmaps that are totally full, but they don't work in your case. 

This is why the flex_bg feature was created - to allow the bitmaps
to be read from disk without seeks.  This also speeds up e2fsck by
the same 96s that would otherwise be wasted waiting for the disk.

Backporting flex_bg to ext3 would be fairly trivial - just disable the checks
for the location of the bitmaps at mount time. However, using it
requires that you reformat your filesystem with "-O flex_bg" to
get the improved layout. 

The other option (if your runtime environment allows it) is to prefetch
the block bitmaps using "dumpe2fs /dev/XXX > /dev/null" before the
filesystem is in use. This still takes 90s, but can be started early in
the boot process on each disk in parallel.

Cheers, Andreas

> On Jul 7, 2014, at 18:16, Theodore Ts'o <tytso@mit.edu> wrote:
> 
> On Mon, Jul 07, 2014 at 05:13:49PM -0400, Benjamin LaHaise wrote:s
>> Hi folks,
>> 
>> I've just ran into a bug with the ext4 codebase in 3.4.91 that doesn't seem 
>> to exist in ext3, and was wondering if anyone has encountered this before.  
>> I have a 7.4TB ext3 filesystem that has been filled with 1.8TB of data.  
>> When this filesystem is freshly mounted, the first write to the filesystem 
>> takes a whopping 96 seconds to complete, during which time the system is 
>> reading about 1000 blocks per second.  Subsequent writes are much quicker.  
>> The problem seems to be that ext4 is loading all of the bitmaps on the 
>> filesystem before the first write proceeds.  The backtrace looks roughly as 
>> follows:
> 
> So the issue is that ext3 will just allocate the first free block it
> can find, even if it is a single free block in block group #1001,
> followed by a single free block in block group #2002.  Ext4 tries a
> harder to find contiguous blocks.
> 
> If you are using an ext3 file system format, the block allocation
> bitmaps are scattered across the entire file system, so we end up
> doing a lot random 4k seeks.
> 
> We can try to be a bit smarter about how we try to search the file
> system for free blocks.
> 
> Out of curiosity, can you send me a copy of the contents of:
> 
> /proc/fs/ext4/dm-XX/mb_groups
> 
> Thanks!!
> 
>                    - Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html