EXT4： About the method of compute journal blocks

EXT4： About the method of compute journal blocks

[yu xing]

Hello All，

         Currently, I am seeing the the make_ext4fs code of
android，the source code is located at system/extras/ext4_utils

I have some question, as following, can somebody help me to explain
these questions? Thanks a lot!

1. The method of compute journal blocks, why  divide 64 :

          static u32 compute_journal_blocks()
{
    u32 journal_blocks = DIV_ROUND_UP(info.len, info.block_size) / 64;
   // is here, why divide 64 ???
    if (journal_blocks < 1024)
        journal_blocks = 1024;
    if (journal_blocks > 32768)
        journal_blocks = 32768;
    return journal_blocks;
}

2. The method of compute jbg_desc_reserve_blocks , why multiply by 1024:
static u32 compute_bg_desc_reserve_blocks()
{
    u32 blocks = DIV_ROUND_UP(info.len, info.block_size);
    u32 block_groups = DIV_ROUND_UP(blocks, info.blocks_per_group);
    u32 bg_desc_blocks = DIV_ROUND_UP(block_groups * sizeof(struct
ext2_group_desc),
            info.block_size);

    u32 bg_desc_reserve_blocks =
            DIV_ROUND_UP(block_groups * 1024 * sizeof(struct
ext2_group_desc),      // is here , why multiply by 1024 ???
                    info.block_size) - bg_desc_blocks;

    if (bg_desc_reserve_blocks > info.block_size / sizeof(u32))
        bg_desc_reserve_blocks = info.block_size / sizeof(u32);

    return bg_desc_reserve_blocks;
}

3. About compute_inodes function , why divide 4 ??
static u32 compute_inodes()
{
    return DIV_ROUND_UP(info.len, info.block_size) / 4;  // is here
why divide 4 ??
}


yu.xing

Re: EXT4： About the method of compute journal blocks

[Theodore Ts'o]

On Tue, Jan 17, 2017 at 11:50:13AM +0800, yu xing wrote:
> 
>          Currently, I am seeing the the make_ext4fs code of
> android，the source code is located at system/extras/ext4_utils

The make_ext4fs code in AOSP is something I've been trying to make go
away.  It exists because many years ago, the people who were working
on Android were desperately trying to avoid all GPLv2 code outside of
the kernel.  So make_ext4fs was a cleanroom rewrite from scratch of
mke2fs, and so they may have gotten a few things wrong.  As a result
file systems made with make_ext4fs are not completely identical to
those made by ext4 (especially when used on the device after a factory
reset) and this has caused some interesting and hard to debug problems
in the past.

E2fsprogs is now in AOSP, and I am hoping to make all of the use cases
of make_ext4fs go away, since it will make everyone's lives much
easier.  This has actually been a bit tricker since make_ext4fs has
some additional functionality that never should have been part of
make_ext4fs, but that is slowly being disentangled (see the very
latest AOSP git trees).   So tiny steps....

For this reason, most of the people on this list aren't going to be
able to answer questions about make_ext4fs, since it's Android
specific code.  I can answer some of these questions, but only because
I've been asked to debug some of these hard to understand bugs in
make_ext4fs (and I really don't want to do that any more, hence the
work to extirpate make_ext4fs from the AOSP source tree :-).

> I have some question, as following, can somebody help me to explain
> these questions? Thanks a lot!
> 
> 1. The method of compute journal blocks, why  divide 64 :
> 
>           static u32 compute_journal_blocks()
> {
>     u32 journal_blocks = DIV_ROUND_UP(info.len, info.block_size) / 64;
>    // is here, why divide 64 ???
>     if (journal_blocks < 1024)
>         journal_blocks = 1024;
>     if (journal_blocks > 32768)
>         journal_blocks = 32768;
>     return journal_blocks;
> }

This is to compute how many blocks should be allocated for the
journal.  It's a hueristic, and I suspect the authors of make_ext4fs
were trying to make the it work more or less like what mke2fs does.
See ext2fs_default_journal_size() in lib/ext2fs/mkjournal.c in
e2fsprogs for the original code.  It's a bit better documented, but in
the end, it's just a hueristic.  The hueristic has changed over the
years, by the way, for better performance on larger, faster devices.
So what make_ext4fs does is different from e2fsprogs's current defaults.

> 2. The method of compute jbg_desc_reserve_blocks , why multiply by 1024:
> static u32 compute_bg_desc_reserve_blocks()
> {
>     u32 blocks = DIV_ROUND_UP(info.len, info.block_size);
>     u32 block_groups = DIV_ROUND_UP(blocks, info.blocks_per_group);
>     u32 bg_desc_blocks = DIV_ROUND_UP(block_groups * sizeof(struct
> ext2_group_desc),
>             info.block_size);
> 
>     u32 bg_desc_reserve_blocks =
>             DIV_ROUND_UP(block_groups * 1024 * sizeof(struct
> ext2_group_desc),      // is here , why multiply by 1024 ???
>                     info.block_size) - bg_desc_blocks;
> 
>     if (bg_desc_reserve_blocks > info.block_size / sizeof(u32))
>         bg_desc_reserve_blocks = info.block_size / sizeof(u32);
> 
>     return bg_desc_reserve_blocks;
> }

These are reserved blocks used so the file system can be on-line
resized.  It's actually a waste of space since android file systems
are never resized, but again, cargo cult algorithms from clean room
reimplementations.  :-)

> 3. About compute_inodes function , why divide 4 ??
> static u32 compute_inodes()
> {
>     return DIV_ROUND_UP(info.len, info.block_size) / 4;  // is here
> why divide 4 ??
> }

Again, this is a hueristic to figure out roughly how many inodes
should be reserved given a particular file system size.  The default
assumes that the average size of files in the file system is 16k, and
allocates enough inodes so that the file system won't run out of
inodes given that average file size.

This is adjustable using mke2fs's command line arguments.  See the man
page for mke2fs for more details.  make_ext4fs was just using a hard
coded default that can't be adjusted, apparently.

Cheers,

					- Ted

Re: EXT4： About the method of compute journal blocks

[yu xing]

Thanks the detail answer from Theodore, I add some comments again , as
following inline answer.

yu.xing

2017-01-18 1:10 GMT+08:00 Theodore Ts'o <tytso@mit.edu>:
> On Tue, Jan 17, 2017 at 11:50:13AM +0800, yu xing wrote:
>>
>>          Currently, I am seeing the the make_ext4fs code of
>> android，the source code is located at system/extras/ext4_utils
>
> The make_ext4fs code in AOSP is something I've been trying to make go
> away.  It exists because many years ago, the people who were working
> on Android were desperately trying to avoid all GPLv2 code outside of
> the kernel.  So make_ext4fs was a cleanroom rewrite from scratch of
> mke2fs, and so they may have gotten a few things wrong.  As a result
> file systems made with make_ext4fs are not completely identical to
> those made by ext4 (especially when used on the device after a factory
> reset) and this has caused some interesting and hard to debug problems
> in the past.
>
> E2fsprogs is now in AOSP, and I am hoping to make all of the use cases
> of make_ext4fs go away, since it will make everyone's lives much
> easier.  This has actually been a bit tricker since make_ext4fs has
> some additional functionality that never should have been part of
> make_ext4fs, but that is slowly being disentangled (see the very
> latest AOSP git trees).   So tiny steps....
>
     [yu.xing]:
      Do you mean, In future , the mke2fs will instead of the make_ext4fs ?
      if true, maybe some functions from make_ext4fs will add to mke2fs.
      such as pack the system.img from out/target/product/xxxxx/system,
      and add selinux file_contexts into system , as the following command list

    make_ext4fs [ -l <len> ] [ -j <journal size> ] [ -b <block_size> ]
    [ -g <blocks per group> ] [ -i <inodes> ] [ -I <inode size> ]
    [ -L <label> ] [ -f ] [ -a <android mountpoint> ] [ -u ]
    [ -S file_contexts ] [ -C fs_config ] [ -T timestamp ]
    [ -z | -s ] [ -w ] [ -c ] [ -J ] [ -v ] [ -B <block_list_file> ]
    <filename> [[<directory>] <target_out_directory>]

     the [<target_out_directory>] arguments is the packing directory
     and the -S arguments is used to include the selinux file_contexts

     but in the mke2fs command, I can not find these functions( the
newest version mk2fs whether or not support these functions,
     if my wrong, please correct me), so maybe add these functions
into mke2fs ?  the mke2fs command list as following:
     Usage: mke2fs [-c|-l filename] [-b block-size] [-C cluster-size]
        [-i bytes-per-inode] [-I inode-size] [-J journal-options]
        [-G flex-group-size] [-N number-of-inodes]
        [-m reserved-blocks-percentage] [-o creator-os]
        [-g blocks-per-group] [-L volume-label] [-M last-mounted-directory]
        [-O feature[,...]] [-r fs-revision] [-E extended-option[,...]]
        [-t fs-type] [-T usage-type ] [-U UUID] [-jnqvDFKSV] device
[blocks-count]

> For this reason, most of the people on this list aren't going to be
> able to answer questions about make_ext4fs, since it's Android
> specific code.  I can answer some of these questions, but only because
> I've been asked to debug some of these hard to understand bugs in
> make_ext4fs (and I really don't want to do that any more, hence the
> work to extirpate make_ext4fs from the AOSP source tree :-).
>
>> I have some question, as following, can somebody help me to explain
>> these questions? Thanks a lot!
>>
>> 1. The method of compute journal blocks, why  divide 64 :
>>
>>           static u32 compute_journal_blocks()
>> {
>>     u32 journal_blocks = DIV_ROUND_UP(info.len, info.block_size) / 64;
>>    // is here, why divide 64 ???
>>     if (journal_blocks < 1024)
>>         journal_blocks = 1024;
>>     if (journal_blocks > 32768)
>>         journal_blocks = 32768;
>>     return journal_blocks;
>> }
>
> This is to compute how many blocks should be allocated for the
> journal.  It's a hueristic, and I suspect the authors of make_ext4fs
> were trying to make the it work more or less like what mke2fs does.
> See ext2fs_default_journal_size() in lib/ext2fs/mkjournal.c in
> e2fsprogs for the original code.  It's a bit better documented, but in
> the end, it's just a hueristic.  The hueristic has changed over the
> years, by the way, for better performance on larger, faster devices.
> So what make_ext4fs does is different from e2fsprogs's current defaults.
>
>> 2. The method of compute jbg_desc_reserve_blocks , why multiply by 1024:
>> static u32 compute_bg_desc_reserve_blocks()
>> {
>>     u32 blocks = DIV_ROUND_UP(info.len, info.block_size);
>>     u32 block_groups = DIV_ROUND_UP(blocks, info.blocks_per_group);
>>     u32 bg_desc_blocks = DIV_ROUND_UP(block_groups * sizeof(struct
>> ext2_group_desc),
>>             info.block_size);
>>
>>     u32 bg_desc_reserve_blocks =
>>             DIV_ROUND_UP(block_groups * 1024 * sizeof(struct
>> ext2_group_desc),      // is here , why multiply by 1024 ???
>>                     info.block_size) - bg_desc_blocks;
>>
>>     if (bg_desc_reserve_blocks > info.block_size / sizeof(u32))
>>         bg_desc_reserve_blocks = info.block_size / sizeof(u32);
>>
>>     return bg_desc_reserve_blocks;
>> }
>
> These are reserved blocks used so the file system can be on-line
> resized.  It's actually a waste of space since android file systems
> are never resized, but again, cargo cult algorithms from clean room
> reimplementations.  :-)
   [yu.xing]:
   For android file system, resize maybe need:
   for the system.img, it is not required the resize , but for the
userdata.img resizing was needed.
   For example, we build a 512M userdata.img , but when we mount it
to data partition,
   we need to resize it for get the rest of space of the emmc device ,
if not, the data partition size is 512M.
>
>> 3. About compute_inodes function , why divide 4 ??
>> static u32 compute_inodes()
>> {
>>     return DIV_ROUND_UP(info.len, info.block_size) / 4;  // is here
>> why divide 4 ??
>> }
>
> Again, this is a hueristic to figure out roughly how many inodes
> should be reserved given a particular file system size.  The default
> assumes that the average size of files in the file system is 16k, and
> allocates enough inodes so that the file system won't run out of
> inodes given that average file size.
>
> This is adjustable using mke2fs's command line arguments.  See the man
> page for mke2fs for more details.  make_ext4fs was just using a hard
> coded default that can't be adjusted, apparently.
>
> Cheers,
>
>                                         - Ted

Re: EXT4： About the method of compute journal blocks

[Theodore Ts'o]

On Wed, Jan 18, 2017 at 01:06:07PM +0800, yu xing wrote:
>       Do you mean, In future , the mke2fs will instead of the make_ext4fs ?
>       if true, maybe some functions from make_ext4fs will add to mke2fs.
>       such as pack the system.img from out/target/product/xxxxx/system,
>       and add selinux file_contexts into system , as the following command list
> 
>     make_ext4fs [ -l <len> ] [ -j <journal size> ] [ -b <block_size> ]
>     [ -g <blocks per group> ] [ -i <inodes> ] [ -I <inode size> ]
>     [ -L <label> ] [ -f ] [ -a <android mountpoint> ] [ -u ]
>     [ -S file_contexts ] [ -C fs_config ] [ -T timestamp ]
>     [ -z | -s ] [ -w ] [ -c ] [ -J ] [ -v ] [ -B <block_list_file> ]
>     <filename> [[<directory>] <target_out_directory>]
> 
>      the [<target_out_directory>] arguments is the packing directory
>      and the -S arguments is used to include the selinux file_contexts
> 
>      but in the mke2fs command, I can not find these functions( the
> newest version mk2fs whether or not support these functions,
>      if my wrong, please correct me), so maybe add these functions
> into mke2fs ?  the mke2fs command list as following:

Take a look at the AOSP master branch for e2fsprogs.  These changes
are not yet part of an official Android release (as they showed up
after the 'N' release) but you'll see that e2fsprogs has been updated
to 1.43 release and in the contrib/android directory, there is a new
e2fsdroid program.  That was some great work done by Adrien
Schildknecht as part of his Google intern project, and he consulted
with me on the design of how to refactor that functionality out of
make_ext4fs.

There are still some changes needed before we can completely get rid
of make_ext4fs (it is called programmatically from the volume manager,
and that last bit of work hasn't been converted yet).  And of course
there has still be some polishing and bug fixing that has been
happening to make sure things are smooth on Mac and Windows build
hosts, etc.  So as usual with the AOSP master branch, understand that
it is a work in progress, and it hasn't gotten the same level of
testing as an official android "letter" release.

Cheers,

						- Ted