Ext4 devel interlock meeting minutes (May 7, 2007)

Ext4 devel interlock meeting minutes (May 7, 2007)

[Avantika Mathur]

Ext4 Developer Interlock Call: 05/07/2007 Meeting Minutes

Attendees: Mingming Cao, Dave Kleikamp, Ted Ts'o, Aneesh Veetil, Takashi 
Sato, Jose Santos, Avantika Mathur,

Minutes can be accessed at: 
http://ext4.wiki.kernel.org/index.php/Ext4_Developer%27s_Conference_Call


Discussed pushing ext4 patches upstream.

- Preallocation Patches: These patches are almost ready to push, once we 
have consensus from all architecture maintainers on the format of the 
system call.  Mingming will help with this.

- Nanosecond Timestamps:  Ted will take a closer look at this patch 
before pushing

- It was proposed to push everything in the ext4 git tree up until the 
unstable patches.  Before pushing these patches, we must run tests on 
the patches and use the check-patches scripts to perform sanity checking.

e2fsprogs:

- Ted recently posted e2fsprogs-1.39-tyt3; which includes patches 
supporting extents.  This is a work in progress.

- Jose used the patches to e2fsprogs sent by Valerie Clement to create 
large filesystems (with >32-bit block numbers)

   -- Ted suggested that these patches be added to e2fsprogs-tyt3, with 
a flag set when in use, indicating that no shared libraries should be used.

- Ted is planning to discontinue use of Mercurial for e2fsprogs, and 
switch entirely to git.  By using git, we can have a stable and 
development branch for e2fsprogs.  This will also reduce the work Ted 
does to create a patch series.

Migration:

- Aneesh looked at the online defragmentation code, and feels it may not 
be usable for migration because it is solely extents based.  Mingming 
pointed out that the new defrag patches also have support for indirect 
mapped blocks, Aneesh will look into these new patches.

- Ted suggested doing the migration through two ioctls, the first 
migrating the inode online via ioctl, and the second using the 
online-defrag ioctl.

Metablock Groups:

- Mingming mentioned that in the metablock group feature, the inode 
metadata is not moved with the rest of the metadata.  Ted will check 
this, as he thought this had been implemented.

- Online resize does not use metablock groups, and is currently limited 
to 2TB.

- Also need to remove sanity checks in the mount code

- Once these issues are fixed, the metablock group feature can be turned 
on by default for ext4 filesysytems

- The extents option will be turned on by default for all ext4 
filesystems with the extents feature enabled.

Re: Ext4 devel interlock meeting minutes (May 7, 2007)

[Andreas Dilger]

On May 09, 2007  10:02 -0700, Avantika Mathur wrote:
> Metablock Groups:
> 
> - Mingming mentioned that in the metablock group feature, the inode 
> metadata is not moved with the rest of the metadata.  Ted will check 
> this, as he thought this had been implemented.
> 
> - Online resize does not use metablock groups, and is currently limited 
> to 2TB.
> 
> - Also need to remove sanity checks in the mount code
> 
> - Once these issues are fixed, the metablock group feature can be turned 
> on by default for ext4 filesysytems

In my recent investigation of META_BG, it appears that BOTH the kernel
and e2fsprogs do NOT handle group metadata (bitmaps, itable) outside the
group.

In the kernel ext[234]_check_descriptors() requires the bitmaps and itable
to be inside the group even though META_BG is supported for a long time.

In e2fsck/super.c check_super_block() also requires that the bitmaps and
itable be inside the group even though e2fsck has claimed META_BG for a
long time.

That means it is virtually impossible to allow the group metadata to move
under the META_BG feature.  It might be possible to predicate this change
under both 64BIT and META_BG, though this is somewhat non-standard.

Alternately, we could allow the relocation of bitmaps and itables under
BIG_BG, which is itself covers the relocation aspects and also allows
the ability to change the group size.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Re: Ext4 devel interlock meeting minutes (May 7, 2007)

[Mingming Cao]

On Wed, 2007-05-09 at 10:37 -0700, Andreas Dilger wrote:
> On May 09, 2007  10:02 -0700, Avantika Mathur wrote:
> > Metablock Groups:
> > 
> > - Mingming mentioned that in the metablock group feature, the inode 
> > metadata is not moved with the rest of the metadata.  Ted will check 
> > this, as he thought this had been implemented.
> > 
> > - Online resize does not use metablock groups, and is currently limited 
> > to 2TB.
> > 
> > - Also need to remove sanity checks in the mount code
> > 
> > - Once these issues are fixed, the metablock group feature can be turned 
> > on by default for ext4 filesysytems
> 
> In my recent investigation of META_BG, it appears that BOTH the kernel
> and e2fsprogs do NOT handle group metadata (bitmaps, itable) outside the
> group.
> 
> In the kernel ext[234]_check_descriptors() requires the bitmaps and itable
> to be inside the group even though META_BG is supported for a long time.
> 
> In e2fsck/super.c check_super_block() also requires that the bitmaps and
> itable be inside the group even though e2fsck has claimed META_BG for a
> long time.
> 
> That means it is virtually impossible to allow the group metadata to move
> under the META_BG feature.

Understand it's not straightforward to turn on this feature by default.
Just try to understand, why the kernel and e2fsprogs issues you
mentioned above are not possible to fix to support full META_BG
feature? 

>   It might be possible to predicate this change
> under both 64BIT and META_BG, though this is somewhat non-standard.
> 
> Alternately, we could allow the relocation of bitmaps and itables under
> BIG_BG, which is itself covers the relocation aspects and also allows
> the ability to change the group size.
> 

Well, I think the previous discussion about BIG_BG feature leads us to
believe that turn on META_BG feature is good enough to support larger fs
and reduce fragmentation(which is the new BIG_BG feature is trying to
address).  After all the relocation of metadata (bitmaps, itables and
group descriptors) is what META_BG feature means.


Regards,
Mingming

Re: Ext4 devel interlock meeting minutes (May 7, 2007)

[Andreas Dilger]

On May 09, 2007  11:10 -0700, Mingming Cao wrote:
> On Wed, 2007-05-09 at 10:37 -0700, Andreas Dilger wrote:
> > That means it is virtually impossible to allow the group metadata to move
> > under the META_BG feature.
> 
> Understand it's not straightforward to turn on this feature by default.
> Just try to understand, why the kernel and e2fsprogs issues you
> mentioned above are not possible to fix to support full META_BG
> feature? 

Because older kernels and e2fsprogs will think they understand META_BG,
and will consider the filesystem corrupt due to "badly placed" bitmaps
and itable.  That is exactly what FEATURE flags are supposed to protect
against, so we can't change the semantics of the META_BG feature at this
time.  I understand that we WANTED to have META_BG have different
semantics, but the implementation didn't follow the design for this flag.

> >   It might be possible to predicate this change
> > under both 64BIT and META_BG, though this is somewhat non-standard.
> > 
> > Alternately, we could allow the relocation of bitmaps and itables under
> > BIG_BG, which is itself covers the relocation aspects and also allows
> > the ability to change the group size.
> > 
> 
> Well, I think the previous discussion about BIG_BG feature leads us to
> believe that turn on META_BG feature is good enough to support larger fs
> and reduce fragmentation(which is the new BIG_BG feature is trying to
> address).  After all the relocation of metadata (bitmaps, itables and
> group descriptors) is what META_BG feature means.

That is what we WANTED the META_BG feature to mean.  What was implemented
in both ext2/3/4 and e2fsck is that META_BG means "group descriptor backups
are in 1st, 2nd, last group in metagroup" and nothing else.  At least the
code is consistent between the two.

So that leaves us in the position that we either need to create a new
INCOMPAT feature flag to allow relocation of bitmaps/itable, OR we
need to make this conditional upon both META_BG and 64BIT being set.
The latter is a bit confusing and is subject to error if only one or
the other flag is checked by accident.  If we make a new INCOMPAT flag
then could almost as easily implement BIG_BG and give us more flexibility
for the future, though I haven't looked at that code in a while.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.