[e2fsprogs] resizing to minimum was failed since 45a78b

[e2fsprogs] resizing to minimum was failed since 45a78b

[Chanho Park]

Hi,

When I tried to resize a loop ext4 image to minimum, I couldn't get correct
result.
Please see below test and results:

1. e2fsprogs 1.43-WIP version:

mke2fs -F -t ext4 foo.img 512M
mount foo.img mnt1
dd if=/dev/zero of=mnt1/test bs=1M count=350
df | grep mnt1
/dev/loop0        499656  358796    104164  78% /mnt1

umount mnt1
e2fsck -f foo.img

resize2fs -M foo.img
resize2fs 1.43-WIP (29-Mar-2015)
Resizing the filesystem on foo.img to 129175 (4k) blocks.
The filesystem on foo.img is now 129175 (4k) blocks long.

ls -al foo.img
-rw-r--r-- 1 root root 529100800 Apr 29 08:00 foo.img

2. e2fsprogs 1.42.9 version:
mke2fs -F -t ext4 bar.img 512M
mount bar.img mnt2
dd if=/dev/zero of=mnt2/test bs=1M count=350
umount mnt2
e2fsck -f bar.img

resize2fs -M bar.img
resize2fs 1.42.9 (28-Dec-2013)
Resizing the filesystem on bar.img to 103592 (4k) blocks.
The filesystem on bar.img is now 103592 blocks long.

ls -al bar.img
-rw-r--r-- 1 root root 424312832 Apr 29 08:04 bar.img

1.43-WIP version: 529100800 -> 529100800
1.42.9 version: 529100800 -> 424312832

Furthermore, the resizing to minimum was failed if the loop image was filled
over 84%.
I found it was changed since (45a78b8 resize2fs: refine minimum required
blocks for flex_bg file systems)
Is it correct or something's wrong?

Best Regards,
Chanho Park

Re: [e2fsprogs] resizing to minimum was failed since 45a78b

[Theodore Ts'o]

On Wed, Apr 29, 2015 at 05:33:20PM +0900, Chanho Park wrote:
> 
> When I tried to resize a loop ext4 image to minimum, I couldn't get correct
> result.

It all depends on your definition of "correct".  I've regretted
accepting the the patch to implement -M, since in practice almost
everyone who has been using really shouldn't have been.  It's useful
for developers who want to stress test resize2fs to make sure it does
the right thing in corner cases.

However, it's a *terrible* idea for anyone to use it for __anything__
else.  The result of using resize2fs -M is a file system where the
files are highly fragmented.  The first real world use of resizde2fs
-M I was aware of, which was some clever Red Hat release engineer
which created a huge file system, installed Red Hat on it, compressed
it down to a minimum size using resize2fs -M, and then burned the
result on a CD-ROM for use as a bootable Live CD system image.  The
resulting fragmentation of files as a result of resize2fs -M meant
that reading certain files from the CD-ROM, which is not known for its
high seek speeds, was, shall we say, less than optimal.

Because I don't think there are mant (any?) real sane/valid production
uses of resize2fs -M, my definition of "correct" for resize -M is that
it doesn't (a) corrupt file systems, or (b) lead to a state when
resize2fs -M might not be able to make forward progress, leaving to a
file-system which is half-resized, corrupt, and requiring expert
surgery from a file system expert to recover.  The reason why I care
about this is because it's not just used by misguided release
engineers trying to create file system images from source builds,
where if there is a corrupted file system, it's not a disaster.

Unfortunately, it's also used by clueless users who usually have _not_
made a backup before trying to use resize2fs -M to shrink their file
system before trying to rearrange their LVM volumes.  And users get
cranky when they lose data.

So the bottom line is I care a lot more about data loss than I do
about shrinking file systems to the absolute minimum size, and I
personaly don't think it's worth a huge amount of effort to try to
make the calculation of the minimum size where resize2fs is guaranteed
to succeed any closer to being exact.

That being said, if you are creating a read-only file system for a
flash image (where seeks are largely free), and you really insist on
using resize2fs -M, it may be that your best bet is to turn off the
flex_bg option, since most of the advantages of flex_bg aren't there
on a read-only flash image (such as might be used for a
system/firmware image).

Alternatively, if you want to invest effort in trying to make
resize2fs -M mimum file system size more accurate, feel free to send
me patches --- plus your proof in terms of code analysis and an
exhaustive test regime to make sure that it works for a wide variety
of file system sizes and free space availability pre-resize2fs.
Personally, I don't think it's a particularly worthwhile use of an
engineer's time, but if you really have a strong business need for
this, feel free work on it and send me the results of your efforts.

Cheers,

					- Ted

RE: [e2fsprogs] resizing to minimum was failed since 45a78b

[Chanho Park]

Hi,

> -----Original Message-----
> From: linux-ext4-owner@vger.kernel.org [mailto:linux-ext4-
> owner@vger.kernel.org] On Behalf Of Theodore Ts'o
> Sent: Wednesday, April 29, 2015 11:29 PM
> To: Chanho Park
> Cc: darrick.wong@oracle.com; linux-ext4@vger.kernel.org
> Subject: Re: [e2fsprogs] resizing to minimum was failed since 45a78b
> 
> On Wed, Apr 29, 2015 at 05:33:20PM +0900, Chanho Park wrote:
> >
> > When I tried to resize a loop ext4 image to minimum, I couldn't get
> correct
> > result.
> 
> It all depends on your definition of "correct".  I've regretted
> accepting the the patch to implement -M, since in practice almost
> everyone who has been using really shouldn't have been.  It's useful
> for developers who want to stress test resize2fs to make sure it does
> the right thing in corner cases.
> 
> However, it's a *terrible* idea for anyone to use it for __anything__
> else.  The result of using resize2fs -M is a file system where the
> files are highly fragmented.  The first real world use of resizde2fs
> -M I was aware of, which was some clever Red Hat release engineer
> which created a huge file system, installed Red Hat on it, compressed
> it down to a minimum size using resize2fs -M, and then burned the
> result on a CD-ROM for use as a bootable Live CD system image.  The
> resulting fragmentation of files as a result of resize2fs -M meant
> that reading certain files from the CD-ROM, which is not known for its
> high seek speeds, was, shall we say, less than optimal.
> 
> Because I don't think there are mant (any?) real sane/valid production
> uses of resize2fs -M, my definition of "correct" for resize -M is that
> it doesn't (a) corrupt file systems, or (b) lead to a state when
> resize2fs -M might not be able to make forward progress, leaving to a
> file-system which is half-resized, corrupt, and requiring expert
> surgery from a file system expert to recover.  The reason why I care
> about this is because it's not just used by misguided release
> engineers trying to create file system images from source builds,
> where if there is a corrupted file system, it's not a disaster.
> 
> Unfortunately, it's also used by clueless users who usually have _not_
> made a backup before trying to use resize2fs -M to shrink their file
> system before trying to rearrange their LVM volumes.  And users get
> cranky when they lose data.

In my situation, I usually use the "resize2fs -M" when making a tizen
platform image. As you know, like android, tizen also uses ext4 filesystem
and flash the image from bootloader. Instead of flashing full image size, we
only flash shrinked filesystem image to reduce flashing time and want to
avoid download huge files. If the partition size is 3GB, we'll make a 3GB
loop image and fill files into it. And I shrink the loop image to the
minimum size and flash it to my phone. When first booting, the system will
do resizing the partition to 3GB original size.

> 
> So the bottom line is I care a lot more about data loss than I do
> about shrinking file systems to the absolute minimum size, and I
> personaly don't think it's worth a huge amount of effort to try to
> make the calculation of the minimum size where resize2fs is guaranteed
> to succeed any closer to being exact.
> 
> That being said, if you are creating a read-only file system for a
> flash image (where seeks are largely free), and you really insist on
> using resize2fs -M, it may be that your best bet is to turn off the
> flex_bg option, since most of the advantages of flex_bg aren't there
> on a read-only flash image (such as might be used for a
> system/firmware image).

I could get below result when I turned off flex_bg option.

ls -al foo.img
-rw-r--r-- 1 root root 390680576 Apr 30 10:02 foo.img

Like android, a root filesystem of tizen is also read-only, but it's only to
protect the filesystem from sudden power off. Sometimes it could be
writeable when installing some system packages and required some system
actions.

> 
> Alternatively, if you want to invest effort in trying to make
> resize2fs -M mimum file system size more accurate, feel free to send
> me patches --- plus your proof in terms of code analysis and an
> exhaustive test regime to make sure that it works for a wide variety
> of file system sizes and free space availability pre-resize2fs.
> Personally, I don't think it's a particularly worthwhile use of an
> engineer's time, but if you really have a strong business need for
> this, feel free work on it and send me the results of your efforts.

Okay. I'll investigate it and make a patch if possible :)

Best Regards,
Chanho Park

Re: [e2fsprogs] resizing to minimum was failed since 45a78b

[Theodore Ts'o]

On Thu, Apr 30, 2015 at 07:12:04PM +0900, Chanho Park wrote:
> 
> In my situation, I usually use the "resize2fs -M" when making a tizen
> platform image. As you know, like android, tizen also uses ext4 filesystem
> and flash the image from bootloader. Instead of flashing full image size, we
> only flash shrinked filesystem image to reduce flashing time and want to
> avoid download huge files. If the partition size is 3GB, we'll make a 3GB
> loop image and fill files into it. And I shrink the loop image to the
> minimum size and flash it to my phone. When first booting, the system will
> do resizing the partition to 3GB original size.

What I would recommend is to estimate how big the file system needs to
doing a "du -s" over the directory that you want to preload onto the
file system size, add say, 5% or so the file system metadata, and then
create the file system that way.  If the file system is close enough
to the right size, then resize2fs -M is much more likely to shrink the
file system to its minimal size, and also reduce the file
fragmentation that results from it.  (One of the better uses for using
resize2fs to slightly shrink a file system was to make room for the
LVM table at the end of the device, for example.  It's when you try to
take, say, a 1TB file system and shrink to a few megabytes that
resize2fs doesn't do a great job, because it really wasn't engineered
for that use case, and it's hard enough to make sure it does so in a
efficient and robust manner as it is.)

You can then take that file system and shrink to a size far larger
than what it was originally created as --- that works much better than
taking a large empty file system, shrinking it down to its minimal
size, and then expanding it to something bigger.  If necessary, you
can use the "mke2fs -E resize=15T" option to make sure enough metadata
blocks have been reserved so the file system can be expanded as much
as you would like.  By default we reserve enough metadata blocks for
the file system to grow by a factor of a thousand, so if you create a
super-tiny file system and then expect to grow it to a multi-TB size,
that mke2fs option will be useful.  It's not strictly necessary for
ext4 (although it was required for ext3), but it will result in a
slightly more efficient file system that will mount more quickly under
ext4.

So if your Tizen image is, say, 390 megabytes, I'd suggest creating a
400 megabyte file system, and load that up with the image.  If you
then use resize2fs -M, it should shirnk it down to something much
closer to 390M.

I suspect you will find that the resulting image will be far more
efficient than trying to start with a 3GB file system, shrink it down
to 400M, and then expand it backup to 3GB.

Best regards,

					- Ted

Re: [e2fsprogs] resizing to minimum was failed since 45a78b

[Darrick J. Wong]

On Thu, Apr 30, 2015 at 11:13:16AM -0400, Theodore Ts'o wrote:
> On Thu, Apr 30, 2015 at 07:12:04PM +0900, Chanho Park wrote:
> > 
> > In my situation, I usually use the "resize2fs -M" when making a tizen
> > platform image. As you know, like android, tizen also uses ext4 filesystem
> > and flash the image from bootloader. Instead of flashing full image size, we
> > only flash shrinked filesystem image to reduce flashing time and want to
> > avoid download huge files. If the partition size is 3GB, we'll make a 3GB
> > loop image and fill files into it. And I shrink the loop image to the
> > minimum size and flash it to my phone. When first booting, the system will
> > do resizing the partition to 3GB original size.
> 
> What I would recommend is to estimate how big the file system needs to
> doing a "du -s" over the directory that you want to preload onto the
> file system size, add say, 5% or so the file system metadata, and then
> create the file system that way.  If the file system is close enough
> to the right size, then resize2fs -M is much more likely to shrink the
> file system to its minimal size, and also reduce the file

/me wonders if the script I threw into the last patchbomb would help?
http://patchwork.ozlabs.org/patch/457482/

If you intend to write to the FS after it's been expanded into the target
device, you should turn journalling on /after/ the expand operation.

--D

> fragmentation that results from it.  (One of the better uses for using
> resize2fs to slightly shrink a file system was to make room for the
> LVM table at the end of the device, for example.  It's when you try to
> take, say, a 1TB file system and shrink to a few megabytes that
> resize2fs doesn't do a great job, because it really wasn't engineered
> for that use case, and it's hard enough to make sure it does so in a
> efficient and robust manner as it is.)
> 
> You can then take that file system and shrink to a size far larger
> than what it was originally created as --- that works much better than
> taking a large empty file system, shrinking it down to its minimal
> size, and then expanding it to something bigger.  If necessary, you
> can use the "mke2fs -E resize=15T" option to make sure enough metadata
> blocks have been reserved so the file system can be expanded as much
> as you would like.  By default we reserve enough metadata blocks for
> the file system to grow by a factor of a thousand, so if you create a
> super-tiny file system and then expect to grow it to a multi-TB size,
> that mke2fs option will be useful.  It's not strictly necessary for
> ext4 (although it was required for ext3), but it will result in a
> slightly more efficient file system that will mount more quickly under
> ext4.
> 
> So if your Tizen image is, say, 390 megabytes, I'd suggest creating a
> 400 megabyte file system, and load that up with the image.  If you
> then use resize2fs -M, it should shirnk it down to something much
> closer to 390M.
> 
> I suspect you will find that the resulting image will be far more
> efficient than trying to start with a 3GB file system, shrink it down
> to 400M, and then expand it backup to 3GB.
> 
> Best regards,
> 
> 					- Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html