Make huge files strictly contiguous (fallocate, bigalloc, e4defrag...)

Make huge files strictly contiguous (fallocate, bigalloc, e4defrag...)

[Contact]

Hi everybody,

I'm trying to create a multiboot medium using Easy2Boot on a 64 GB USB
key. Easy2Boot [1] is a "software" (well a collection of files really)
that relies on grub4dos [2] to work and thus requires payload files to
be strictly contiguous to do its trick. It's supposed to be set up once
and then one just has to add files to boot from to a folder to modify
the multiboot setup.

The ISO files I have are between 230 MB and 2 GB each. I divided my USB
drive into two partitions : first one, 16 GB ntfs formatted so that
grub4dos actually works (for some reasons it doesn't with ext4, plus I
want to be able to use it with other OS); second one 48 GB formatted as
ext4 with e2fsprogs version 1.42.9 from Archlinux. However when I try
to boot from those files I systematically get the "file not contiguous"
error in Easy2Boot.

Question is: is it possible to make huge files strictly contiguous on
ex4, and if yes how?


*******
Details
*******

I tried different mkfs.ext4 options so as to cater to my use case, but
to no avail. I tried pushing the inode_ratio to its maximum value (64
MB) and using fallocate besides regular copy:
# mkfs.ext4 -m 0 -L iso -i 67108864 -E root_owner=1000:100 /dev/sdc2
# fallocate -l 1235451904
/path/to/mounted/usb/folder/debian-live-7.2-amd64-gnome-desktop_fal.iso
# cp debian-live-7.2-amd64-gnome-desktop.iso
/path/to/folder/debian-live-7.2-amd64-gnome-desktop_fal.iso

But files aren't contiguous. e4defrag seems to confirm it:
# e4defrag -c '/run/media/neitsab/iso/_ISO'
<Fragmented files>                             now/best       size/ext
1. /run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso
                                                13/1         119378 KB
2. /run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64_fal.iso
                                                13/1         119378 KB

Total/best extents                26/2
Average size per extent            119378 KB
Fragmentation score                0
[0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag]
This directory (/run/media/neitsab/iso/_ISO) does not need defragmentation.
Done.

Then I tried to use bigalloc and clusters:
# mkfs.ext4 -m 0 -L iso -i 67108864 -O bigalloc -C 256M -E
root_owner=1000:100 /dev/sdc2
mke2fs 1.42.9 (28-Dec-2013)

Warning: the bigalloc feature is still under development
See https://ext4.wiki.kernel.org/index.php/Bigalloc for more information

warning: 53248 blocks unused.

Filesystem label=iso
OS type: Linux
Block size=4096 (log=2)
Cluster size=268435456 (log=18)
Stride=0 blocks, Stripe width=0 blocks
720 inodes, 11665408 blocks
0 blocks (0.00%) reserved for the super user
First data block=0
Root directory owner=1000:100
1 block group
2146959360 blocks per group, 32760 clusters per group
720 inodes per group

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

It took it some lengthy time to allocate the group tables, then regular
copy worked but fallocate hanged (process uninterruptible) and I had to
reboot the machine, which led me to a kernel panic mentionning
fallocate. The file copied via Nautilus still wasn't contiguous enough
for Easy2Boot.

So what's left? Is it at all possible to make completely contiguous
files on ext4? Not even on a fresh FS with plenty of free space?

Thanks for your comments!


[1] http://www.easy2boot.com/introduction/
[2] https://code.google.com/p/grub4dos-chenall/

Re: Make huge files strictly contiguous (fallocate, bigalloc, e4defrag...)

[Lukáš Czerner]

On Thu, 17 Apr 2014, Contact wrote:

> Date: Thu, 17 Apr 2014 04:18:30 +0200
> From: Contact <neitsab@ovh.fr>
> To: "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>
> Subject: Make huge files strictly contiguous (fallocate, bigalloc,
>     e4defrag...)
> 
> Hi everybody,
> 
> I'm trying to create a multiboot medium using Easy2Boot on a 64 GB USB
> key. Easy2Boot [1] is a "software" (well a collection of files really)
> that relies on grub4dos [2] to work and thus requires payload files to
> be strictly contiguous to do its trick. It's supposed to be set up once
> and then one just has to add files to boot from to a folder to modify
> the multiboot setup.
> 
> The ISO files I have are between 230 MB and 2 GB each. I divided my USB
> drive into two partitions : first one, 16 GB ntfs formatted so that
> grub4dos actually works (for some reasons it doesn't with ext4, plus I
> want to be able to use it with other OS); second one 48 GB formatted as
> ext4 with e2fsprogs version 1.42.9 from Archlinux. However when I try
> to boot from those files I systematically get the "file not contiguous"
> error in Easy2Boot.
> 
> Question is: is it possible to make huge files strictly contiguous on
> ex4, and if yes how?
> 
> 
> *******
> Details
> *******
> 
> I tried different mkfs.ext4 options so as to cater to my use case, but
> to no avail. I tried pushing the inode_ratio to its maximum value (64
> MB) and using fallocate besides regular copy:
> # mkfs.ext4 -m 0 -L iso -i 67108864 -E root_owner=1000:100 /dev/sdc2
> # fallocate -l 1235451904
> /path/to/mounted/usb/folder/debian-live-7.2-amd64-gnome-desktop_fal.iso
> # cp debian-live-7.2-amd64-gnome-desktop.iso
> /path/to/folder/debian-live-7.2-amd64-gnome-desktop_fal.iso

This is not how it it supposed to be used. Yes fallocate
preallocates the file, but cp will truncate it so fallocate will
certainly not help you in any way. In order for fallocate to be
useful you'll have to write into the file without actually
truncating it (dd can do this if you do not want to write your own
program)

Also the file is probably as contiguous as it could be. Here is what
I get on the file system with default mkfs options.

# e4defrag -c /mnt/test/file1
<File>                                         now/best      size/ext
/mnt/test/file1                                 10/1         120649 KB

But that does not tell the whole story. See

xfs_io -f -c "fiemap -v" /mnt/test/file1
/mnt/test/file1:
 EXT: FILE-OFFSET        BLOCK-RANGE       TOTAL FLAGS
   0: [0..262143]:       2768896..3031039 262144   0x0
   1: [262144..524287]:  3031040..3293183 262144   0x0
   2: [524288..786431]:  3293184..3555327 262144   0x0
   3: [786432..1048575]: 3555328..3817471 262144   0x0
   4: [1048576..1310719]: 3817472..4079615 262144   0x0
   5: [1310720..1425407]: 4079616..4194303 114688   0x0
   6: [1425408..1687551]: 4456448..4718591 262144   0x0
   7: [1687552..1949695]: 4718592..4980735 262144   0x0
   8: [1949696..2211839]: 4980736..5242879 262144   0x0
   9: [2211840..2412991]: 5242880..5444031 201152   0x1

(Note that the output is in 512B blocks)

As you can see the file is mostly contiguous, but it is divided into
several extents because of two reasons.

1. The extent in ext4 has a limited size of 32768 blocks for
initialized extent and 32767 block for unwritten extent. So when we
exceed that size we need another extent which might be physically
contiguous on disk with the previous one.

2. Ext4 divides disk space into allocation groups of certain size
(cluster size * 8)blocks. Now with flex_bg medatada such as inode
tables, block bitmaps and so one are packed closely together so the
do not have to be stored with each block group and you'll get more
contiguous data space.

However we're still storing backup superblock and Groups descriptors
in certain groups and those are the gaps you're seeing in the fiemap
list.

For detailed overview you can use dumpe2fs to see what is allocated
where on the file system.

Bigalloc file system should help since you'll get much larger
contiguous data spaces since the cluster size would be much bigger
hence you'll get much bigger block groups. But you'll still get
multiple extents (I do not really know how grub4dos recognizes
contiguous files) even though there will be mostly contiguous.

But then again you'll have Group descriptors and backup superblocks
in some groups so potentially some files might end up not
contiguous. And even though you can turn off backup superblock you
can not turn off writing group descriptors.

So no, at the moment there is not way to make really *really* sure
that the file you're creating even with fallocate will allways be
strictly contiguous. It'll mostly work, but it really depends on
where the file will be put on the file system and how big is the
file.

Also I've been talking about the case where nothing else is using
the file system, when that's not the case. It might happen that the
allocation will interfere (even though we're trying to allocate as
contiguous file as we can). And also the file system free space will
become more fragmented over time as it is used, but it really
depends on the workload.


Now for the problem with bigalloc, I am not sure what kernel version
are you using, but it's probably old. fallocate on bigalloc should
work just fine.

Hope it helps.
Thanks!
-Lukas



> 
> But files aren't contiguous. e4defrag seems to confirm it:
> # e4defrag -c '/run/media/neitsab/iso/_ISO'
> <Fragmented files>                             now/best       size/ext
> 1. /run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso
>                                                 13/1         119378 KB
> 2. /run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64_fal.iso
>                                                 13/1         119378 KB
> 
> Total/best extents                26/2
> Average size per extent            119378 KB
> Fragmentation score                0
> [0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag]
> This directory (/run/media/neitsab/iso/_ISO) does not need defragmentation.
> Done.
> 
> Then I tried to use bigalloc and clusters:
> # mkfs.ext4 -m 0 -L iso -i 67108864 -O bigalloc -C 256M -E
> root_owner=1000:100 /dev/sdc2
> mke2fs 1.42.9 (28-Dec-2013)
> 
> Warning: the bigalloc feature is still under development
> See https://ext4.wiki.kernel.org/index.php/Bigalloc for more information
> 
> warning: 53248 blocks unused.
> 
> Filesystem label=iso
> OS type: Linux
> Block size=4096 (log=2)
> Cluster size=268435456 (log=18)
> Stride=0 blocks, Stripe width=0 blocks
> 720 inodes, 11665408 blocks
> 0 blocks (0.00%) reserved for the super user
> First data block=0
> Root directory owner=1000:100
> 1 block group
> 2146959360 blocks per group, 32760 clusters per group
> 720 inodes per group
> 
> Allocating group tables: done
> Writing inode tables: done
> Creating journal (32768 blocks): done
> Writing superblocks and filesystem accounting information: done
> 
> It took it some lengthy time to allocate the group tables, then regular
> copy worked but fallocate hanged (process uninterruptible) and I had to
> reboot the machine, which led me to a kernel panic mentionning
> fallocate. The file copied via Nautilus still wasn't contiguous enough
> for Easy2Boot.
> 
> So what's left? Is it at all possible to make completely contiguous
> files on ext4? Not even on a fresh FS with plenty of free space?
> 
> Thanks for your comments!
> 
> 
> [1] http://www.easy2boot.com/introduction/
> [2] https://code.google.com/p/grub4dos-chenall/
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Make huge files strictly contiguous (fallocate, bigalloc, e4defrag...)

[Theodore Ts'o]

On Thu, Apr 17, 2014 at 04:18:30AM +0200, Contact wrote:
> 
> I'm trying to create a multiboot medium using Easy2Boot on a 64 GB USB
> key. Easy2Boot [1] is a "software" (well a collection of files really)
> that relies on grub4dos [2] to work and thus requires payload files to
> be strictly contiguous to do its trick. It's supposed to be set up once
> and then one just has to add files to boot from to a folder to modify
> the multiboot setup.
> 
> The ISO files I have are between 230 MB and 2 GB each. I divided my USB
> drive into two partitions : first one, 16 GB ntfs formatted so that
> grub4dos actually works (for some reasons it doesn't with ext4, plus I
> want to be able to use it with other OS); second one 48 GB formatted as
> ext4 with e2fsprogs version 1.42.9 from Archlinux. However when I try
> to boot from those files I systematically get the "file not contiguous"
> error in Easy2Boot.
> 
> Question is: is it possible to make huge files strictly contiguous on
> ex4, and if yes how?

Most of the time, files don't have to be "strictly contiguous", so
that's not something that we've spent a lot of time trying to achieve.

There is a way to do this if you are willing to use the tip of the
e2fsprogs "maint" branch, and then you put something like this into
your mke2fs.conf file:

	easy2boot = {
		features = extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,^resize_inode,sparse_super2
		hash_alg = half_md4
		num_backup_sb = 0
		packed_meta_blocks = 1
		make_hugefiles = 1
		inode_ratio = 4194304
		hugefiles_dir = /
		hugefiles_name = my-big-file
		hugefiles_digits = 0
		hugefiles_size = 16G
		num_hugefiles = 1
		zero_hugefiles = false
	}

Then "mke2fs -T easy2boot /dev/sdc1" will create an ext4 filesystem
with a file called /my-big-file which is guaranteed to be contiguous.

For your particular use case, where you want to create a new
filesystem when you want to create your strictly contiguous file, this
might be the best way to go.

Cheers,

						- Ted

Re: Make huge files strictly contiguous (fallocate, bigalloc, e4defrag...)

[Contact]

Le 17/04/2014 14:04, Lukáš Czerner a écrit :
>
> This is not how it it supposed to be used. Yes fallocate
> preallocates the file, but cp will truncate it so fallocate will
> certainly not help you in any way. In order for fallocate to be
> useful you'll have to write into the file without actually
> truncating it (dd can do this if you do not want to write your own
> program)
>
> Also the file is probably as contiguous as it could be. Here is what
> I get on the file system with default mkfs options.
>
> # e4defrag -c /mnt/test/file1
> <File>                                         now/best      size/ext
> /mnt/test/file1                                 10/1         120649 KB
>
> But that does not tell the whole story. See
>
> xfs_io -f -c "fiemap -v" /mnt/test/file1
> /mnt/test/file1:
>  EXT: FILE-OFFSET        BLOCK-RANGE       TOTAL FLAGS
>    0: [0..262143]:       2768896..3031039 262144   0x0
>    1: [262144..524287]:  3031040..3293183 262144   0x0
>    2: [524288..786431]:  3293184..3555327 262144   0x0
>    3: [786432..1048575]: 3555328..3817471 262144   0x0
>    4: [1048576..1310719]: 3817472..4079615 262144   0x0
>    5: [1310720..1425407]: 4079616..4194303 114688   0x0
>    6: [1425408..1687551]: 4456448..4718591 262144   0x0
>    7: [1687552..1949695]: 4718592..4980735 262144   0x0
>    8: [1949696..2211839]: 4980736..5242879 262144   0x0
>    9: [2211840..2412991]: 5242880..5444031 201152   0x1
>
> (Note that the output is in 512B blocks)
>
> As you can see the file is mostly contiguous, but it is divided into
> several extents because of two reasons.
>
> 1. The extent in ext4 has a limited size of 32768 blocks for
> initialized extent and 32767 block for unwritten extent. So when we
> exceed that size we need another extent which might be physically
> contiguous on disk with the previous one.
>
> 2. Ext4 divides disk space into allocation groups of certain size
> (cluster size * 8)blocks. Now with flex_bg medatada such as inode
> tables, block bitmaps and so one are packed closely together so the
> do not have to be stored with each block group and you'll get more
> contiguous data space.
>
> However we're still storing backup superblock and Groups descriptors
> in certain groups and those are the gaps you're seeing in the fiemap
> list.
>
> For detailed overview you can use dumpe2fs to see what is allocated
> where on the file system.
>

Thanks, that was much interesting. I had delved a bit into ext4 data
structure before posting but was never able to get a clear grasp on the
limitations concerning contiguity.

So, I tried the correct way you recommended about how to use fallocate:

# mkfs.ext4 -m 0 -L iso -i 67108864 -E root_owner=1000:100 /dev/sdc2
$ fallocate -l 1589166080
'/run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso'
$ dd if='/home/neitsab/iso/Manjaro/manjaro-gnome-0.8.9-x86_64.iso'
of='/run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso'

However grub4dos displayed the error message about file being
non-contiguous. Output of xfs_io:
$ xfs_io -f -c "fiemap -v"
'/run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso'
/run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso:
 EXT: FILE-OFFSET        BLOCK-RANGE       TOTAL FLAGS
   0: [0..262143]:       3457024..3719167 262144   0x0
   1: [262144..524287]:  3719168..3981311 262144   0x0
   2: [524288..737279]:  3981312..4194303 212992   0x0
   3: [737280..999423]:  4456448..4718591 262144   0x0
   4: [999424..1261567]: 4718592..4980735 262144   0x0
   5: [1261568..1523711]: 4980736..5242879 262144   0x0
   6: [1523712..1785855]: 5242880..5505023 262144   0x0
   7: [1785856..2047999]: 5505024..5767167 262144   0x0
   8: [2048000..2310143]: 5767168..6029311 262144   0x0
   9: [2310144..2572287]: 6029312..6291455 262144   0x0
  10: [2572288..2834431]: 6291456..6553599 262144   0x0
  11: [2834432..3096575]: 6569984..6832127 262144   0x0
  12: [3096576..3103839]: 6832128..6839391   7264   0x1

Second and third extents aren't contiguous, neither are tenth and
eleventh. But it was close!

Anyway, thanks to your explanations I understand why perfect contiguity
isn't so likely. However I remember using with success a Perl script
from 2007 called defragfs [1] recommended by Easy2Boot's author. After
running this script a couple of times on the key's ISO folder I was able
to boot on most of the files. Problem with this approach, is that AFAIK
defragmentation is pretty harmful for flash drives, so I wasn't willing
to do it every time I add an ISO.

> Bigalloc file system should help since you'll get much larger
> contiguous data spaces since the cluster size would be much bigger
> hence you'll get much bigger block groups. But you'll still get
> multiple extents (I do not really know how grub4dos recognizes
> contiguous files) even though there will be mostly contiguous.
>
> But then again you'll have Group descriptors and backup superblocks
> in some groups so potentially some files might end up not
> contiguous. And even though you can turn off backup superblock you
> can not turn off writing group descriptors.
>
> So no, at the moment there is not way to make really *really* sure
> that the file you're creating even with fallocate will allways be
> strictly contiguous. It'll mostly work, but it really depends on
> where the file will be put on the file system and how big is the
> file.
>
> Also I've been talking about the case where nothing else is using
> the file system, when that's not the case. It might happen that the
> allocation will interfere (even though we're trying to allocate as
> contiguous file as we can). And also the file system free space will
> become more fragmented over time as it is used, but it really
> depends on the workload.
>
>
> Now for the problem with bigalloc, I am not sure what kernel version
> are you using, but it's probably old. fallocate on bigalloc should
> work just fine.
>
> Hope it helps.
> Thanks!
> -Lukas
>

Concerning fallocate with bigalloc, I don't think it is related to my
kernel which is rather recent:
$ uname -a
Linux arch-clevo 3.14.1-1-ARCH #1 SMP PREEMPT Mon Apr 14 20:40:47 CEST
2014 x86_64 GNU/Linux

So, I tried again to use fallocate on the bigalloc'ed fs, it hanged
again (uninterruptible sleep state, D in ps aux:
$ ps aux | grep fallocate
neitsab   3673  0.0  0.0      0     0 ?        D    19:36   0:00 [fallocate]
kill/kill -9 don't change anything. gnome-system-monitor gives in the
process properties for fallocate: "Wait channel |
call_rwsem_down_write_failed".

So after searching a bit more, it seems to be related to a driver
issue (see [2] and [3] for a discussion and more info), apparently
fallocate is waiting on some I/O that never come... So that's on my USB
key (for info it is a Sandisk Extreme 64 GB USB 3.0, model SDCZ80-0654G,
using driver xhci_hcd).

As I've been advised in another discussion to be cautious with bigalloc
because it is still in development, I'll leave it aside for now.


> Le 17/04/2014 17:24, Theodore Ts'o a écrit :
> 
> Most of the time, files don't have to be "strictly contiguous", so
> that's not something that we've spent a lot of time trying to achieve.
> 
> There is a way to do this if you are willing to use the tip of the
> e2fsprogs "maint" branch, and then you put something like this into
> your mke2fs.conf file:
> 
> 	easy2boot = {
> 		features = extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,^resize_inode,sparse_super2
> 		hash_alg = half_md4
> 		num_backup_sb = 0
> 		packed_meta_blocks = 1
> 		make_hugefiles = 1
> 		inode_ratio = 4194304
> 		hugefiles_dir = /
> 		hugefiles_name = my-big-file
> 		hugefiles_digits = 0
> 		hugefiles_size = 16G
> 		num_hugefiles = 1
> 		zero_hugefiles = false
> 	}
> 
> Then "mke2fs -T easy2boot /dev/sdc1" will create an ext4 filesystem
> with a file called /my-big-file which is guaranteed to be contiguous.
> 
> For your particular use case, where you want to create a new
> filesystem when you want to create your strictly contiguous file, this
> might be the best way to go.
> 

Thanks a lot for the detailed solution! If I want mke2fs to create
multiple contiguous files (this is for a multiboot medium so I'm gonna
have quite a few of them, e.g. 20 files of 2 GB so as to account for the
maximum), which variables should I modify in the mke2fs.conf entry?
hugefiles_size and num_hugefiles?

Although I'd like to test this solution, I still have to get into git
and compiling, moreover I'm looking for a less involving solution so for
now I think I'll just stick to regular e2fsprogs and try to find another
multiboot utility.

But thanks for your quick and detailed answers!

-- Bastien

[1] http://defragfs.sourceforge.net/
[2] http://linuxgazette.net/issue83/tag/6.html
[3] http://www.novell.com/support/kb/doc.php?id=7002725
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Make huge files strictly contiguous (fallocate, bigalloc, e4defrag...)

[Contact]

Just a little update:
Nonetheless my issues with fallocate and the bigalloc'ed fs, I tried to
dd a couple of ISOs on the key and check whether I could boot from them.
And to my great surprise... The answer was yes.

$ ls -l /run/media/neitsab/iso/_ISO/
total 4980736
drwxr-xr-x 2 neitsab users       4096 17 avril 21:38 .
drwxr-xr-x 4 neitsab users       4096 17 avril 19:36 ..
-rw------- 1 neitsab users  964689920 17 avril 21:37 gnome-3.12.iso
-rw------- 1 neitsab users 1589166080 17 avril 20:45
manjaro-gnome-0.8.9-x86_64.iso
-rw------- 1 neitsab users  993001472 17 avril 21:38
openSUSE-13.1-GNOME-Live-x86_64.iso

I can boot from the two others but not from manjaro's one, which happens
to be the biggest too. A comparison of xfs_io output :

$ xfs_io -f -c "fiemap -v"
'/run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso'
/run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso:
 EXT: FILE-OFFSET        BLOCK-RANGE       TOTAL FLAGS
   0: [0..262143]:       3145728..3407871 262144   0x0
   1: [262144..524287]:  3407872..3670015 262144   0x0
   2: [524288..786431]:  3670016..3932159 262144   0x0
   3: [786432..1048575]: 3932160..4194303 262144   0x0
   4: [1048576..1310719]: 4194304..4456447 262144   0x0
   5: [1310720..1572863]: 4456448..4718591 262144   0x0
   6: [1572864..1835007]: 5242880..5505023 262144   0x0
   7: [1835008..2097151]: 5505024..5767167 262144   0x0
   8: [2097152..2359295]: 5767168..6029311 262144   0x0
   9: [2359296..2621439]: 6029312..6291455 262144   0x0
  10: [2621440..2883583]: 6291456..6553599 262144   0x0
  11: [2883584..3103839]: 6553600..6773855 220256   0x1

$ xfs_io -f -c "fiemap -v"
'/run/media/neitsab/iso/_ISO/openSUSE-13.1-GNOME-Live-x86_64.iso'
/run/media/neitsab/iso/_ISO/openSUSE-13.1-GNOME-Live-x86_64.iso:
 EXT: FILE-OFFSET        BLOCK-RANGE         TOTAL FLAGS
   0: [0..262143]:       8912896..9175039   262144   0x0
   1: [262144..524287]:  9175040..9437183   262144   0x0
   2: [524288..786431]:  9437184..9699327   262144   0x0
   3: [786432..1048575]: 9699328..9961471   262144   0x0
   4: [1048576..1310719]: 9961472..10223615  262144   0x0
   5: [1310720..1572863]: 10223616..10485759 262144   0x0
   6: [1572864..1835007]: 10485760..10747903 262144   0x0
   7: [1835008..1939455]: 10747904..10852351 104448   0x1

$ xfs_io -f -c "fiemap -v" '/run/media/neitsab/iso/_ISO/gnome-3.12.iso'
/run/media/neitsab/iso/_ISO/gnome-3.12.iso:
 EXT: FILE-OFFSET        BLOCK-RANGE       TOTAL FLAGS
   0: [0..262143]:       6815744..7077887 262144   0x0
   1: [262144..524287]:  7077888..7340031 262144   0x0
   2: [524288..786431]:  7340032..7602175 262144   0x0
   3: [786432..1048575]: 7602176..7864319 262144   0x0
   4: [1048576..1310719]: 7864320..8126463 262144   0x0
   5: [1310720..1572863]: 8126464..8388607 262144   0x0
   6: [1572864..1835007]: 8388608..8650751 262144   0x0
   7: [1835008..1884159]: 8650752..8699903  49152   0x1

So they're all contiguous but Manjaro. I guess that brings us back to
the randomness of block allocation as explained by Lukáš and to an
effect of file size on the probability to be non-contiguous (which is
rather logical)!

Thanks all for your explanations,
Cheers
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Make huge files strictly contiguous (fallocate, bigalloc, e4defrag...)

[Lukáš Czerner]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3652 bytes --]

On Thu, 17 Apr 2014, Contact wrote:

> Date: Thu, 17 Apr 2014 20:41:34 +0200
> From: Contact <neitsab@ovh.fr>
> To: Theodore Ts'o <tytso@mit.edu>, lczerner@redhat.com
> Cc: "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>
> Subject: Re: Make huge files strictly contiguous (fallocate, bigalloc,
>     e4defrag...)
> 
> Le 17/04/2014 14:04, Lukáš Czerner a écrit :
> >
> > This is not how it it supposed to be used. Yes fallocate
> > preallocates the file, but cp will truncate it so fallocate will
> > certainly not help you in any way. In order for fallocate to be
> > useful you'll have to write into the file without actually
> > truncating it (dd can do this if you do not want to write your own
> > program)
> >
> > Also the file is probably as contiguous as it could be. Here is what
> > I get on the file system with default mkfs options.
> >
> > # e4defrag -c /mnt/test/file1
> > <File>                                         now/best      size/ext
> > /mnt/test/file1                                 10/1         120649 KB
> >
> > But that does not tell the whole story. See
> >
> > xfs_io -f -c "fiemap -v" /mnt/test/file1
> > /mnt/test/file1:
> >  EXT: FILE-OFFSET        BLOCK-RANGE       TOTAL FLAGS
> >    0: [0..262143]:       2768896..3031039 262144   0x0
> >    1: [262144..524287]:  3031040..3293183 262144   0x0
> >    2: [524288..786431]:  3293184..3555327 262144   0x0
> >    3: [786432..1048575]: 3555328..3817471 262144   0x0
> >    4: [1048576..1310719]: 3817472..4079615 262144   0x0
> >    5: [1310720..1425407]: 4079616..4194303 114688   0x0
> >    6: [1425408..1687551]: 4456448..4718591 262144   0x0
> >    7: [1687552..1949695]: 4718592..4980735 262144   0x0
> >    8: [1949696..2211839]: 4980736..5242879 262144   0x0
> >    9: [2211840..2412991]: 5242880..5444031 201152   0x1
> >
> > (Note that the output is in 512B blocks)
> >
> > As you can see the file is mostly contiguous, but it is divided into
> > several extents because of two reasons.
> >
> > 1. The extent in ext4 has a limited size of 32768 blocks for
> > initialized extent and 32767 block for unwritten extent. So when we
> > exceed that size we need another extent which might be physically
> > contiguous on disk with the previous one.
> >
> > 2. Ext4 divides disk space into allocation groups of certain size
> > (cluster size * 8)blocks. Now with flex_bg medatada such as inode
> > tables, block bitmaps and so one are packed closely together so the
> > do not have to be stored with each block group and you'll get more
> > contiguous data space.
> >
> > However we're still storing backup superblock and Groups descriptors
> > in certain groups and those are the gaps you're seeing in the fiemap
> > list.
> >
> > For detailed overview you can use dumpe2fs to see what is allocated
> > where on the file system.
> >
> 
> Thanks, that was much interesting. I had delved a bit into ext4 data
> structure before posting but was never able to get a clear grasp on the
> limitations concerning contiguity.
> 
> So, I tried the correct way you recommended about how to use fallocate:
> 
> # mkfs.ext4 -m 0 -L iso -i 67108864 -E root_owner=1000:100 /dev/sdc2
> $ fallocate -l 1589166080
> '/run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso'
> $ dd if='/home/neitsab/iso/Manjaro/manjaro-gnome-0.8.9-x86_64.iso'
> of='/run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso'

The way you're using dd for this purpose is wrong. I
said that dd can write to the file without truncating it first, but
it's not doing it by default. You have to use 'notrunc' see man
page. But I think that it would not have made a difference anyway.

-Lukas

Re: Make huge files strictly contiguous (fallocate, bigalloc, e4defrag...)

[Contact]

Le ven. 18 avril 2014 10:45:03 CEST, Lukáš Czerner a écrit :
>
> The way you're using dd for this purpose is wrong. I
> said that dd can write to the file without truncating it first, but
> it's not doing it by default. You have to use 'notrunc' see man
> page. But I think that it would not have made a difference anyway.
>
Thansk, I stand corrected. I'll try it to see whether I can get better 
results with it.

> And even though you can turn off backup superblock you
> can not turn off writing group descriptors.
>
While we're at it, how can I do this? I searched the 
mkfs.ext4/mke2fs.conf/ext4/tune2fs man pages for the option to turn off 
writing backup superblock you mentioned, however the closest I found is 
tune2fs -O ^sparse_super which is part of the defaults mke2fs.conf 
options... Could you point me at it so that I can try to get this 
source of non-contiguity away?

Thanks,
-Bastien


--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Make huge files strictly contiguous (fallocate, bigalloc, e4defrag...)

[Theodore Ts'o]

On Sun, Apr 20, 2014 at 04:37:54PM +0200, Contact wrote:
> While we're at it, how can I do this? I searched the 
> mkfs.ext4/mke2fs.conf/ext4/tune2fs man pages for the option to turn off 
> writing backup superblock you mentioned, however the closest I found is 
> tune2fs -O ^sparse_super which is part of the defaults mke2fs.conf 
> options... Could you point me at it so that I can try to get this 
> source of non-contiguity away?

This requires recompiling e2fsprogs using the tip of the maint branch,
as I had suggested earlier (and which you had ruled out because you
didn't want to build your own e2fsprogs from source).

Once you do this, and you put the following in 

	hugefiles = {
		features = extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,^resize_inode,sparse_super2
		hash_alg = half_md4
		num_backup_sb = 0
		packed_meta_blocks = 1
		flex_bg_size = 262144
		inode_ratio = 4194304
		make_hugefiles = 1
		hugefiles_dir = /boot
		hugefiles_name = contig-files
		hugefiles_digits = 1
		hugefiles_size = 256M
		num_hugefiles = 3
	}

Using the sparse_super2 feature and setting num_backup_sb=0 disables
the backup superblocks.  Then setting flex_bg_size to a very large
value (262144 in this case), will force all of the allocation bitmaps
and inodetables to the very beginning of the file system.

The inode_ratio setting simply radically reduces the number of inodes
in the file system.  That was there because originally I was using
this this command set with the hugefiles_size set to 1 or 2 gigabytes,
and filling the disk with completely with large files (num_hugefiles =
0 means "create as many huge files as necessary to fill the disk). 

The rest of the "hugefiles" related settings are just simply allowing
mke2fs to create the contiguous files for you.

Note that because this uses a new compat feature, "sparse_super2", if
you use the new mke2fs to create such a file system, you will also
need to make sure the file system is checked using a version of e2fsck
that supports this new feature as well.  No kernel changes are
required to support this, though.

Cheers,

					- Ted

Re: Make huge files strictly contiguous (fallocate, bigalloc, e4defrag...)

[Contact]

Le dim. 20 avril 2014 21:01:46 CEST, Theodore Ts'o a écrit :
>
> This requires recompiling e2fsprogs using the tip of the maint branch,
> as I had suggested earlier (and which you had ruled out because you
> didn't want to build your own e2fsprogs from source).
>
Thanks, that explains why I wasn't able to find this option in regular 
e2fsprogs.

> Once you do this, and you put the following in
>
> 	hugefiles = {
> 		features = extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,^resize_inode,sparse_super2
> 		hash_alg = half_md4
> 		num_backup_sb = 0
> 		packed_meta_blocks = 1
> 		flex_bg_size = 262144
> 		inode_ratio = 4194304
> 		make_hugefiles = 1
> 		hugefiles_dir = /boot
> 		hugefiles_name = contig-files
> 		hugefiles_digits = 1
> 		hugefiles_size = 256M
> 		num_hugefiles = 3
> 	}
>
> Using the sparse_super2 feature and setting num_backup_sb=0 disables
> the backup superblocks.  Then setting flex_bg_size to a very large
> value (262144 in this case), will force all of the allocation bitmaps
> and inodetables to the very beginning of the file system.
>
The tip and stanza look very useful though, I'll reconsider retrieving 
and compiling maint branch to try, thanks.

> Note that because this uses a new compat feature, "sparse_super2", if
> you use the new mke2fs to create such a file system, you will also
> need to make sure the file system is checked using a version of e2fsck
> that supports this new feature as well.  No kernel changes are
> required to support this, though.
>
Any plans to get this feature into master at some point in the 
development?

--Bastien

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Make huge files strictly contiguous (fallocate, bigalloc, e4defrag...)

[Theodore Ts'o]

On Sun, Apr 20, 2014 at 09:38:30PM +0200, Contact wrote:
> > Note that because this uses a new compat feature, "sparse_super2", if
> > you use the new mke2fs to create such a file system, you will also
> > need to make sure the file system is checked using a version of e2fsck
> > that supports this new feature as well.  No kernel changes are
> > required to support this, though.
> >
> Any plans to get this feature into master at some point in the 
> development?

The maint branch is regularly merged into the master and next
branches, so it's there today.  The maint branch will very shortly be
the source of a 1.42.10 maintenance release.

The master branch will eventualy be a 1.43 release; it has some much
more disruptive changes, and adds support for metadata checksums,
inline data, etc.  It's likely that 1.42.10 will be the last
maintenance release, and the focus will be on stablizing 1.43 and
making sure we're really ready to support the ABI's for the new
library functions in libext2fs to support these new features _forever_.

Cheers,

					- Ted