linux-ext4.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH V5 00/23] ext4: Add inline data support.
@ 2012-06-30 15:41 Tao Ma
  2012-06-30 15:45 ` [PATCH V5 01/23] ext4: Move extra inode read to a new function Tao Ma
  2012-08-13 15:04 ` [PATCH V5 00/23] ext4: Add inline data support Theodore Ts'o
  0 siblings, 2 replies; 29+ messages in thread
From: Tao Ma @ 2012-06-30 15:41 UTC (permalink / raw)
  To: linux-ext4

Hi list,

this is another try for the inline data support for ext4.
I have runned all the xfstests cases and fixes the bugs it found.
Hope it is stable now for the inclusion.

So in genernal, I have runned several test cases and try to prove
that it works.

A fsstats shows the directory information for my home.
(The test tools can be found here. http://www.pdsi-scidac.org/fsstats/.
Thanks goes to Andreas Dilger).

directory size (entries): Range of entries, count of directories in range, number of dirs in range as a % of total num of dirs, number of dirs in this range or smaller as a % total number of dirs, total entries in range, number of entries in range as a % of total number of entries, number of entries in this range or smaller as a % of total number of entries.
  count=13671 avg=11.58 ents
  min=0.00 ents max=4562.00 ents
  [   0-   1 ents]:  5198 (38.02%) ( 38.02% cumulative)   3322.00 ents ( 2.10%) (  2.10% cumulative) 
  [   2-   3 ents]:  2246 (16.43%) ( 54.45% cumulative)   5477.00 ents ( 3.46%) (  5.56% cumulative) 
  [   4-   7 ents]:  2266 (16.58%) ( 71.03% cumulative)  12037.00 ents ( 7.60%) ( 13.16% cumulative) 
  [   8-  15 ents]:  1877 (13.73%) ( 84.76% cumulative)  20269.00 ents (12.80%) ( 25.96% cumulative) 
  [  16-  31 ents]:  1126 ( 8.24%) ( 92.99% cumulative)  24196.00 ents (15.28%) ( 41.24% cumulative) 
  [  32-  63 ents]:   554 ( 4.05%) ( 97.04% cumulative)  23772.00 ents (15.01%) ( 56.26% cumulative) 
  [  64- 127 ents]:   234 ( 1.71%) ( 98.76% cumulative)  20919.00 ents (13.21%) ( 69.47% cumulative) 
  [ 128- 255 ents]:   119 ( 0.87%) ( 99.63% cumulative)  20654.00 ents (13.05%) ( 82.52% cumulative) 
  [ 256- 511 ents]:    40 ( 0.29%) ( 99.92% cumulative)  12941.00 ents ( 8.17%) ( 90.69% cumulative) 
  [ 512-1023 ents]:     6 ( 0.04%) ( 99.96% cumulative)   4147.00 ents ( 2.62%) ( 93.31% cumulative) 
  [1024-2047 ents]:     3 ( 0.02%) ( 99.99% cumulative)   3487.00 ents ( 2.20%) ( 95.51% cumulative) 
  [2048-4095 ents]:     1 ( 0.01%) ( 99.99% cumulative)   2544.00 ents ( 1.61%) ( 97.12% cumulative) 
  [4096-8191 ents]:     1 ( 0.01%) (100.00% cumulative)   4562.00 ents ( 2.88%) (100.00% cumulative) 

So with inode size equals 256, more than 50% directory can be inlined.

I have also runned several other test cases to prove the peroformance gain,
And I'd like to describe the result here. For more test details, please see below.

For a dir iteration test, if most of the dirs are inlined, we can see a 10 fold
speed up(240ms vs. 2250ms) in dir iteration(try to find a not-exist file). Yes,
that is expected. ;)

With a normal kernel tar and directory iteration test, we can find a 1.5% space
saving(with cluster size=4096, I would imagine a better result with bigalloc enabled).
As for the directory iteration, we can see a 22.4% speedup.

I ran postmark for 2 senarios, one with file size between 11 and 500 which is good
for inline data test, and another with file size between 100 and 50000. We can see
a 32.8%(49s vs. 73s) improvement for the first one, and 3.6%(805s vs. 835s) for the
latter.

The final test case is the fs_mark. I ran the different sync modes, and we can see
more than 100% increase if we does the sync and this is really a huge win.

1. normal test case of taring a kernel source and find a file:
time tar zxvf linux-3.4.tar.gz -C $MNT_DIR
df -hm|grep $MNT_DIR
umount $MNT_DIR
mount -t ext4 $DEVICE $MNT_DIR
time find $MNT_DIR -name a

inline_data result
in find real 1350ms
in find sys 230ms
in find user 90ms

normal result
in find real 1840ms
in find sys 340ms
in find user 110ms

inline_data + journal result
in find real 1590ms
in find sys 230ms
in find user 80ms

normal journal result
in find real 1900ms
in find sys 360ms
in find user 100ms

2. postmark test case 1:
set location $MNT_DIR
set subdirectories 2570
set size 11 500
set transactions 51110
set number 977770

with inline data result:
Time:
	49 seconds total
	2 seconds of transactions (25555 per second)

Files:
	1003234 created (20474 per second)
		Creation alone: 977770 files (29629 per second)
		Mixed with transactions: 25464 files (12732 per second)
	25561 read (12780 per second)
	25548 appended (12774 per second)
	1003234 deleted (20474 per second)
		Deletion alone: 977588 files (69827 per second)
		Mixed with transactions: 25646 files (12823 per second)

Data:
	6.25 megabytes read (130.59 kilobytes per second)
	247.18 megabytes written (5.04 megabytes per second)

Without inline data:
Time:
	73 seconds total
	25 seconds of transactions (2044 per second)

Files:
	1003234 created (13742 per second)
		Creation alone: 977770 files (31540 per second)
		Mixed with transactions: 25464 files (1018 per second)
	25561 read (1022 per second)
	25548 appended (1021 per second)
	1003234 deleted (13742 per second)
		Deletion alone: 977588 files (57505 per second)
		Mixed with transactions: 25646 files (1025 per second)

Data:
	6.25 megabytes read (87.66 kilobytes per second)
	247.18 megabytes written (3.39 megabytes per second)

postmark test 2:
set location $MNT_DIR
set subdirectories 2570
set size 100 50000
set transactions 51110
set number 977770

with inline data:
Time:
	805 seconds total
	263 seconds of transactions (194 per second)

Files:
	1003213 created (1246 per second)
		Creation alone: 977770 files (1975 per second)
		Mixed with transactions: 25443 files (96 per second)
	25597 read (97 per second)
	25513 appended (97 per second)
	1003213 deleted (1246 per second)
		Deletion alone: 977546 files (20798 per second)
		Mixed with transactions: 25667 files (97 per second)

without inline data:
Time:
	835 seconds total
	289 seconds of transactions (176 per second)

Files:
	1003213 created (1201 per second)
		Creation alone: 977770 files (2011 per second)
		Mixed with transactions: 25443 files (88 per second)
	25597 read (88 per second)
	25513 appended (88 per second)
	1003213 deleted (1201 per second)
		Deletion alone: 977546 files (16292 per second)
		Mixed with transactions: 25667 files (88 per second)

3. fs_mark test:
fs_mark -d $MNT_DIR -D 256 -s 256 -n 10000 -t 256 -S [0|1|2]

                         inline+nojournal noinline+nojournal
Files/sec for sync 0     63757.4          63086.4
Files/sec for sync 1     38976.7          16983.5
Files/sec for sync 2     35796.7          14682.6

                         inline+journal   noinline+journal
Files/sec for sync 0     63523.4          61092.1
Files/sec for sync 1     34441.4          15061.8
Files/sec for sync 2     40745.3          14083.4

git diff --stat
 fs/ext4/Makefile  |    2 +-
 fs/ext4/dir.c     |   39 +-
 fs/ext4/ext4.h    |   88 +++-
 fs/ext4/extents.c |   22 +-
 fs/ext4/ialloc.c  |    4 +
 fs/ext4/inline.c  | 1827 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/ext4/inode.c   |  237 ++++++--
 fs/ext4/namei.c   |  462 +++++++++-----
 fs/ext4/xattr.c   |   87 ++-
 fs/ext4/xattr.h   |  277 ++++++++
 10 files changed, 2781 insertions(+), 264 deletions(-)

any comments are welcomed.

Thanks
Tao

^ permalink raw reply	[flat|nested] 29+ messages in thread

end of thread, other threads:[~2012-08-15  2:51 UTC | newest]

Thread overview: 29+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-06-30 15:41 [PATCH V5 00/23] ext4: Add inline data support Tao Ma
2012-06-30 15:45 ` [PATCH V5 01/23] ext4: Move extra inode read to a new function Tao Ma
2012-06-30 15:45   ` [PATCH V5 02/23] ext4: export inline xattr functions Tao Ma
2012-06-30 15:45   ` [PATCH V5 03/23] ext4: Add the basic function for inline data support Tao Ma
2012-06-30 15:45   ` [PATCH V5 04/23] ext4: Add read support for inline data Tao Ma
2012-06-30 15:45   ` [PATCH V5 05/23] ext4: Add normal write " Tao Ma
2012-06-30 15:45   ` [PATCH V5 06/23] ext4: Add journalled " Tao Ma
2012-06-30 15:45   ` [PATCH V5 07/23] ext4: Add delalloc " Tao Ma
2012-06-30 15:45   ` [PATCH V5 08/23] ext4: Create a new function ext4_init_new_dir Tao Ma
2012-06-30 15:45   ` [PATCH V5 09/23] ext4: Refactor __ext4_check_dir_entry to accepts start and size Tao Ma
2012-06-30 15:45   ` [PATCH V5 10/23] ext4: Create __ext4_insert_dentry for dir entry insertion Tao Ma
2012-06-30 15:45   ` [PATCH V5 11/23] ext4: let add_dir_entry handle inline data properly Tao Ma
2012-06-30 15:45   ` [PATCH V5 12/23] ext4: Let ext4_readdir handle inline data Tao Ma
2012-06-30 15:45   ` [PATCH V5 13/23] ext4: Create a new function search_dir Tao Ma
2012-06-30 15:45   ` [PATCH V5 14/23] ext4: let ext4_find_entry handle inline data Tao Ma
2012-06-30 15:45   ` [PATCH V5 15/23] ext4: make ext4_delete_entry generic Tao Ma
2012-06-30 15:45   ` [PATCH V5 16/23] ext4: let ext4_delete_entry handle inline data Tao Ma
2012-06-30 15:45   ` [PATCH V5 17/23] ext4: let empty_dir handle inline dir Tao Ma
2012-06-30 15:45   ` [PATCH V5 18/23] ext4: let ext4_rename " Tao Ma
2012-06-30 15:45   ` [PATCH V5 19/23] ext4: Let fiemap work with inline data Tao Ma
2012-06-30 15:45   ` [PATCH V5 20/23] ext4: Evict inline data out if we needs to strore xattr in inode Tao Ma
2012-06-30 15:45   ` [PATCH V5 21/23] ext4: let ext4_truncate handle inline data correctly Tao Ma
2012-06-30 15:45   ` [PATCH V5 22/23] ext4: let fallocate " Tao Ma
2012-06-30 15:45   ` [PATCH V5 23/23] ext4: Enable ext4 inline support Tao Ma
2012-08-13 15:04 ` [PATCH V5 00/23] ext4: Add inline data support Theodore Ts'o
2012-08-14  2:43   ` Andreas Dilger
2012-08-14  7:04   ` Tao Ma
2012-08-14 15:19     ` Theodore Ts'o
2012-08-15  2:51       ` Andreas Dilger

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).