From: Jacek Luczak <difrost.kernel@gmail.com>
To: linux-ext4@vger.kernel.org,
linux-fsdevel <linux-fsdevel@vger.kernel.org>,
LKML <linux-kernel@vger.kernel.org>,
linux-btrfs@vger.kernel.org, chris.mason@oracle.com
Subject: Re: getdents - ext4 vs btrfs performance
Date: Wed, 29 Feb 2012 14:55:11 +0100 [thread overview]
Message-ID: <CADDYkjRni4i3QvWo+hOn-Ct4mFoTpL1-ydmE7fbzYJmHFhS3_Q@mail.gmail.com> (raw)
In-Reply-To: <CADDYkjS5VJeYyHzqumazQ0qKg+HwA6GO+zYSJj7rkHNZFwjcoQ@mail.gmail.com>
Hi Chris,
the last one was borked :) Please check this one.
-jacek
2012/2/29 Jacek Luczak <difrost.kernel@gmail.com>:
> Hi All,
>
> /*Sorry for sending incomplete email, hit wrong button :) I guess I
> can't use Gmail */
>
> Long story short: We've found that operations on a directory structur=
e
> holding many dirs takes ages on ext4.
>
> The Question: Why there's that huge difference in ext4 and btrfs? See
> below test results for real values.
>
> Background: I had to backup a Jenkins directory holding workspace for
> few projects which were co from svn (implies lot of extra .svn dirs).
> The copy takes lot of time (at least more than I've expected) and
> process was mostly in D (disk sleep). I've dig more and done some
> extra test to see if this is not a regression on block/fs site. To
> isolate the issue I've also performed same tests on btrfs.
>
> Test environment configuration:
> 1) HW: HP ProLiant BL460 G6, 48 GB of memory, 2x 6 core Intel X5670 H=
T
> enabled, Smart Array P410i, RAID 1 on top of 2x 10K RPM SAS HDDs.
> 2) Kernels: All tests were done on following kernels:
> =A0- 2.6.39.4-3 -- the build ID (3) is used here for internal tacking=
of
> config changes mostly. In -3 we've introduced ,,fix readahead pipelin=
e
> break caused by block plug'' patch. Otherwise it's pure 2.6.39.4.
> =A0- 3.2.7 -- latest kernel at the time of testing (3.2.8 has been
> release recently).
> 3) A subject of tests, directory holding:
> =A0- 54GB of data (measured on ext4)
> =A0- 1978149 files
> =A0- 844008 directories
> 4) Mount options:
> =A0- ext4 -- errors=3Dremount-ro,noatime,
> data=3Dwriteback
> =A0- btrfs -- noatime,nodatacow and for later investigation on
> copression effect: noatime,nodatacow,compress=3Dlzo
>
> In all tests I've been measuring time of execution. Following tests
> were performed:
> - find . -type d
> - find . -type f
> - cp -a
> - rm -rf
>
> Ext4 results:
> | Type =A0 =A0 | 2.6.39.4-3 =A0 | 3.2.7
> | Dir cnt =A0| 17m 40sec =A0| 11m 20sec
> | File cnt | =A017m 36sec | 11m 22sec
> | Copy =A0 =A0| 1h 28m =A0 =A0 =A0 =A0| 1h 27m
> | Remove| 3m 43sec =A0 =A0| 3m 38sec
>
> Btrfs results (without lzo comression):
> | Type =A0 =A0 | 2.6.39.4-3 =A0 | 3.2.7
> | Dir cnt =A0| 2m 22sec =A0| 2m 21sec
> | File cnt | =A02m 26sec | 2m 23sec
> | Copy =A0 =A0| 36m 22sec | 39m 35sec
> | Remove| 7m 51sec =A0 | 10m 43sec
>
> From above one can see that copy takes close to 1h less on btrfs. I'v=
e
> done strace counting times of calls, results are as follows (from
> 3.2.7):
> 1) Ext4 (only to elements):
> % time =A0 =A0 seconds =A0usecs/call =A0 =A0 calls =A0 =A0errors sysc=
all
> ------ ----------- ----------- --------- --------- ----------------
> =A057.01 =A0 13.257850 =A0 =A0 =A0 =A0 =A0 1 =A015082163 =A0 =A0 =A0 =
=A0 =A0 read
> =A023.40 =A0 =A05.440353 =A0 =A0 =A0 =A0 =A0 3 =A0 1687702 =A0 =A0 =A0=
=A0 =A0 getdents
> =A06.15 =A0 =A01.430559 =A0 =A0 =A0 =A0 =A0 0 =A0 3672418 =A0 =A0 =A0=
=A0 =A0 lstat
> =A03.80 =A0 =A00.883767 =A0 =A0 =A0 =A0 =A0 0 =A013106961 =A0 =A0 =A0=
=A0 =A0 write
> =A02.32 =A0 =A00.539959 =A0 =A0 =A0 =A0 =A0 0 =A0 4794099 =A0 =A0 =A0=
=A0 =A0 open
> =A01.69 =A0 =A00.393589 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0843695 =A0 =A0 =A0=
=A0 =A0 mkdir
> =A01.28 =A0 =A00.296700 =A0 =A0 =A0 =A0 =A0 0 =A0 5637802 =A0 =A0 =A0=
=A0 =A0 setxattr
> =A00.80 =A0 =A00.186539 =A0 =A0 =A0 =A0 =A0 0 =A0 7325195 =A0 =A0 =A0=
=A0 =A0 stat
>
> 2) Btrfs:
> % time =A0 =A0 seconds =A0usecs/call =A0 =A0 calls =A0 =A0errors sysc=
all
> ------ ----------- ----------- --------- --------- ----------------
> 53.38 =A0 =A09.486210 =A0 =A0 =A0 =A0 =A0 1 =A015179751 =A0 =A0 =A0 =A0=
=A0 read
> 11.38 =A0 =A02.021662 =A0 =A0 =A0 =A0 =A0 1 =A0 1688328 =A0 =A0 =A0 =A0=
=A0 getdents
> =A010.64 =A0 =A01.890234 =A0 =A0 =A0 =A0 =A0 0 =A0 4800317 =A0 =A0 =A0=
=A0 =A0 open
> =A06.83 =A0 =A01.213723 =A0 =A0 =A0 =A0 =A0 0 =A013201590 =A0 =A0 =A0=
=A0 =A0 write
> =A04.85 =A0 =A00.862731 =A0 =A0 =A0 =A0 =A0 0 =A0 5644314 =A0 =A0 =A0=
=A0 =A0 setxattr
> =A03.50 =A0 =A00.621194 =A0 =A0 =A0 =A0 =A0 1 =A0 =A0844008 =A0 =A0 =A0=
=A0 =A0 mkdir
> =A02.75 =A0 =A00.489059 =A0 =A0 =A0 =A0 =A0 0 =A0 3675992 =A0 =A0 =A0=
=A0 1 lstat
> =A01.71 =A0 =A00.303544 =A0 =A0 =A0 =A0 =A0 0 =A0 5644314 =A0 =A0 =A0=
=A0 =A0 llistxattr
> =A01.50 =A0 =A00.265943 =A0 =A0 =A0 =A0 =A0 0 =A0 1978149 =A0 =A0 =A0=
=A0 =A0 utimes
> =A01.02 =A0 =A00.180585 =A0 =A0 =A0 =A0 =A0 0 =A0 5644314 =A0 =A08440=
08 getxattr
>
> On btrfs getdents takes much less time which prove the bottleneck in
> copy time on ext4 is this syscall. In 2.6.39.4 it shows even less tim=
e
> for getdents:
> % time =A0 =A0 seconds =A0usecs/call =A0 =A0 calls =A0 =A0errors sysc=
all
> ------ ----------- ----------- --------- --------- ----------------
> =A050.77 =A0 10.978816 =A0 =A0 =A0 =A0 =A0 1 =A015033132 =A0 =A0 =A0 =
=A0 =A0 read
> =A014.46 =A0 =A03.125996 =A0 =A0 =A0 =A0 =A0 1 =A0 4733589 =A0 =A0 =A0=
=A0 =A0 open
> =A07.15 =A0 =A01.546311 =A0 =A0 =A0 =A0 =A0 0 =A0 5566988 =A0 =A0 =A0=
=A0 =A0 setxattr
> =A05.89 =A0 =A01.273845 =A0 =A0 =A0 =A0 =A0 0 =A0 3626505 =A0 =A0 =A0=
=A0 =A0 lstat
> =A05.81 =A0 =A01.255858 =A0 =A0 =A0 =A0 =A0 1 =A0 1667050 =A0 =A0 =A0=
=A0 =A0 getdents
> =A05.66 =A0 =A01.224403 =A0 =A0 =A0 =A0 =A0 0 =A013083022 =A0 =A0 =A0=
=A0 =A0 write
> =A03.40 =A0 =A00.735114 =A0 =A0 =A0 =A0 =A0 1 =A0 =A0833371 =A0 =A0 =A0=
=A0 =A0 mkdir
> =A01.96 =A0 =A00.424881 =A0 =A0 =A0 =A0 =A0 0 =A0 5566988 =A0 =A0 =A0=
=A0 =A0 llistxattr
>
>
> Why so huge difference in the getdents timings?
>
> -Jacek
next prev parent reply other threads:[~2012-02-29 13:55 UTC|newest]
Thread overview: 61+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-02-29 13:52 getdents - ext4 vs btrfs performance Jacek Luczak
2012-02-29 13:55 ` Jacek Luczak [this message]
2012-02-29 14:07 ` Jacek Luczak
2012-02-29 14:21 ` Jacek Luczak
2012-02-29 14:42 ` Chris Mason
2012-02-29 14:55 ` Jacek Luczak
2012-03-01 13:35 ` Jacek Luczak
2012-03-01 13:50 ` Hillf Danton
2012-03-01 14:03 ` Jacek Luczak
2012-03-01 14:18 ` Chris Mason
2012-03-01 14:43 ` Jacek Luczak
2012-03-01 14:51 ` Chris Mason
2012-03-01 14:57 ` Jacek Luczak
2012-03-01 18:42 ` Ted Ts'o
2012-03-02 9:51 ` Jacek Luczak
2012-03-01 4:44 ` Theodore Tso
2012-03-01 14:38 ` Chris Mason
2012-03-02 10:05 ` Jacek Luczak
2012-03-02 14:00 ` Chris Mason
2012-03-02 14:16 ` Jacek Luczak
2012-03-02 14:26 ` Chris Mason
2012-03-02 19:32 ` Ted Ts'o
2012-03-02 19:50 ` Chris Mason
2012-03-05 13:10 ` Jan Kara
2012-03-03 22:41 ` Jacek Luczak
2012-03-04 10:25 ` Jacek Luczak
2012-03-05 11:32 ` Jacek Luczak
2012-03-06 0:37 ` Chris Mason
2012-03-08 17:02 ` Phillip Susi
2012-03-09 11:29 ` Lukas Czerner
2012-03-09 14:34 ` Chris Mason
2012-03-10 0:09 ` Andreas Dilger
2012-03-10 4:48 ` Ted Ts'o
2012-03-11 10:30 ` Andreas Dilger
2012-03-11 16:13 ` Ted Ts'o
2012-03-15 10:42 ` Jacek Luczak
2012-03-18 20:56 ` Ted Ts'o
2012-03-13 19:05 ` Phillip Susi
2012-03-13 19:53 ` Ted Ts'o
2012-03-13 20:22 ` Phillip Susi
2012-03-13 21:33 ` Ted Ts'o
2012-03-14 2:48 ` Yongqiang Yang
2012-03-14 2:51 ` Ted Ts'o
2012-03-14 14:17 ` Zach Brown
2012-03-14 16:48 ` Ted Ts'o
2012-03-14 17:37 ` Zach Brown
2012-03-14 8:12 ` Lukas Czerner
2012-03-14 9:29 ` Yongqiang Yang
2012-03-14 9:38 ` Lukas Czerner
2012-03-14 12:50 ` Ted Ts'o
2012-03-14 14:34 ` Lukas Czerner
2012-03-14 17:02 ` Ted Ts'o
2012-03-14 19:17 ` Chris Mason
2012-03-14 14:28 ` Phillip Susi
2012-03-14 16:54 ` Ted Ts'o
2012-03-10 3:52 ` Ted Ts'o
2012-03-15 7:59 ` Jacek Luczak
-- strict thread matches above, loose matches on Subject: below --
2012-02-29 13:31 Jacek Luczak
2012-02-29 13:51 ` Chris Mason
2012-02-29 14:00 ` Lukas Czerner
2012-02-29 14:05 ` Chris Mason
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CADDYkjRni4i3QvWo+hOn-Ct4mFoTpL1-ydmE7fbzYJmHFhS3_Q@mail.gmail.com \
--to=difrost.kernel@gmail.com \
--cc=chris.mason@oracle.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).